Admin Dashboards trying to auth against localhost even with APPLICATION_URL set

Help me!
1

Hello - we recently migrated from Liana v7 to v8 and our admin dashboards are all unreachable. The dashboard appears to be attempting to query localhost during the /authentication call, but I can’t find any configuration on our end that could cause this.

During our upgrade from v6 to v7 we were initially tripped up by the addition of the APPLICATION_URL environment variable, but our environments currently have this variable set. We were running the latest version of v7 in our environments for a week or so without issue before our upgrade to v8.

We were able to get v8 running on local environments before sending the changes out to our cloud environments. However, we’re now experiencing this issue on both local and cloud envs. As far as we know, nothing has changed configuration wise before and after this failure. I thought that the url this /authentication call should hit was controlled by the APPLICATION_URL environment variable, but I’ve verified that variable is set but the auth call here is still trying to hit localhost.

Feature(s) impacted

  • Admin dashboard

Observed behavior

image
image1666×1048 221 KB

Expected behavior

The failing auth call above hits our application’s url and succeeds.

Context

  • Project name: Trex
  • Environment name: Production
  • Agent type & version: forest-express-mongoose v8.6.5
  • Recent changes made on your end if any: Upgrade from v7 latest to v8.6.5
2

Hi @zach !
Let’s try to sort this out :slight_smile:
Do you have any logs on your backend when trying to authenticate ?
Did you thoroughly follow the ugrade to v8 documentation ?

3

Hi Nicolas,

Yes, we followed the v7 to v8 upgrade docs during this process. Here’s what I’m seeing in my local dev environment’s backend.

0|admin-server  | [forest] 🌳🌳🌳  Unable to register the client
0|admin-server  | {
0|admin-server  |   "configuration": {
0|admin-server  |     "authorization_endpoint": "https://api.forestadmin.com/oidc/auth",
0|admin-server  |     "device_authorization_endpoint": "https://api.forestadmin.com/oidc/device/auth",
0|admin-server  |     "claims_parameter_supported": false,
0|admin-server  |     "claims_supported": [
0|admin-server  |       "sub",
0|admin-server  |       "email",
0|admin-server  |       "sid",
0|admin-server  |       "auth_time",
0|admin-server  |       "iss"
0|admin-server  |     ],
0|admin-server  |     "code_challenge_methods_supported": [
0|admin-server  |       "S256"
0|admin-server  |     ],
0|admin-server  |     "end_session_endpoint": "https://api.forestadmin.com/oidc/session/end",
0|admin-server  |     "grant_types_supported": [
0|admin-server  |       "authorization_code",
0|admin-server  |       "urn:ietf:params:oauth:grant-type:device_code"
0|admin-server  |     ],
0|admin-server  |     "id_token_signing_alg_values_supported": [
0|admin-server  |       "HS256",
0|admin-server  |       "RS256"
0|admin-server  |     ],
0|admin-server  |     "issuer": "https://api.forestadmin.com",
0|admin-server  |     "jwks_uri": "https://api.forestadmin.com/oidc/jwks",
0|admin-server  |     "registration_endpoint": "https://api.forestadmin.com/oidc/reg",
0|admin-server  |     "response_modes_supported": [
0|admin-server  |       "query"
0|admin-server  |     ],
0|admin-server  |     "response_types_supported": [
0|admin-server  |       "code",
0|admin-server  |       "none"
0|admin-server  |     ],
0|admin-server  |     "scopes_supported": [
0|admin-server  |       "openid",
0|admin-server  |       "email",
0|admin-server  |       "profile"
0|admin-server  |     ],
0|admin-server  |     "subject_types_supported": [
0|admin-server  |       "public"
0|admin-server  |     ],
0|admin-server  |     "token_endpoint_auth_methods_supported": [
0|admin-server  |       "none"
0|admin-server  |     ],
0|admin-server  |     "token_endpoint_auth_signing_alg_values_supported": [],
0|admin-server  |     "token_endpoint": "https://api.forestadmin.com/oidc/token",
0|admin-server  |     "request_object_signing_alg_values_supported": [
0|admin-server  |       "HS256",
0|admin-server  |       "RS256"
0|admin-server  |     ],
0|admin-server  |     "request_parameter_supported": false,
0|admin-server  |     "request_uri_parameter_supported": true,
0|admin-server  |     "require_request_uri_registration": true,
0|admin-server  |     "claim_types_supported": [
0|admin-server  |       "normal"
0|admin-server  |     ]
0|admin-server  |   },
0|admin-server  |   "registration": {
0|admin-server  |     "redirect_uris": [
0|admin-server  |       "https://admin.zach.atspoke-local.com/forest/authentication/callback"
0|admin-server  |     ],
0|admin-server  |     "token_endpoint_auth_method": "none"
0|admin-server  |   },
0|admin-server  |   "error": {
0|admin-server  |     "name": "RequestError",
0|admin-server  |     "code": "ERR_TLS_CERT_ALTNAME_INVALID",
0|admin-server  |     "timings": {
0|admin-server  |       "start": 1649866699760,
0|admin-server  |       "socket": 1649866699761,
0|admin-server  |       "lookup": 1649866699762,
0|admin-server  |       "connect": 1649866699762,
0|admin-server  |       "error": 1649866699773,
0|admin-server  |       "phases": {
0|admin-server  |         "wait": 1,
0|admin-server  |         "dns": 1,
0|admin-server  |         "tcp": 0,
0|admin-server  |         "total": 13
0|admin-server  |       }
0|admin-server  |     }
0|admin-server  |   }
0|admin-server  | }
0|admin-server  | [forest] 🌳🌳🌳  Unexpected error: Hostname/IP does not match certificate's altnames: Host: localhost. is not in the cert's altnames: DNS:*.zach.askspoke-local.com, DNS:*.zach.atspoke-local.com, DNS:zach.askspoke-local.com, DNS:zach.atspoke-local.com
0|admin-server  | {
0|admin-server  |   "name": "RequestError",
0|admin-server  |   "code": "ERR_TLS_CERT_ALTNAME_INVALID",
0|admin-server  |   "timings": {
0|admin-server  |     "start": 1649866699760,
0|admin-server  |     "socket": 1649866699761,
0|admin-server  |     "lookup": 1649866699762,
0|admin-server  |     "connect": 1649866699762,
0|admin-server  |     "error": 1649866699773,
0|admin-server  |     "phases": {
0|admin-server  |       "wait": 1,
0|admin-server  |       "dns": 1,
0|admin-server  |       "tcp": 0,
0|admin-server  |       "total": 13
0|admin-server  |     }
0|admin-server  |   },
0|admin-server  |   "stack": "RequestError: Hostname/IP does not match certificate's altnames: Host: localhost. is not in the cert's altnames: DNS:*.zach.askspoke-local.com, DNS:*.zach.atspoke-local.com, DNS:zach.askspoke-local.com, DNS:zach.atspoke-local.com\n    at ClientRequest.<anonymous> (/Users/zach/Repos/spoke/node_modules/got/dist/source/core/index.js:962:111)\n    at Object.onceWrapper (events.js:422:26)\n    at ClientRequest.emit (events.js:327:22)\n    at ClientRequest.EventEmitter.emit (domain.js:467:12)\n    at ClientRequest.origin.emit (/Users/zach/Repos/spoke/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at TLSSocket.socketErrorListener (_http_client.js:469:9)\n    at TLSSocket.emit (events.js:315:20)\n    at TLSSocket.EventEmitter.emit (domain.js:467:12)\n    at emitErrorNT (internal/streams/destroy.js:106:8)\n    at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21)\n    at Object.checkServerIdentity (tls.js:297:12)\n    at TLSSocket.onConnectSecure (_tls_wrap.js:1507:27)\n    at TLSSocket.emit (events.js:315:20)\n    at TLSSocket.EventEmitter.emit (domain.js:467:12)\n    at TLSSocket._finishInit (_tls_wrap.js:932:8)\n    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)"
0|admin-server  | }

This second error is slightly different, complaining about TLS validation, but the cause is the same. For some reason the admin backend appears to be trying to make a request to itself (localhost:443) and getting an error. Locally I get this TLS validation because I have a webserver running on 443. In our cloud environments we get a connection refused as there’s nothing running on that port in the container our admin backend runs in.

I was mistaken in my original statement that the admin dashboard is attempting to query localhost - it’s actually hitting the /authentication endpoint on the host correctly - but from there the backend is attempting to query localhost for some reason. What could be causing this?

4

Thanks for your answer @zach, while I read all the info you’ve given me again, could you share an anonymized version of the .env file ?
Just to check what variables are set, and when it’s not sensitive, their value.

5

No problem - I’ve gone ahead and DMed you an anonymized .env file, there’s nothing sensitive in it at this point, but I’d rather be overly cautious. Feel free to reply in this thread for visibility though, especially if the answer may help others.

6

So this config was working on v7 ?
Do you have one instance of the backend running or multiple ?
Is your domain accessible externaly ? Or is it a reverse proxy setup with nginx for exemple, usign self signed certs ?

7

Nothing about our environment has changed between v7 and v8 as far as I am aware. My local environment is accessible externally via ngrok tunnels on specific subdomains (the admin.zach.atspoke-local.com included), but most local traffic is served via nginx. Our certificates are not self signed.

The same issues present themselves in our cloud environments. We have noticed that one of our engineers was able to configure a new forest project that worked without issue, using the same branch of our codebase that is running in environments that are seeing failures.

Again, the issue seems to be that the backend is attempting to make requests to a localhost domain for some reason, and this is causing our admin backend to 500 during the /authentication call. The backend is reachable, and the TLS error shown above is just a symptom of the fact that I happen to have something running on port 443 in my local environment. This is not a TLS issue generally. In our remote environments the error is just an ECONNREFUSED error because there is nothing running at that port.

8

@anon94532230 We’ve done some more digging on our end and came across this post from last year. It seemed promising - the error message mentioned in the title was the exact one we were seeing in our cloud environments.

We determined that this was an instance where one of our other dependencies was interfering with the openid-client dependency of forest-express.

The culprit in our case, which was google-auth-library. We found that we had a dependency using a version of google-auth-library earlier than v3.x. Upgrading this package to the point that it was using a 3.x version of google-auth-library seems like it is resolving our issues.

I don’t see any mention of the possibility that this can happen in the upgrade docs for v7, when this dependency was introduced. A note anywhere in this documentation that there are some known issues with other project dependencies causing these issues may have prevented us from running into this.

1 Like
9

Hi @zach !
Glad to hear oyu solved your issue !
I’ll share yoour remark with the people handling our documentation.