mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2026-02-05 00:29:40 +03:00
Seeing nginx reverse proxy error on upgrade to 1.25.1 from 1.25.0 #1325
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @githubbiswb on GitHub (Jul 18, 2022).
Subject of the issue
Seeing nginx reverse proxy error on upgrade to 1.25.1 from 1.25.0
Deployment environment
Docker swarm for both the reverse proxy and the valutwarden image
vaultwarden version:
Install method:
Clients used:
Reverse proxy and version:
MySQL/MariaDB or PostgreSQL version:
Other relevant details:
Steps to reproduce
Expected behaviour
Actual behaviour
Troubleshooting data
@BlackDex commented on GitHub (Jul 18, 2022):
I'm using nginx as well, and have no issues.
Please check and verify the nginx config with the examples here https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples
There could also be some other nginx configs which could cause these timeouts or disconnects.
Maybe try to enable debug logging on the Vaultwarden side and check the logs when this happens.
Could you also post the support string which you can generate via the diagnostics page in the admin interface.
@githubbiswb commented on GitHub (Jul 18, 2022):
Just a reminder, this doesn't happen on version 1.25.0 and does happen in version 1.25.1, so while it could be the proxy, odd it only activates after the upgrade
My reverse proxy config is this one
https://github.com/linuxserver/reverse-proxy-confs/blob/master/bitwarden.subdomain.conf.sample
My nginx config is this one
https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf
My debug logs from the container output
[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/folders/
[2022-07-18 18:05:07.542][routes][INFO] GET /api/folders/
[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders/
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/folders/
[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders//delete
[2022-07-18 18:05:07.542][routes][INFO] GET /api/hibp/breach?
[2022-07-18 18:05:07.542][routes][INFO] GET /api/now
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<_org_id>/users/<_org_user_id>/accept
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id>
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<col_id>
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete-user/<org_user_id>
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id>/user/<org_user_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/details
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/users
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<coll_id>/users
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/delete
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/import
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/keys
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/leave
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/<pol_type>
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/policies/<pol_type>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/token?
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/tax
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users/<org_user_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users/<org_user_id>
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id> [1]
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/users/<org_user_id> [1]
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/confirm
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/delete
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<user_org>/reinvite
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/confirm
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/invite
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/public-keys
[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/reinvite
[2022-07-18 18:05:07.542][routes][INFO] GET /api/plans
[2022-07-18 18:05:07.542][routes][INFO] GET /api/plans/sales-tax-rates
[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends
[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/sends/
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends/
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends//remove-password
[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/<send_id>/<file_id>?
[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/<send_id>/access/file/<file_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/
[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/access/<access_id>
[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/file
[2022-07-18 18:05:07.542][routes][INFO] GET /api/settings/domains
[2022-07-18 18:05:07.542][routes][INFO] POST /api/settings/domains
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/settings/domains
[2022-07-18 18:05:07.542][routes][INFO] GET /api/sync?<data..>
[2022-07-18 18:05:07.542][routes][INFO] GET /api/two-factor
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/authenticator
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/authenticator
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/disable
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/disable
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/duo
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/duo
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/email
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-authenticator
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-duo
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-email
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-recover
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn-challenge
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-yubikey
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/recover
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email-login
[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/two-factor/webauthn
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/webauthn
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/webauthn
[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/yubikey
[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/yubikey
[2022-07-18 18:05:07.542][routes][INFO] GET /api/users//public-key
[2022-07-18 18:05:07.542][routes][INFO] GET /api/version
[2022-07-18 18:05:07.542][routes][INFO] GET /app-id.json
[2022-07-18 18:05:07.542][routes][INFO] GET /attachments//<file_id>
[2022-07-18 18:05:07.542][routes][INFO] GET /icons//icon.png
[2022-07-18 18:05:07.542][routes][INFO] POST /identity/accounts/prelogin
[2022-07-18 18:05:07.542][routes][INFO] POST /identity/connect/token
[2022-07-18 18:05:07.542][routes][INFO] GET /notifications/hub
[2022-07-18 18:05:07.542][routes][INFO] POST /notifications/hub/negotiate
[2022-07-18 18:05:07.542][routes][INFO] GET /vw_static/
[2022-07-18 18:05:07.542][start][INFO] Rocket has launched from http://0.0.0.0:80
[2022-07-18 18:05:37.522][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2022-07-18 18:06:07.481][request][INFO] GET /alive
[2022-07-18 18:06:07.482][response][INFO] (alive) GET /alive => 200 OK
My output from the admin interface support string
Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page)
Show Running Config
Environment settings which are overridden:
@githubbiswb commented on GitHub (Jul 18, 2022):
And to make this even more interesting, I actually run a coop site which has a live vaultwarden server as well as a reverse proxy there and I am NOT seeing the issues 1.25.1. It matches my main site exactly in configs, but only runs on a single server. Still in swarm mode, but just on one box
@BlackDex commented on GitHub (Jul 18, 2022):
Do you have the other server in swarm mode?? Because Vaultwarden doesn't support failover design. Stuff like sessions and websockets will probably fail or cause issues.
@githubbiswb commented on GitHub (Jul 18, 2022):
Yep, the other is in swarm mode, I did realize one difference, I don't have any active clients connected to the backup coop server, so that could be where it generates the error is when clients make connections, which I assume the IOS app does, and my firefox browswer is almost always open and signed in to my main server.
Also agreed on the failover situation with valutwarden, swarm COULD be used this way but I don't use it this way for valutwarden, I only run a single instance of the container on the swarm at a time. But swarm will spin up a new version if it sees the one it is supposed to be running is down, but that isn't really the failover you are talking about where 2 are running at once and keeping in sync
And again, these errors started in 1.25.1 and I have been running valutwarden/bitwarden_rs for a very long time now
EDIT: My coop site now shows the error, only once, but it came after I logged into the web interface at the coop site to poke around in it
@BlackDex commented on GitHub (Jul 19, 2022):
I'm not seeing logs of Vaultwarden during that specific issue, that is what we need to try and guess what is happening.
It still could be a reverse proxy config issues. There is a change in the websockets code done in that release. It could be that it now sends more data and exceeds the current limits of the nginx config. But we need more info/logs during that specific issue. Either
debugortrace.I have checked my own server which uses nginx also (custom config) but I'm not seeing any issues there in the logs.
@stefan0xC commented on GitHub (Jul 19, 2022):
I had to add
proxy_http_version 1.1;to my nginx config (in thelocation /notifications/hubblock) in order to get a successful upgrade (status code 101 Switching Protocols) instead of the premature close (502 Bad Gateway)@githubbiswb commented on GitHub (Jul 19, 2022):
So my proxy was already proxy_http_version 1.1 as that comes from the main proxy config. But what the linuxserver.io guys do not have in their config is the two lines
I added those and no more errors!
Thanks for the help @stefan0xC and @BlackDex I can close the issue, but I wasn't sure if @stefan0xC was saying something did need fixed and then I wouldn't want to close it on that, so just let me know
@BlackDex commented on GitHub (Jul 19, 2022):
Glad you solved it.
I think those are there too, but maybe there is something wrong with there config. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf#L22..L26
They use a
mapto configure those values. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/nginx.conf#L53..L56