Seeing nginx reverse proxy error on upgrade to 1.25.1 from 1.25.0 #1325

Closed
opened 2026-02-05 00:37:57 +03:00 by OVERLORD · 9 comments
Owner

Originally created by @githubbiswb on GitHub (Jul 18, 2022).

Subject of the issue

Seeing nginx reverse proxy error on upgrade to 1.25.1 from 1.25.0

Deployment environment

Docker swarm for both the reverse proxy and the valutwarden image

  • vaultwarden version:

    • 1.25.1 gives the error 1.25.0 does not
  • Install method:

    • docker compose with docker swarm files, happy to share sanitized configs if anyone wants to see them
  • Clients used:

    • Firefox plugin
    • IOS app
  • Reverse proxy and version:

    • SWAG from the linuxserver.io team linuxserver/swag:1.29.0-ls137
  • MySQL/MariaDB or PostgreSQL version:

    • SQLLite
  • Other relevant details:

    • I am not seeing any performance problems but it is filling the logs on the reverse proxy

Steps to reproduce

  • Upgarde to the new version has the issue, downgrade to 1.25.0 and the issue disappears

Expected behaviour

  • Not get that error

Actual behaviour

  • I get that error

Troubleshooting data

Originally created by @githubbiswb on GitHub (Jul 18, 2022). ### Subject of the issue Seeing nginx reverse proxy error on upgrade to 1.25.1 from 1.25.0 * I see this error repeted around every 4 mins * 2022/07/18 13:22:47 [error] 527#527: *9949 upstream prematurely closed connection while reading response header from upstream, client: 10.0.0.5, server: bitwarden.*, request: "GET /notifications/hub?access_token=biglongrandomtokenhere HTTP/1.1", upstream: "http://192.168.255.70:3012/notifications/hub?access_token=biglongrandomtokenhere ### Deployment environment Docker swarm for both the reverse proxy and the valutwarden image * vaultwarden version: * 1.25.1 gives the error 1.25.0 does not * Install method: * docker compose with docker swarm files, happy to share sanitized configs if anyone wants to see them * Clients used: * Firefox plugin * IOS app * Reverse proxy and version: * SWAG from the linuxserver.io team linuxserver/swag:1.29.0-ls137 * MySQL/MariaDB or PostgreSQL version: * SQLLite * Other relevant details: * I am not seeing any performance problems but it is filling the logs on the reverse proxy ### Steps to reproduce * Upgarde to the new version has the issue, downgrade to 1.25.0 and the issue disappears ### Expected behaviour * Not get that error ### Actual behaviour * I get that error ### Troubleshooting data * 2022/07/18 13:22:47 [error] 527#527: *9949 upstream prematurely closed connection while reading response header from upstream, client: 10.0.0.5, server: bitwarden.*, request: "GET /notifications/hub?access_token=biglongrandomtokenhere HTTP/1.1", upstream: "http://192.168.255.70:3012/notifications/hub?access_token=biglongrandomtokenhere
Author
Owner

@BlackDex commented on GitHub (Jul 18, 2022):

I'm using nginx as well, and have no issues.
Please check and verify the nginx config with the examples here https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples

There could also be some other nginx configs which could cause these timeouts or disconnects.

Maybe try to enable debug logging on the Vaultwarden side and check the logs when this happens.

Could you also post the support string which you can generate via the diagnostics page in the admin interface.

@BlackDex commented on GitHub (Jul 18, 2022): I'm using nginx as well, and have no issues. Please check and verify the nginx config with the examples here https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples There could also be some other nginx configs which could cause these timeouts or disconnects. Maybe try to enable debug logging on the Vaultwarden side and check the logs when this happens. Could you also post the support string which you can generate via the diagnostics page in the admin interface.
Author
Owner

@githubbiswb commented on GitHub (Jul 18, 2022):

Just a reminder, this doesn't happen on version 1.25.0 and does happen in version 1.25.1, so while it could be the proxy, odd it only activates after the upgrade

My reverse proxy config is this one
https://github.com/linuxserver/reverse-proxy-confs/blob/master/bitwarden.subdomain.conf.sample

My nginx config is this one
https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf

My debug logs from the container output
[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/folders/

[2022-07-18 18:05:07.542][routes][INFO] GET /api/folders/

[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders/

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/folders/

[2022-07-18 18:05:07.542][routes][INFO] POST /api/folders//delete

[2022-07-18 18:05:07.542][routes][INFO] GET /api/hibp/breach?

[2022-07-18 18:05:07.542][routes][INFO] GET /api/now

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<_org_id>/users/<_org_user_id>/accept

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id>

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<col_id>

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete-user/<org_user_id>

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id>/user/<org_user_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/details

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/users

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<coll_id>/users

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/delete

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/import

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/keys

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/leave

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/<pol_type>

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/policies/<pol_type>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/token?

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/tax

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users/<org_user_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users/<org_user_id>

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id> [1]

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/users/<org_user_id> [1]

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/confirm

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/delete

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<user_org>/reinvite

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/confirm

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/invite

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/public-keys

[2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/reinvite

[2022-07-18 18:05:07.542][routes][INFO] GET /api/plans

[2022-07-18 18:05:07.542][routes][INFO] GET /api/plans/sales-tax-rates

[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends

[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/sends/

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends/

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends//remove-password

[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/<send_id>/<file_id>?

[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/<send_id>/access/file/<file_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/

[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/access/<access_id>

[2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/file

[2022-07-18 18:05:07.542][routes][INFO] GET /api/settings/domains

[2022-07-18 18:05:07.542][routes][INFO] POST /api/settings/domains

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/settings/domains

[2022-07-18 18:05:07.542][routes][INFO] GET /api/sync?<data..>

[2022-07-18 18:05:07.542][routes][INFO] GET /api/two-factor

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/authenticator

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/authenticator

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/disable

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/disable

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/duo

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/duo

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/email

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-authenticator

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-duo

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-email

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-recover

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn-challenge

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-yubikey

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/recover

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email-login

[2022-07-18 18:05:07.542][routes][INFO] DELETE /api/two-factor/webauthn

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/webauthn

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/webauthn

[2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/yubikey

[2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/yubikey

[2022-07-18 18:05:07.542][routes][INFO] GET /api/users//public-key

[2022-07-18 18:05:07.542][routes][INFO] GET /api/version

[2022-07-18 18:05:07.542][routes][INFO] GET /app-id.json

[2022-07-18 18:05:07.542][routes][INFO] GET /attachments//<file_id>

[2022-07-18 18:05:07.542][routes][INFO] GET /icons//icon.png

[2022-07-18 18:05:07.542][routes][INFO] POST /identity/accounts/prelogin

[2022-07-18 18:05:07.542][routes][INFO] POST /identity/connect/token

[2022-07-18 18:05:07.542][routes][INFO] GET /notifications/hub

[2022-07-18 18:05:07.542][routes][INFO] POST /notifications/hub/negotiate

[2022-07-18 18:05:07.542][routes][INFO] GET /vw_static/

[2022-07-18 18:05:07.542][start][INFO] Rocket has launched from http://0.0.0.0:80

[2022-07-18 18:05:37.522][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins

[2022-07-18 18:06:07.481][request][INFO] GET /alive

[2022-07-18 18:06:07.482][response][INFO] (alive) GET /alive => 200 OK

My output from the admin interface support string

Your environment (Generated via diagnostics page)

  • Vaultwarden version: v1.25.1
  • Web-vault version: v2022.6.2
  • Running within Docker: true (Base: Debian)
  • Environment settings overridden: false
  • Uses a reverse proxy: true
  • IP Header check: true (X-Real-IP)
  • Internet access: true
  • Internet access via a proxy: false
  • DNS Check: true
  • Time Check: true
  • Domain Configuration Check: true
  • HTTPS Check: true
  • Database type: SQLite
  • Database version: 3.35.4
  • Clients used:
  • Reverse proxy and version:
  • Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden:

{
  "_duo_akey": null,
  "_enable_duo": false,
  "_enable_email_2fa": false,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_ip_header_enabled": true,
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "****/**.*******",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://*********.*****.***/",
  "domain_origin": "*****://*********.*****.***",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 5 * * * *",
  "emergency_request_timeout_schedule": "0 5 * * * *",
  "enable_db_wal": true,
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": null,
  "log_level": "debug",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "password_hints_allowed": true,
  "password_iterations": 100000,
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": false,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_explicit_tls": null,
  "smtp_from": "",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": null,
  "smtp_password": null,
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_syslog": false,
  "user_attachment_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": true,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}
@githubbiswb commented on GitHub (Jul 18, 2022): Just a reminder, this doesn't happen on version 1.25.0 and does happen in version 1.25.1, so while it could be the proxy, odd it only activates after the upgrade My reverse proxy config is this one https://github.com/linuxserver/reverse-proxy-confs/blob/master/bitwarden.subdomain.conf.sample My nginx config is this one https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf My debug logs from the container output [2022-07-18 18:05:07.542][routes][INFO] POST /api/folders [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/folders/<uuid> [2022-07-18 18:05:07.542][routes][INFO] GET /api/folders/<uuid> [2022-07-18 18:05:07.542][routes][INFO] POST /api/folders/<uuid> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/folders/<uuid> [2022-07-18 18:05:07.542][routes][INFO] POST /api/folders/<uuid>/delete [2022-07-18 18:05:07.542][routes][INFO] GET /api/hibp/breach?<username> [2022-07-18 18:05:07.542][routes][INFO] GET /api/now [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<_org_id>/users/<_org_user_id>/accept [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id> [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id> [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<col_id> [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/collections/<col_id>/delete-user/<org_user_id> [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/collections/<col_id>/user/<org_user_id> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/details [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/collections/<coll_id>/users [2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/collections/<coll_id>/users [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/delete [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/import [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/keys [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/leave [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/<pol_type> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/policies/<pol_type> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/policies/token?<token> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/tax [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/organizations/<org_id>/users/<org_user_id> [2022-07-18 18:05:07.542][routes][INFO] GET /api/organizations/<org_id>/users/<org_user_id> [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id> [1] [2022-07-18 18:05:07.542][routes][INFO] PUT /api/organizations/<org_id>/users/<org_user_id> [1] [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/confirm [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<org_user_id>/delete [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/<user_org>/reinvite [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/confirm [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/invite [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/public-keys [2022-07-18 18:05:07.542][routes][INFO] POST /api/organizations/<org_id>/users/reinvite [2022-07-18 18:05:07.542][routes][INFO] GET /api/plans [2022-07-18 18:05:07.542][routes][INFO] GET /api/plans/sales-tax-rates [2022-07-18 18:05:07.542][routes][INFO] GET /api/sends [2022-07-18 18:05:07.542][routes][INFO] POST /api/sends [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/sends/<id> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends/<id> [2022-07-18 18:05:07.542][routes][INFO] PUT /api/sends/<id>/remove-password [2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/<send_id>/<file_id>?<t> [2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/<send_id>/access/file/<file_id> [2022-07-18 18:05:07.542][routes][INFO] GET /api/sends/<uuid> [2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/access/<access_id> [2022-07-18 18:05:07.542][routes][INFO] POST /api/sends/file [2022-07-18 18:05:07.542][routes][INFO] GET /api/settings/domains [2022-07-18 18:05:07.542][routes][INFO] POST /api/settings/domains [2022-07-18 18:05:07.542][routes][INFO] PUT /api/settings/domains [2022-07-18 18:05:07.542][routes][INFO] GET /api/sync?<data..> [2022-07-18 18:05:07.542][routes][INFO] GET /api/two-factor [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/authenticator [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/authenticator [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/disable [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/disable [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/duo [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/duo [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/email [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-authenticator [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-duo [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-email [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-recover [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-webauthn-challenge [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/get-yubikey [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/recover [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/send-email-login [2022-07-18 18:05:07.542][routes][INFO] DELETE /api/two-factor/webauthn [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/webauthn [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/webauthn [2022-07-18 18:05:07.542][routes][INFO] POST /api/two-factor/yubikey [2022-07-18 18:05:07.542][routes][INFO] PUT /api/two-factor/yubikey [2022-07-18 18:05:07.542][routes][INFO] GET /api/users/<uuid>/public-key [2022-07-18 18:05:07.542][routes][INFO] GET /api/version [2022-07-18 18:05:07.542][routes][INFO] GET /app-id.json [2022-07-18 18:05:07.542][routes][INFO] GET /attachments/<uuid>/<file_id> [2022-07-18 18:05:07.542][routes][INFO] GET /icons/<domain>/icon.png [2022-07-18 18:05:07.542][routes][INFO] POST /identity/accounts/prelogin [2022-07-18 18:05:07.542][routes][INFO] POST /identity/connect/token [2022-07-18 18:05:07.542][routes][INFO] GET /notifications/hub [2022-07-18 18:05:07.542][routes][INFO] POST /notifications/hub/negotiate [2022-07-18 18:05:07.542][routes][INFO] GET /vw_static/<filename> [2022-07-18 18:05:07.542][start][INFO] Rocket has launched from http://0.0.0.0:80 [2022-07-18 18:05:37.522][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins [2022-07-18 18:06:07.481][request][INFO] GET /alive [2022-07-18 18:06:07.482][response][INFO] (alive) GET /alive => 200 OK My output from the admin interface support string ### Your environment (Generated via diagnostics page) * Vaultwarden version: v1.25.1 * Web-vault version: v2022.6.2 * Running within Docker: true (Base: Debian) * Environment settings overridden: false * Uses a reverse proxy: true * IP Header check: true (X-Real-IP) * Internet access: true * Internet access via a proxy: false * DNS Check: true * Time Check: true * Domain Configuration Check: true * HTTPS Check: true * Database type: SQLite * Database version: 3.35.4 * Clients used: * Reverse proxy and version: * Other relevant information: ### Config (Generated via diagnostics page) <details><summary>Show Running Config</summary> **Environment settings which are overridden:** ```json { "_duo_akey": null, "_enable_duo": false, "_enable_email_2fa": false, "_enable_smtp": true, "_enable_yubico": true, "_ip_header_enabled": true, "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "****/**.*******", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "*****://*********.*****.***/", "domain_origin": "*****://*********.*****.***", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 5 * * * *", "emergency_request_timeout_schedule": "0 5 * * * *", "enable_db_wal": true, "extended_logging": true, "helo_name": null, "hibp_api_key": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "invitation_org_name": "Vaultwarden", "invitations_allowed": true, "ip_header": "X-Real-IP", "job_poll_interval_ms": 30000, "log_file": null, "log_level": "debug", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "password_hints_allowed": true, "password_iterations": 100000, "reload_templates": false, "require_device_email": false, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": false, "signups_allowed": false, "signups_domains_whitelist": "", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_explicit_tls": null, "smtp_from": "", "smtp_from_name": "Vaultwarden", "smtp_host": null, "smtp_password": null, "smtp_port": 587, "smtp_security": "starttls", "smtp_ssl": null, "smtp_timeout": 15, "smtp_username": null, "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_syslog": false, "user_attachment_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": true, "websocket_port": 3012, "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null } ``` </details>
Author
Owner

@githubbiswb commented on GitHub (Jul 18, 2022):

And to make this even more interesting, I actually run a coop site which has a live vaultwarden server as well as a reverse proxy there and I am NOT seeing the issues 1.25.1. It matches my main site exactly in configs, but only runs on a single server. Still in swarm mode, but just on one box

@githubbiswb commented on GitHub (Jul 18, 2022): And to make this even more interesting, I actually run a coop site which has a live vaultwarden server as well as a reverse proxy there and I am NOT seeing the issues 1.25.1. It matches my main site exactly in configs, but only runs on a single server. Still in swarm mode, but just on one box
Author
Owner

@BlackDex commented on GitHub (Jul 18, 2022):

Do you have the other server in swarm mode?? Because Vaultwarden doesn't support failover design. Stuff like sessions and websockets will probably fail or cause issues.

@BlackDex commented on GitHub (Jul 18, 2022): Do you have the other server in swarm mode?? Because Vaultwarden doesn't support failover design. Stuff like sessions and websockets will probably fail or cause issues.
Author
Owner

@githubbiswb commented on GitHub (Jul 18, 2022):

Yep, the other is in swarm mode, I did realize one difference, I don't have any active clients connected to the backup coop server, so that could be where it generates the error is when clients make connections, which I assume the IOS app does, and my firefox browswer is almost always open and signed in to my main server.

Also agreed on the failover situation with valutwarden, swarm COULD be used this way but I don't use it this way for valutwarden, I only run a single instance of the container on the swarm at a time. But swarm will spin up a new version if it sees the one it is supposed to be running is down, but that isn't really the failover you are talking about where 2 are running at once and keeping in sync

And again, these errors started in 1.25.1 and I have been running valutwarden/bitwarden_rs for a very long time now

EDIT: My coop site now shows the error, only once, but it came after I logged into the web interface at the coop site to poke around in it

@githubbiswb commented on GitHub (Jul 18, 2022): Yep, the other is in swarm mode, I did realize one difference, I don't have any active clients connected to the backup coop server, so that could be where it generates the error is when clients make connections, which I assume the IOS app does, and my firefox browswer is almost always open and signed in to my main server. Also agreed on the failover situation with valutwarden, swarm COULD be used this way but I don't use it this way for valutwarden, I only run a single instance of the container on the swarm at a time. But swarm will spin up a new version if it sees the one it is supposed to be running is down, but that isn't really the failover you are talking about where 2 are running at once and keeping in sync And again, these errors started in 1.25.1 and I have been running valutwarden/bitwarden_rs for a very long time now EDIT: My coop site now shows the error, only once, but it came after I logged into the web interface at the coop site to poke around in it
Author
Owner

@BlackDex commented on GitHub (Jul 19, 2022):

I'm not seeing logs of Vaultwarden during that specific issue, that is what we need to try and guess what is happening.

It still could be a reverse proxy config issues. There is a change in the websockets code done in that release. It could be that it now sends more data and exceeds the current limits of the nginx config. But we need more info/logs during that specific issue. Either debug or trace.

I have checked my own server which uses nginx also (custom config) but I'm not seeing any issues there in the logs.

@BlackDex commented on GitHub (Jul 19, 2022): I'm not seeing logs of Vaultwarden during that specific issue, that is what we need to try and guess what is happening. It still could be a reverse proxy config issues. There is a change in the websockets code done in that release. It could be that it now sends more data and exceeds the current limits of the nginx config. But we need more info/logs during that specific issue. Either `debug` or `trace`. I have checked my own server which uses nginx also (custom config) but I'm not seeing any issues there in the logs.
Author
Owner

@stefan0xC commented on GitHub (Jul 19, 2022):

I had to add proxy_http_version 1.1; to my nginx config (in the location /notifications/hub block) in order to get a successful upgrade (status code 101 Switching Protocols) instead of the premature close (502 Bad Gateway)

        location /notifications/hub {
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass http://vaultwarden-ws;
        }
@stefan0xC commented on GitHub (Jul 19, 2022): I had to add `proxy_http_version 1.1;` to my nginx config (in the `location /notifications/hub` block) in order to get a successful upgrade (status code 101 Switching Protocols) instead of the premature close (502 Bad Gateway) ``` location /notifications/hub { proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_pass http://vaultwarden-ws; } ```
Author
Owner

@githubbiswb commented on GitHub (Jul 19, 2022):

So my proxy was already proxy_http_version 1.1 as that comes from the main proxy config. But what the linuxserver.io guys do not have in their config is the two lines

  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";

I added those and no more errors!

Thanks for the help @stefan0xC and @BlackDex I can close the issue, but I wasn't sure if @stefan0xC was saying something did need fixed and then I wouldn't want to close it on that, so just let me know

@githubbiswb commented on GitHub (Jul 19, 2022): So my proxy was already proxy_http_version 1.1 as that comes from the main proxy config. But what the linuxserver.io guys do not have in their config is the two lines ``` proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; ``` I added those and no more errors! Thanks for the help @stefan0xC and @BlackDex I can close the issue, but I wasn't sure if @stefan0xC was saying something did need fixed and then I wouldn't want to close it on that, so just let me know
Author
Owner

@BlackDex commented on GitHub (Jul 19, 2022):

So my proxy was already proxy_http_version 1.1 as that comes from the main proxy config. But what the linuxserver.io guys do not have in their config is the two lines

  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";

I added those and no more errors!

Glad you solved it.

I think those are there too, but maybe there is something wrong with there config. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf#L22..L26

They use a map to configure those values. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/nginx.conf#L53..L56

Thanks for the help @stefan0xC and @BlackDex I can close the issue, but I wasn't sure if @stefan0xC was saying something did need fixed and then I wouldn't want to close it on that, so just let me know

@BlackDex commented on GitHub (Jul 19, 2022): > So my proxy was already proxy_http_version 1.1 as that comes from the main proxy config. But what the linuxserver.io guys do not have in their config is the two lines > > ``` > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > ``` > > I added those and no more errors! Glad you solved it. I think those are there too, but maybe there is something wrong with there config. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/proxy.conf#L22..L26 They use a `map` to configure those values. https://github.com/linuxserver/docker-swag/blob/master/root/defaults/nginx.conf#L53..L56 > > Thanks for the help @stefan0xC and @BlackDex I can close the issue, but I wasn't sure if @stefan0xC was saying something did need fixed and then I wouldn't want to close it on that, so just let me know
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/vaultwarden#1325