starred/vaultwarden
1
0
Fork 0
mirror of https://github.com/dani-garcia/vaultwarden.git synced 2025-12-09 09:13:02 +03:00
Code Issues 429 Packages Projects Releases 10 Wiki Activity

Vaultwarden starts timing out after some time without a restart #1306

New Issue
Closed
opened 2025-10-09 17:10:50 +03:00 by OVERLORD · 8 comments
OVERLORD commented 2025-10-09 17:10:50 +03:00
Owner
Copy Link

Originally created by @trwnh on GitHub.

Subject of the issue

I installed vaultwarden and vaultwarden-web via Arch Linux community repos, and everything seems to work fine after setup and configuration... however, several hours after leaving Vaultwarden unattended and then coming back to it to try to do something (e.g. a sync operation), it is found to give an error, and attempting to load the web vault does not work. Loading the URL gives a 502 or 504 (bad gateway or gateway timeout), and loading the IP directly gives a timeout.

Deployment environment

  • vaultwarden version: 1.22.2-1
  • Install method: Arch Linux community repo
  • Clients used: web vault, Chrome extension, Firefox extension, Windows desktop, Linux desktop, Android, iOS
  • Reverse proxy and version: nginx 1.20.1
  • MySQL/MariaDB or PostgreSQL version: n/a
  • Other relevant details: n/a

Steps to reproduce

/etc/vaultwarden.env: (Everything else is commented out as default)

DATA_FOLDER=/var/lib/vaultwarden
WEB_VAULT_FOLDER=/usr/share/webapps/vaultwarden-web
WEB_VAULT_ENABLED=true
WEBSOCKET_ENABLED=true
WEBSOCKET_ADDRESS=0.0.0.0
WEBSOCKET_PORT=3012
EXTENDED_LOGGING=true
LOG_FILE=/var/log/vaultwarden.log
LOG_LEVEL=trace
SIGNUPS_ALLOWED=true
SIGNUPS_VERIFY=true
DOMAIN=https://vault.domain.tld
ROCKET_ADDRESS=0.0.0.0
ROCKET_PORT=8089
ROCKET_LIMITS={json=10485760}

nginx config:

user http;
worker_processes 1;
events {worker_connections  1024;}
http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
            '$status $body_bytes_sent "$http_referer" '
            '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    gzip  on;
    gzip_comp_level 6;
    gzip_vary on;
    gzip_min_length 1000;
    gzip_proxied    expired no-cache no-store private auth;
    gzip_types
        text/plain
        text/css
        text/js
        text/xml
        text/javascript
        application/javascript
        application/x-javascript
        application/json
        application/xml
        application/rss+xml
        image/svg+xml;

include /etc/nginx/sites/*.conf;

    client_max_body_size 512M;
}

nginx site.conf:

server {
  server_name             vault.domain.tld;
  listen                  443 ssl http2;
  listen                  [::]:443 ssl http2;

  ssl_certificate         /etc/letsencrypt/live/vault.domain.tld/fullchain.pem;
  ssl_certificate_key     /etc/letsencrypt/live/vault.domain.tld/privkey.pem;
  include                 /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam             /etc/letsencrypt/ssl-dhparams.pem;

  access_log              logs/vault.domain.tld-access.log main;
  error_log               logs/vault.domain.tld-error.log;


  location / {
  proxy_pass http://localhost:8089;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_headers_hash_max_size 512;
  proxy_headers_hash_bucket_size 64;
  }

  location /notifications/hub {
  proxy_pass http://localhost:3012;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  }

  location /notifications/hub/negotiate {
  proxy_pass http://localhost:8089;
  }
}
server {
    server_name             vault.domain.tld;
    listen                  80;
    listen                  [::]:80;
    return 301              https://$host$request_uri;
}

Expected behaviour

Vaultwarden continues to run indefinitely and listen for requests

Actual behaviour

Vaultwarden begins timing out and requires a restart

Troubleshooting data

Nothing seems to come up in the logs. I've enabled LOG_LEVEL=debug and am still not seeing anything out of the ordinary. It just seems to stop processing requests after some amount of time, despite the service still running. As a last resort I have enabled LOG_LEVEL=trace and am waiting for it to start timing out again.

systemctl status shows this:

● vaultwarden.service - Bitwarden Server (Rust Edition)
     Loaded: loaded (/usr/lib/systemd/system/vaultwarden.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/vaultwarden.service.d
             └─override.conf
     Active: active (running) since Sat 2021-08-28 02:33:18 UTC; 1h 25min ago
       Docs: https://github.com/dani-garcia/vaultwarden
   Main PID: 1234836 (vaultwarden)
      Tasks: 10 (limit: 1150)
     Memory: 16.9M
        CPU: 1.192s
     CGroup: /system.slice/vaultwarden.service
             └─1234836 /usr/bin/vaultwarden

fwiw override.conf is just this:

[Service]
CapabilityBoundingSet=
AmbientCapabilities=
PrivateUsers=yes

Literally the only suspicious thing that I can see is that when Vaultwarden starts to timeout, the memory usage drops to about 7-8M instead of the 16-17M it usually hovers around while working. I doubt it's being killed by the system, because I have a fair amount of RAM left open:

RAM: 595 MB / 973 MB
Originally created by @trwnh on GitHub. ### Subject of the issue I installed vaultwarden and vaultwarden-web via Arch Linux community repos, and everything seems to work fine after setup and configuration... however, several hours after leaving Vaultwarden unattended and then coming back to it to try to do something (e.g. a sync operation), it is found to give an error, and attempting to load the web vault does not work. Loading the URL gives a 502 or 504 (bad gateway or gateway timeout), and loading the IP directly gives a timeout. ### Deployment environment * vaultwarden version: 1.22.2-1 * Install method: Arch Linux `community` repo * Clients used: web vault, Chrome extension, Firefox extension, Windows desktop, Linux desktop, Android, iOS * Reverse proxy and version: nginx 1.20.1 * MySQL/MariaDB or PostgreSQL version: n/a * Other relevant details: n/a ### Steps to reproduce /etc/vaultwarden.env: (Everything else is commented out as default) ``` DATA_FOLDER=/var/lib/vaultwarden WEB_VAULT_FOLDER=/usr/share/webapps/vaultwarden-web WEB_VAULT_ENABLED=true WEBSOCKET_ENABLED=true WEBSOCKET_ADDRESS=0.0.0.0 WEBSOCKET_PORT=3012 EXTENDED_LOGGING=true LOG_FILE=/var/log/vaultwarden.log LOG_LEVEL=trace SIGNUPS_ALLOWED=true SIGNUPS_VERIFY=true DOMAIN=https://vault.domain.tld ROCKET_ADDRESS=0.0.0.0 ROCKET_PORT=8089 ROCKET_LIMITS={json=10485760} ``` nginx config: ``` user http; worker_processes 1; events {worker_connections 1024;} http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log logs/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; gzip on; gzip_comp_level 6; gzip_vary on; gzip_min_length 1000; gzip_proxied expired no-cache no-store private auth; gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/rss+xml image/svg+xml; include /etc/nginx/sites/*.conf; client_max_body_size 512M; } ``` nginx site.conf: ``` server { server_name vault.domain.tld; listen 443 ssl http2; listen [::]:443 ssl http2; ssl_certificate /etc/letsencrypt/live/vault.domain.tld/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/vault.domain.tld/privkey.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; access_log logs/vault.domain.tld-access.log main; error_log logs/vault.domain.tld-error.log; location / { proxy_pass http://localhost:8089; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_headers_hash_max_size 512; proxy_headers_hash_bucket_size 64; } location /notifications/hub { proxy_pass http://localhost:3012; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } location /notifications/hub/negotiate { proxy_pass http://localhost:8089; } } server { server_name vault.domain.tld; listen 80; listen [::]:80; return 301 https://$host$request_uri; } ``` ### Expected behaviour Vaultwarden continues to run indefinitely and listen for requests ### Actual behaviour Vaultwarden begins timing out and requires a restart ### Troubleshooting data Nothing seems to come up in the logs. I've enabled `LOG_LEVEL=debug` and am still not seeing anything out of the ordinary. It just seems to stop processing requests after some amount of time, despite the service still running. As a last resort I have enabled `LOG_LEVEL=trace` and am waiting for it to start timing out again. systemctl status shows this: ``` ● vaultwarden.service - Bitwarden Server (Rust Edition) Loaded: loaded (/usr/lib/systemd/system/vaultwarden.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/vaultwarden.service.d └─override.conf Active: active (running) since Sat 2021-08-28 02:33:18 UTC; 1h 25min ago Docs: https://github.com/dani-garcia/vaultwarden Main PID: 1234836 (vaultwarden) Tasks: 10 (limit: 1150) Memory: 16.9M CPU: 1.192s CGroup: /system.slice/vaultwarden.service └─1234836 /usr/bin/vaultwarden ``` fwiw override.conf is just this: ``` [Service] CapabilityBoundingSet= AmbientCapabilities= PrivateUsers=yes ``` Literally the only suspicious thing that I can see is that when Vaultwarden starts to timeout, the memory usage drops to about 7-8M instead of the 16-17M it usually hovers around while working. I doubt it's being killed by the system, because I have a fair amount of RAM left open: ``` RAM: 595 MB / 973 MB ```
OVERLORD commented 2025-10-09 17:10:52 +03:00
Author
Owner
Copy Link

@BlackDex commented on GitHub:

Well, off it is something in Vaultwarden, then it has to be triggered by something. Only thing i can think of from the top of my mind are the jobs running from time to time. But that should show at least a lot live that it wanted to start it i think. Either that or it broke somehow before that without any logging.

Have you also tried to login to http://localhost:8089 or http://127.0.0.1:8089 when this happens?

@BlackDex commented on GitHub: Well, off it is something in Vaultwarden, then it has to be triggered by something. Only thing i can think of from the top of my mind are the jobs running from time to time. But that should show at least a lot live that it wanted to start it i think. Either that or it broke somehow before that without any logging. Have you also tried to login to http://localhost:8089 or http://127.0.0.1:8089 when this happens?
OVERLORD commented 2025-10-09 17:10:52 +03:00
Author
Owner
Copy Link

@BlackDex commented on GitHub:

I suggest to check what @jjlin mentioned
So items like ss -t to see which items are still connected.
And increase the workers of Vaultwarden

@BlackDex commented on GitHub: I suggest to check what @jjlin mentioned So items like `ss -t` to see which items are still connected. And increase the workers of Vaultwarden
OVERLORD commented 2025-10-09 17:10:52 +03:00
Author
Owner
Copy Link

@trwnh commented on GitHub:

Have you also tried to login to http://localhost:8089 or http://127.0.0.1:8089 when this happens?

Trying to load by direct IP does not work, it just tries to load endlessly. Trying to load from the hostname leads to an nginx gateway timeout.

@trwnh commented on GitHub: > Have you also tried to login to http://localhost:8089 or http://127.0.0.1:8089 when this happens? Trying to load by direct IP does not work, it just tries to load endlessly. Trying to load from the hostname leads to an nginx gateway timeout.
OVERLORD commented 2025-10-09 17:10:53 +03:00
Author
Owner
Copy Link

@trwnh commented on GitHub:

Last 1000 logged trace lines: https://gist.github.com/trwnh/fd8e9e9563c7599526fac1260e7d7d30

@trwnh commented on GitHub: Last 1000 logged trace lines: https://gist.github.com/trwnh/fd8e9e9563c7599526fac1260e7d7d30
OVERLORD commented 2025-10-09 17:10:53 +03:00
Author
Owner
Copy Link

@jjlin commented on GitHub:

This sounds similar to #950 and #1515. See those threads for background.

@jjlin commented on GitHub: This sounds similar to #950 and #1515. See those threads for background.
OVERLORD commented 2025-10-09 17:10:54 +03:00
Author
Owner
Copy Link

@trwnh commented on GitHub:

Looking at nginx error-log I see basically these 2 lines or variants thereof:

2021/08/28 04:57:11 [error] 389203#389203: *138560 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxx:xxxx, server: vault.domain.tld, request: "GET /api/accounts/revision-date HTTP/2.0", upstream: "http://[::1]:8089/api/accounts/revision-date", host: "vault.domain.tld"
2021/08/28 07:18:44 [error] 389203#389203: *140573 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxx, server: vault.domain.tld, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:8089/", host: "vault.domain.tld"

nginx access-log looks normal, just shows me attempting to connect.

ss -t returns the following:

State              Recv-Q         Send-Q                                    Local Address:Port                                                    Peer Address:Port                Process
ESTAB              0              0                                          <my-server-ip>:8089                                                    185.56.80.46:40576
ESTAB              0              0                                          <my-server-ip>:8089                                                    185.56.80.46:12990
CLOSE-WAIT         79             0                                             127.0.0.1:8089                                                       127.0.0.1:59374
CLOSE-WAIT         79             0                                             127.0.0.1:8089                                                       127.0.0.1:59370
CLOSE-WAIT         595            0                                             127.0.0.1:8089                                                       127.0.0.1:59364
ESTAB              0              0                                          <my-server-ip>:8089                                                    185.56.80.46:39717
CLOSE-WAIT         370            0                                          <my-server-ip>:8089                                                   <my-home-ip>:57324
CLOSE-WAIT         79             0                                             127.0.0.1:8089                                                       127.0.0.1:59372
CLOSE-WAIT         370            0                                          <my-server-ip>:8089                                                   <my-home-ip>:51603
CLOSE-WAIT         621            0                                             127.0.0.1:8089                                                       127.0.0.1:59366
CLOSE-WAIT         621            0                                             127.0.0.1:8089                                                       127.0.0.1:59362
CLOSE-WAIT         84             0                                             127.0.0.1:8089                                                       127.0.0.1:59376
CLOSE-WAIT         621            0                                             127.0.0.1:8089                                                       127.0.0.1:59368
@trwnh commented on GitHub: Looking at nginx error-log I see basically these 2 lines or variants thereof: ``` 2021/08/28 04:57:11 [error] 389203#389203: *138560 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxx:xxxx, server: vault.domain.tld, request: "GET /api/accounts/revision-date HTTP/2.0", upstream: "http://[::1]:8089/api/accounts/revision-date", host: "vault.domain.tld" 2021/08/28 07:18:44 [error] 389203#389203: *140573 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxx, server: vault.domain.tld, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:8089/", host: "vault.domain.tld" ``` nginx access-log looks normal, just shows me attempting to connect. `ss -t` returns the following: ``` State Recv-Q Send-Q Local Address:Port Peer Address:Port Process ESTAB 0 0 <my-server-ip>:8089 185.56.80.46:40576 ESTAB 0 0 <my-server-ip>:8089 185.56.80.46:12990 CLOSE-WAIT 79 0 127.0.0.1:8089 127.0.0.1:59374 CLOSE-WAIT 79 0 127.0.0.1:8089 127.0.0.1:59370 CLOSE-WAIT 595 0 127.0.0.1:8089 127.0.0.1:59364 ESTAB 0 0 <my-server-ip>:8089 185.56.80.46:39717 CLOSE-WAIT 370 0 <my-server-ip>:8089 <my-home-ip>:57324 CLOSE-WAIT 79 0 127.0.0.1:8089 127.0.0.1:59372 CLOSE-WAIT 370 0 <my-server-ip>:8089 <my-home-ip>:51603 CLOSE-WAIT 621 0 127.0.0.1:8089 127.0.0.1:59366 CLOSE-WAIT 621 0 127.0.0.1:8089 127.0.0.1:59362 CLOSE-WAIT 84 0 127.0.0.1:8089 127.0.0.1:59376 CLOSE-WAIT 621 0 127.0.0.1:8089 127.0.0.1:59368 ```
OVERLORD commented 2025-10-09 17:10:54 +03:00
Author
Owner
Copy Link

@jjlin commented on GitHub:

185.56.80.46 is presumably someone probing your server and keeping connections open for some reason. This is exactly what's discussed in #950 and #1515.

nginx annoyingly doesn't have strict SNI support built in, but you can simulate it. See, e.g., https://security.stackexchange.com/a/107918.

@jjlin commented on GitHub: `185.56.80.46` is presumably someone probing your server and keeping connections open for some reason. This is exactly what's discussed in #950 and #1515. nginx annoyingly doesn't have strict SNI support built in, but you can simulate it. See, e.g., https://security.stackexchange.com/a/107918.
OVERLORD commented 2025-10-09 17:10:54 +03:00
Author
Owner
Copy Link

@BlackDex commented on GitHub:

I have a default aka _ host also on my nginx which indeed responds to simple ip traffic.

@BlackDex commented on GitHub: I have a default aka `_` host also on my nginx which indeed responds to simple ip traffic.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
better for forum
bug
bug
bug
documentation
duplicate
enhancement
future Vault
good first issue
help wanted
low priority
notes
pull-request
Mirrored from GitHub Pull Request
 question
SSO
Third party
troubleshooting
troubleshooting
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/vaultwarden#1306
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?