Items Taking 15 Seconds to Save with PUSH_ENABLED=true #1605

New Issue

OVERLORD · 2026-02-05T01:19:31+03:00

OVERLORD commented

2026-02-05 01:19:31 +03:00

Originally created by @quexten on GitHub (Jun 13, 2023).

Subject of the issue

With the PUSH_ENABLED=true environment varibale (merged in #3304) set (and PUSH_INSTALLATION_ID and PUSH_INSTALLATION_KEY), updating an item takes 15 seconds. When disabling the flag, the request takes only ~30ms

The log shows ~100 requests to bitwarden's push servers being made in that time.

Deployment environment

Your environment (Generated via diagnostics page)

Vaultwarden version: v1.28.1-bd883de7
Web-vault version: v2023.5.0
OS/Arch: linux/aarch64
Running within Docker: true (Base: Alpine)
Environment settings overridden: false
Uses a reverse proxy: true
IP Header check: true (X-Forwarded-For)
Internet access: true
Internet access via a proxy: false
DNS Check: true
Browser/Server Time Check: true
Server/NTP Time Check: true
Domain Configuration Check: true
HTTPS Check: true
Database type: SQLite
Database version: 3.41.2
Clients used: Web
Reverse proxy and version: Traefik v3.0
Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden:

{
  "_duo_akey": null,
  "_enable_duo": false,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "***************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://*********************",
  "domain_origin": "*****://*********************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "CENSORED",
  "invitations_allowed": true,
  "ip_header": "X-Forwarded-For",
  "job_poll_interval_ms": 30000,
  "log_file": "/data/vaultwarden.log",
  "log_level": "debug",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 100000,
  "push_enabled": false,
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": true,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": true,
  "signups_allowed": true,
  "signups_domains_whitelist": "",
  "signups_verify": true,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": false,
  "smtp_from": "*********************",
  "smtp_from_name": "Bitwarden",
  "smtp_host": "*******************",
  "smtp_password": "***",
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": true,
  "smtp_timeout": 15,
  "smtp_username": "***********",
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": true,
  "websocket_port": 3012,
  "yubico_client_id": "54614",
  "yubico_secret_key": "***",
  "yubico_server": null
}

push_enabled is false here since I disabled the feature temporarily.

Steps to reproduce

Enable mobile push and edit/create an entry.

Expected behaviour

Take the same amount of time.

Actual behaviour

It takes >10 seconds, nearly a hundred connections are started.

Troubleshooting data

On debug mode, a lot of connections to push.bitwarden.com are opened. I have attached the relevant part of the log.
log.txt

If required I'm happy to do some network traces/debugging to further triage the issue.

Originally created by @quexten on GitHub (Jun 13, 2023).   ### Subject of the issue With the PUSH_ENABLED=true environment varibale (merged in #3304) set (and PUSH_INSTALLATION_ID and PUSH_INSTALLATION_KEY), updating an item takes 15 seconds. When disabling the flag, the request takes only ~30ms ![image](https://github.com/dani-garcia/vaultwarden/assets/11866552/00463e45-d21f-4e16-950c-fb3b471e20ab) The log shows ~100 requests to bitwarden's push servers being made in that time. ### Deployment environment ### Your environment (Generated via diagnostics page) * Vaultwarden version: v1.28.1-bd883de7 * Web-vault version: v2023.5.0 * OS/Arch: linux/aarch64 * Running within Docker: true (Base: Alpine) * Environment settings overridden: false * Uses a reverse proxy: true * IP Header check: true (X-Forwarded-For) * Internet access: true * Internet access via a proxy: false * DNS Check: true * Browser/Server Time Check: true * Server/NTP Time Check: true * Domain Configuration Check: true * HTTPS Check: true * Database type: SQLite * Database version: 3.41.2 * Clients used: Web * Reverse proxy and version: Traefik v3.0 * Other relevant information: ### Config (Generated via diagnostics page) <details><summary>Show Running Config</summary> **Environment settings which are overridden:** ```json { "_duo_akey": null, "_enable_duo": false, "_enable_email_2fa": true, "_enable_smtp": true, "_enable_yubico": true, "_icon_service_csp": "", "_icon_service_url": "", "_ip_header_enabled": true, "_smtp_img_src": "cid:", "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_session_lifetime": 20, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "***************", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "*****://*********************", "domain_origin": "*****://*********************", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 3 * * * *", "emergency_request_timeout_schedule": "0 7 * * * *", "enable_db_wal": true, "event_cleanup_schedule": "0 10 0 * * *", "events_days_retain": null, "extended_logging": true, "helo_name": null, "hibp_api_key": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "invitation_expiration_hours": 120, "invitation_org_name": "CENSORED", "invitations_allowed": true, "ip_header": "X-Forwarded-For", "job_poll_interval_ms": 30000, "log_file": "/data/vaultwarden.log", "log_level": "debug", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "org_events_enabled": false, "org_groups_enabled": false, "password_hints_allowed": true, "password_iterations": 100000, "push_enabled": false, "push_installation_id": "***", "push_installation_key": "***", "push_relay_uri": "https://push.bitwarden.com", "reload_templates": false, "require_device_email": true, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sendmail_command": null, "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": true, "signups_allowed": true, "signups_domains_whitelist": "", "signups_verify": true, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_embed_images": true, "smtp_explicit_tls": false, "smtp_from": "*********************", "smtp_from_name": "Bitwarden", "smtp_host": "*******************", "smtp_password": "***", "smtp_port": 587, "smtp_security": "starttls", "smtp_ssl": true, "smtp_timeout": 15, "smtp_username": "***********", "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_sendmail": false, "use_syslog": false, "user_attachment_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": true, "websocket_port": 3012, "yubico_client_id": "54614", "yubico_secret_key": "***", "yubico_server": null } ``` </details> push_enabled is false here since I disabled the feature temporarily. ### Steps to reproduce Enable mobile push and edit/create an entry. ### Expected behaviour  Take the same amount of time. ### Actual behaviour  It takes >10 seconds, nearly a hundred connections are started. ### Troubleshooting data On debug mode, a lot of connections to push.bitwarden.com are opened. I have attached the relevant part of the log. [log.txt](https://github.com/dani-garcia/vaultwarden/files/11729185/log.txt) If required I'm happy to do some network traces/debugging to further triage the issue.

OVERLORD closed this issue

2026-02-05 01:19:33 +03:00

OVERLORD commented

2026-02-05 01:19:36 +03:00

@quexten commented on GitHub (Jun 13, 2023):

Ok, I have investigated my devices table in sql. Over the years (since 2020) ~100 have accumulated (from one time browser sessions and such). The number of these corresponds exactly with the amount of connections being made. The "push_uuid" field is "null" on all of these though, so I would have expected no requests to be made to the push servers for these.

@quexten commented on GitHub (Jun 13, 2023): Ok, I have investigated my devices table in sql. Over the years (since 2020) ~100 have accumulated (from one time browser sessions and such). The number of these corresponds exactly with the amount of connections being made. The "push_uuid" field is "null" on all of these though, so I would have expected no requests to be made to the push servers for these.

OVERLORD commented

2026-02-05 01:19:38 +03:00

@quexten commented on GitHub (Jun 13, 2023):

After manually deleting the outdated device rows from the SQLite database, it now saves in 1.7 seconds.

As far as I can tell, there are 3 steps to improve the situation:

Do not send push notifications for devices where the push uuid is null
Run the push in the background so the save request is not blocked
(Delete outdated devices periodically from the table)

@quexten commented on GitHub (Jun 13, 2023): After manually deleting the outdated device rows from the SQLite database, it now saves in 1.7 seconds. As far as I can tell, there are 3 steps to improve the situation: - Do not send push notifications for devices where the push uuid is null - Run the push in the background so the save request is not blocked - (Delete outdated devices periodically from the table)

OVERLORD commented

2026-02-05 01:19:40 +03:00

@ShlomiD83 commented on GitHub (Jun 13, 2023):

Push didn't work for me.
Do I need to do anything else beside enabling the env variables and add ID & key?

@ShlomiD83 commented on GitHub (Jun 13, 2023): Push didn't work for me. Do I need to do anything else beside enabling the env variables and add ID & key?

OVERLORD commented

2026-02-05 01:19:42 +03:00

@BlackDex commented on GitHub (Jun 13, 2023):

Push didn't work for me. Do I need to do anything else beside enabling the env variables and add ID & key?

Check https://github.com/dani-garcia/vaultwarden/discussions/2469#discussioncomment-6159012

@BlackDex commented on GitHub (Jun 13, 2023): > Push didn't work for me. Do I need to do anything else beside enabling the env variables and add ID & key? Check https://github.com/dani-garcia/vaultwarden/discussions/2469#discussioncomment-6159012

OVERLORD commented

2026-02-05 01:19:46 +03:00

@ShlomiD83 commented on GitHub (Jun 13, 2023):

Push didn't work for me. Do I need to do anything else beside enabling the env variables and add ID & key?

Check #2469 (reply in thread)

sorry :)
Thanks.

@ShlomiD83 commented on GitHub (Jun 13, 2023): > > Push didn't work for me. Do I need to do anything else beside enabling the env variables and add ID & key? > > Check [#2469 (reply in thread)](https://github.com/dani-garcia/vaultwarden/discussions/2469#discussioncomment-6159012) sorry :) Thanks.

OVERLORD commented

2026-02-05 01:19:48 +03:00

@kamilos956 commented on GitHub (Oct 23, 2023):

Dude, thank you for your invastigation. I was wondering why my vaultwarden takes few secs to save or modify an entry. After clearing the device table, every changes or new entry take below 1 sec. Thanks once again!

@kamilos956 commented on GitHub (Oct 23, 2023): Dude, thank you for your invastigation. I was wondering why my vaultwarden takes few secs to save or modify an entry. After clearing the device table, every changes or new entry take below 1 sec. Thanks once again!

OVERLORD commented

2026-02-05 01:19:49 +03:00

@luc-ass commented on GitHub (Oct 26, 2023):

Hey @kamilos956 or @quexten, could you elaborate on how to clear old devices? I've been experiencing sluggish performance when saving entries as well... Is there a way to just "clear all" devices?

@luc-ass commented on GitHub (Oct 26, 2023): Hey @kamilos956 or @quexten, could you elaborate on how to clear old devices? I've been experiencing sluggish performance when saving entries as well... Is there a way to just "clear all" devices?

OVERLORD commented

2026-02-05 01:19:51 +03:00

@quexten commented on GitHub (Oct 26, 2023):

For me the easiest way was to open the sqlite database in a separate application, and deleting the rows of the old devices manually.

@quexten commented on GitHub (Oct 26, 2023): For me the easiest way was to open the sqlite database in a separate application, and deleting the rows of the old devices manually.

OVERLORD commented

2026-02-05 01:19:54 +03:00

@stefan0xC commented on GitHub (Oct 26, 2023):

@luc-ass the easiest way to clear all devices for an affected user is to open the /admin panel and click on Deauthorize sessions in the users overview.
ecb31c85d6/src/api/admin.rs (L420)

@stefan0xC commented on GitHub (Oct 26, 2023): @luc-ass the easiest way to clear all devices for an affected user is to open the `/admin` panel and click on `Deauthorize sessions` in the users overview. https://github.com/dani-garcia/vaultwarden/blob/ecb31c85d68202f9d215c8ce12939158fb858af9/src/api/admin.rs#L420

OVERLORD commented

2026-02-05 01:19:57 +03:00

@kamilos956 commented on GitHub (Oct 26, 2023):

From terminal go to the directory where sqlite is stored, then run it by 'sqlite3 db.sqlite3', the run command 'delete from devices where push_uuid is null;' and that all.

@kamilos956 commented on GitHub (Oct 26, 2023): From terminal go to the directory where sqlite is stored, then run it by 'sqlite3 db.sqlite3', the run command 'delete from devices where push_uuid is null;' and that all.

OVERLORD commented

2026-02-05 01:20:03 +03:00

@kamilos956 commented on GitHub (Nov 12, 2023):

After clearing old sessions, new ones does not contain pushid. Only mobile client has it, so always new login from device cause about 10 sec lag white edit or create new entry.

@kamilos956 commented on GitHub (Nov 12, 2023): After clearing old sessions, new ones does not contain pushid. Only mobile client has it, so always new login from device cause about 10 sec lag white edit or create new entry.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/vaultwarden#1605