organization vault and members pages hang; 500 internal error #2115

Closed
opened 2026-02-05 03:18:04 +03:00 by OVERLORD · 6 comments
Owner

Originally created by @ebdavison on GitHub (Dec 19, 2024).

Vaultwarden Support String

Your environment (Generated via diagnostics page)

  • Vaultwarden version: v1.32.6
  • Web-vault version: v2024.6.2c
  • OS/Arch: linux/x86_64
  • Running within a container: true (Base: Debian)
  • Database type: SQLite
  • Database version: 3.46.0
  • Environment settings overridden!: false
  • Uses a reverse proxy: true
  • IP Header check: true (X-Real-IP)
  • Internet access: true
  • Internet access via a proxy: false
  • DNS Check: true
  • Browser/Server Time Check: true
  • Server/NTP Time Check: true
  • Domain Configuration Check: false
  • HTTPS Check: true
  • Websocket Check: true
  • HTTP Response Checks: false

Config & Details (Generated via diagnostics page)

Show Config & Details

Failed HTTP Checks:

2FA Connector calls:
Header: 'x-frame-options' is present while it should not

Config:

{
  "_duo_akey": null,
  "_enable_duo": true,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_max_note_size": 10000,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": null,
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "***************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": true,
  "disable_icon_download": false,
  "domain": "*****://**************************************",
  "domain_origin": "*****://**************************************",
  "domain_path": "",
  "domain_set": true,
  "duo_context_purge_schedule": "30 * * * * *",
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "duo_use_iframe": false,
  "email_2fa_auto_fallback": false,
  "email_2fa_enforce_on_verified_invite": false,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "enable_websocket": true,
  "enforce_single_org_with_reset_pw_policy": false,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "http_request_block_non_global_ips": true,
  "http_request_block_regex": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "increase_note_size_limit": false,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": "/data/bitwarden.log",
  "log_level": "warn",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 600000,
  "push_enabled": false,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "*****************************",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*****************",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": "********************",
  "smtp_password": "***",
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": "*****************",
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

Vaultwarden Build Version

1.32.6

Deployment method

Official Container Image

Custom deployment method

No response

Reverse Proxy

nginx 1.22.1

Host/Server Operating System

Linux

Operating System Version

Fedora 35

Clients

Web Vault

Client Version

No response

Steps To Reproduce

  1. Login
  2. Go to Admin Console
  3. Click on either Vault or Members
  4. Internal 500 error appears in JS console

Expected Result

vault items and/or member list to show up

Actual Result

erros in console

Failed to load resource: the server responded with a status of 500 ()
https://bw.myservercom:48080/api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true

Logs

Nothing in the docker logs
Nothing in the nginx logs

Screenshots or Videos

image

image

Additional Context

No response

Originally created by @ebdavison on GitHub (Dec 19, 2024). ### Vaultwarden Support String ### Your environment (Generated via diagnostics page) * Vaultwarden version: v1.32.6 * Web-vault version: v2024.6.2c * OS/Arch: linux/x86_64 * Running within a container: true (Base: Debian) * Database type: SQLite * Database version: 3.46.0 * Environment settings overridden!: false * Uses a reverse proxy: true * IP Header check: true (X-Real-IP) * Internet access: true * Internet access via a proxy: false * DNS Check: true * Browser/Server Time Check: true * Server/NTP Time Check: true * Domain Configuration Check: false * HTTPS Check: true * Websocket Check: true * HTTP Response Checks: false ### Config & Details (Generated via diagnostics page) <details><summary>Show Config & Details</summary> **Failed HTTP Checks:** ```yaml 2FA Connector calls: Header: 'x-frame-options' is present while it should not ``` **Config:** ```json { "_duo_akey": null, "_enable_duo": true, "_enable_email_2fa": true, "_enable_smtp": true, "_enable_yubico": true, "_icon_service_csp": "", "_icon_service_url": "", "_ip_header_enabled": true, "_max_note_size": 10000, "_smtp_img_src": "cid:", "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_session_lifetime": 20, "admin_token": null, "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "auth_request_purge_schedule": "30 * * * * *", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "***************", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": true, "disable_icon_download": false, "domain": "*****://**************************************", "domain_origin": "*****://**************************************", "domain_path": "", "domain_set": true, "duo_context_purge_schedule": "30 * * * * *", "duo_host": null, "duo_ikey": null, "duo_skey": null, "duo_use_iframe": false, "email_2fa_auto_fallback": false, "email_2fa_enforce_on_verified_invite": false, "email_attempts_limit": 3, "email_change_allowed": true, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 3 * * * *", "emergency_request_timeout_schedule": "0 7 * * * *", "enable_db_wal": true, "enable_websocket": true, "enforce_single_org_with_reset_pw_policy": false, "event_cleanup_schedule": "0 10 0 * * *", "events_days_retain": null, "experimental_client_feature_flags": "fido2-vault-credentials", "extended_logging": true, "helo_name": null, "hibp_api_key": null, "http_request_block_non_global_ips": true, "http_request_block_regex": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "increase_note_size_limit": false, "invitation_expiration_hours": 120, "invitation_org_name": "Vaultwarden", "invitations_allowed": true, "ip_header": "X-Real-IP", "job_poll_interval_ms": 30000, "log_file": "/data/bitwarden.log", "log_level": "warn", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "org_events_enabled": false, "org_groups_enabled": false, "password_hints_allowed": true, "password_iterations": 600000, "push_enabled": false, "push_identity_uri": "https://identity.bitwarden.com", "push_installation_id": "***", "push_installation_key": "***", "push_relay_uri": "https://push.bitwarden.com", "reload_templates": false, "require_device_email": false, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sendmail_command": null, "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": false, "signups_allowed": true, "signups_domains_whitelist": "*****************************", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_embed_images": true, "smtp_explicit_tls": null, "smtp_from": "*****************", "smtp_from_name": "Vaultwarden", "smtp_host": "********************", "smtp_password": "***", "smtp_port": 587, "smtp_security": "starttls", "smtp_ssl": null, "smtp_timeout": 15, "smtp_username": "*****************", "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_sendmail": false, "use_syslog": false, "user_attachment_limit": null, "user_send_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null } ``` </details> ### Vaultwarden Build Version 1.32.6 ### Deployment method Official Container Image ### Custom deployment method _No response_ ### Reverse Proxy nginx 1.22.1 ### Host/Server Operating System Linux ### Operating System Version Fedora 35 ### Clients Web Vault ### Client Version _No response_ ### Steps To Reproduce 1. Login 2. Go to Admin Console 3. Click on either Vault or Members 4. <screen hangs> 5. Internal 500 error appears in JS console ### Expected Result vault items and/or member list to show up ### Actual Result erros in console ``` Failed to load resource: the server responded with a status of 500 () https://bw.myservercom:48080/api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true ``` ### Logs ```text Nothing in the docker logs Nothing in the nginx logs ``` ### Screenshots or Videos ![image](https://github.com/user-attachments/assets/2db37f8d-7fd2-4401-87ec-ed732249e3f1) ![image](https://github.com/user-attachments/assets/55ec75fc-41f4-4990-96df-b7da46fdf26f) ### Additional Context _No response_
OVERLORD added the bug label 2026-02-05 03:18:04 +03:00
Author
Owner

@BlackDex commented on GitHub (Dec 19, 2024):

It shows an internal server error. Could you provide the Vaultwarden logs please.
And if you see nothing strange the set the LOG_LEVEL to debug and try again.

You should at least have some logs of Vaultwarden which shows the request is being made. Same btw for nginx, the request should be seen there.

@BlackDex commented on GitHub (Dec 19, 2024): It shows an internal server error. Could you provide the Vaultwarden logs please. And if you see nothing strange the set the `LOG_LEVEL` to `debug` and try again. You should at least have some logs of Vaultwarden which shows the request is being made. Same btw for nginx, the request should be seen there.
Author
Owner

@ebdavison commented on GitHub (Dec 19, 2024):

Nothing in the vaultwarden logs. I will set the LOG_LEVEL and see what shows up.

@ebdavison commented on GitHub (Dec 19, 2024): Nothing in the vaultwarden logs. I will set the `LOG_LEVEL` and see what shows up.
Author
Owner

@ebdavison commented on GitHub (Dec 19, 2024):

[2024-12-19 14:22:58.580][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true
[2024-12-19 14:22:58.584][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections
[2024-12-19 14:22:58.586][response][INFO] (get_org_collections) GET /api/organizations/<org_id>/collections => 200 OK
[2024-12-19 14:22:58.591][panic][ERROR] thread 'rocket-worker-thread' panicked at 'called `Option::unwrap()` on a `None` value': src/db/models/organization.rs:445
   0: vaultwarden::init_logging::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::panicking::panic
   7: core::option::unwrap_failed
   8: vaultwarden::db::models::organization::UserOrganization::to_json_user_details::{{closure}}
   9: vaultwarden::api::core::organizations::get_org_users::into_info::monomorphized_function::{{closure}}
  10: rocket::server::<impl rocket::rkt::Rocket<rocket::phase::Orbit>>::route::{{closure}}
  11: rocket::server::hyper_service_fn::{{closure}}::{{closure}}
  12: tokio::runtime::task::raw::poll
  13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  14: tokio::runtime::scheduler::multi_thread::worker::run
  15: tokio::runtime::task::raw::poll
  16: std::sys::backtrace::__rust_begin_short_backtrace
  17: core::ops::function::FnOnce::call_once{{vtable.shim}}
  18: std::sys::pal::unix::thread::Thread::new::thread_start
  19: <unknown>
  20: __clone

[2024-12-19 14:22:58.614][rocket::server::_][ERROR] Handler get_org_users panicked.
[2024-12-19 14:22:58.615][rocket::server::_][WARN] A panic is treated as an internal server error.
[2024-12-19 14:22:58.615][rocket::server::_][WARN] No 500 catcher registered. Using Rocket default.
[2024-12-19 14:22:58.616][response][INFO] (get_org_users) GET /api/organizations/<org_id>/users?<data..> => 500 Internal Server Error
@ebdavison commented on GitHub (Dec 19, 2024): ``` [2024-12-19 14:22:58.580][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true [2024-12-19 14:22:58.584][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections [2024-12-19 14:22:58.586][response][INFO] (get_org_collections) GET /api/organizations/<org_id>/collections => 200 OK [2024-12-19 14:22:58.591][panic][ERROR] thread 'rocket-worker-thread' panicked at 'called `Option::unwrap()` on a `None` value': src/db/models/organization.rs:445 0: vaultwarden::init_logging::{{closure}} 1: std::panicking::rust_panic_with_hook 2: std::panicking::begin_panic_handler::{{closure}} 3: std::sys::backtrace::__rust_end_short_backtrace 4: rust_begin_unwind 5: core::panicking::panic_fmt 6: core::panicking::panic 7: core::option::unwrap_failed 8: vaultwarden::db::models::organization::UserOrganization::to_json_user_details::{{closure}} 9: vaultwarden::api::core::organizations::get_org_users::into_info::monomorphized_function::{{closure}} 10: rocket::server::<impl rocket::rkt::Rocket<rocket::phase::Orbit>>::route::{{closure}} 11: rocket::server::hyper_service_fn::{{closure}}::{{closure}} 12: tokio::runtime::task::raw::poll 13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task 14: tokio::runtime::scheduler::multi_thread::worker::run 15: tokio::runtime::task::raw::poll 16: std::sys::backtrace::__rust_begin_short_backtrace 17: core::ops::function::FnOnce::call_once{{vtable.shim}} 18: std::sys::pal::unix::thread::Thread::new::thread_start 19: <unknown> 20: __clone [2024-12-19 14:22:58.614][rocket::server::_][ERROR] Handler get_org_users panicked. [2024-12-19 14:22:58.615][rocket::server::_][WARN] A panic is treated as an internal server error. [2024-12-19 14:22:58.615][rocket::server::_][WARN] No 500 catcher registered. Using Rocket default. [2024-12-19 14:22:58.616][response][INFO] (get_org_users) GET /api/organizations/<org_id>/users?<data..> => 500 Internal Server Error ```
Author
Owner

@ebdavison commented on GitHub (Dec 19, 2024):

and the matching nginx logs:

/var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections HTTP/2.0" 200 715 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.myserver.com" sn="bw.retirementprosperitygroup.com" rt=0.005 ua="127.0.0.1:8080" us="200" ut="0.005" ul="1331" cs=-
/var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true HTTP/2.0" 500 169 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.retirementprosperitygroup.com" sn="bw.myserver.com" rt=0.038 ua="[::1]:8080" us="500" ut="0.037" ul="169" cs=-
@ebdavison commented on GitHub (Dec 19, 2024): and the matching nginx logs: ``` /var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections HTTP/2.0" 200 715 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.myserver.com" sn="bw.retirementprosperitygroup.com" rt=0.005 ua="127.0.0.1:8080" us="200" ut="0.005" ul="1331" cs=- /var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true HTTP/2.0" 500 169 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.retirementprosperitygroup.com" sn="bw.myserver.com" rt=0.038 ua="[::1]:8080" us="500" ut="0.037" ul="169" cs=- ```
Author
Owner

@BlackDex commented on GitHub (Dec 19, 2024):

That is a very strange issue. While the code could made a bit more safe, this actually could never happen unless the database is corrupted or someone deleted a user manually with foreign keys disabled.

@BlackDex commented on GitHub (Dec 19, 2024): That is a very strange issue. While the code could made a bit more safe, this actually could never happen unless the database is corrupted or someone deleted a user manually with foreign keys disabled.
Author
Owner

@BlackDex commented on GitHub (Dec 20, 2024):

Since this would cause inaccurate data and seems like a serious database issue on which Vaultwarden could not do anything it self. We have Foreign Keys enabled for a reason to prevent these kind of issue.

And, this is not an actual bug in Vaultwarden per say, as we would probably still panic here or error out, I'm going to move this to discussions.

I would suggest to check the integerty of the database and try to figure out which user is still a member of the org but not in the users table anymore.

@BlackDex commented on GitHub (Dec 20, 2024): Since this would cause inaccurate data and seems like a serious database issue on which Vaultwarden could not do anything it self. We have Foreign Keys enabled for a reason to prevent these kind of issue. And, this is not an actual bug in Vaultwarden per say, as we would probably still panic here or error out, I'm going to move this to discussions. I would suggest to check the integerty of the database and try to figure out which user is still a member of the org but not in the `users` table anymore.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/vaultwarden#2115