mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2025-12-10 01:10:09 +03:00
Intermittently all items in VW fail to load with 500 Internal Server Error and 'Error loading ciphers: DatabaseError(Unknown, "cache entry already complete")' #626
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ccben87 on GitHub.
Subject of the issue
Intermittently all items in Vault do not display and browser receives 500 Internal Server Error. Logs indicate "[panic][ERROR] thread 'rocket-worker-thread' panicked at 'Error loading ciphers: DatabaseError(Unknown, "cache entry already complete")'"
Logging out and in a few times eventually causes passwords to load and once loaded it continues to work. Multiple users have had this issue.
Deployment environment
Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page)
Show Running Config
Environment settings which are overridden:
Running Docker testing-alpine image inside Kubernetes environment (single replica of Vaultwarden) and connecting to a cluster of 3 Postgres servers (1 primary, 2 replicas) with asynchronous replication.
Web Vault
Kong Gateway proxying to NGINX
Postgres 16.0
Steps to reproduce
Unsure how to reproduce. This has just started out of nowhere after using Vaultwarden for some time. Possibly somehow DB has become corrupted?
Expected behaviour
Web Vault should always load passwords. DB errors are occurring.
Actual behaviour
Sometimes when signing into Web Vault the passwords don't display and the browser receives a 500 Internal Server Error (logs indicate a database error). Otherwise clicking about the various pages works but no passwords are listed. After signing in and out a few times, the experience returns to normal.
Troubleshooting data
@ccben87 commented on GitHub:
Are you able to please advise what the Postgres setting for that would be? Happy to turn it off but not really sure which setting that would be.
@ccben87 commented on GitHub:
No idea what caused it but it seems to have been a corruption in our database and I can't replicate it after restoring from backup. Thanks for your suggestions. Closing this off.
@BlackDex commented on GitHub:
I have no idea my self.
This might help https://wiki.postgresql.org/wiki/CachingResultspgPool
But it could also be an issue with either your storage or memory of the host. Not sure if you have checked the database for corruption?
Maybe this helps https://wiki.postgresql.org/wiki/Corruption
I'm no PostgreSQL expert, so i have no clue either here.
I only know this not an error coming from Vaultwarden but from the database it self.
@BlackDex commented on GitHub:
I suggest to turn off query result caching at the PostgreSQL config.
This is not something we can fix on our side i think.
@ccben87 commented on GitHub:
Really not sure what's going on here. Found this page discussing the error and sounds like it should have been fixed in Postgres long ago: https://postgrespro.com/list/thread-id/2552153
SQL server log:
{"level":"info","ts":"2023-12-04T05:15:18Z","logger":"postgres","msg":"record","logging_pod":"postgres-vaultwarden-1","record":{"log_time":"2023-12-04 05:15:18.936 UTC","user_name":"app","database_name":"app","process_id":"111872","connection_from":"192.168.210.194:60206","session_id":"656d5fb8.1b500","session_line_num":"1","command_tag":"SELECT","session_start_time":"2023-12-04 05:12:24 UTC","virtual_transaction_id":"14/9926","transaction_id":"0","error_severity":"ERROR","sql_state_code":"XX000","message":"cache entry already complete","query":"SELECT DISTINCT \"ciphers\".\"uuid\", \"ciphers\".\"created_at\", \"ciphers\".\"updated_at\", \"ciphers\".\"user_uuid\", \"ciphers\".\"organization_uuid\", \"ciphers\".\"key\", \"ciphers\".\"atype\", \"ciphers\".\"name\", \"ciphers\".\"notes\", \"ciphers\".\"fields\", \"ciphers\".\"data\", \"ciphers\".\"password_history\", \"ciphers\".\"deleted_at\", \"ciphers\".\"reprompt\" FROM ((((((\"ciphers\" LEFT OUTER JOIN \"ciphers_collections\" ON (\"ciphers\".\"uuid\" = \"ciphers_collections\".\"cipher_uuid\")) LEFT OUTER JOIN \"users_organizations\" ON (((\"ciphers\".\"organization_uuid\" = \"users_organizations\".\"org_uuid\") AND (\"users_organizations\".\"user_uuid\" = $1)) AND (\"users_organizations\".\"status\" = $2))) LEFT OUTER JOIN \"users_collections\" ON ((\"ciphers_collections\".\"collection_uuid\" = \"users_collections\".\"collection_uuid\") AND (\"users_organizations\".\"user_uuid\" = \"users_collections\".\"user_uuid\"))) LEFT OUTER JOIN \"groups_users\" ON (\"groups_users\".\"users_organizations_uuid\" = \"users_organizations\".\"uuid\")) LEFT OUTER JOIN \"groups\" ON (\"groups\".\"uuid\" = \"groups_users\".\"groups_uuid\")) LEFT OUTER JOIN \"collections_groups\" ON ((\"collections_groups\".\"collections_uuid\" = \"ciphers_collections\".\"collection_uuid\") AND (\"collections_groups\".\"groups_uuid\" = \"groups\".\"uuid\"))) WHERE (((((\"ciphers\".\"user_uuid\" = $3) OR (\"users_organizations\".\"access_all\" = $4)) OR (\"users_collections\".\"user_uuid\" = $5)) OR (\"groups\".\"access_all\" = $6)) OR (\"collections_groups\".\"collections_uuid\" IS NOT NULL))","backend_type":"client backend","query_id":"0"}}@ccben87 commented on GitHub:
No idea what caused it but it seems to have been a corruption in our database and I can't replicate it after restoring from backup. Thanks for your suggestions. Closing this off.