mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2026-02-05 00:29:40 +03:00
Can't login since update 1.30 #1762
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @soberhofer on GitHub (Nov 9, 2023).
Subject of the issue
After updating to 1.30 vaultwarden seems to work fine on all the clients i am already logged in on.
When trying to log in via the Web Vault i get a "Username or password incorrect" message. I can verify that the username is correct, since i can still log in by approving the login from another device.
The password must also be correct, as i have copy pasted it and i can successfully unlock the vault on the desktop App, mobile App and Browser extensions. I get the same error when using a private Tab / different Browser / different Device / different Network. Interesting thing is: Another user can login via the web Vault just fine
I turned on debug logging on my vaultwarden container, but no additional information is given there.
Deployment environment
Install method: docker vaultwarden/server:latest image. ARM Platform
Clients used: web vault, desktop, iOS
Reverse proxy and version: traefik 1.7.34
MySQL/MariaDB or PostgreSQL version: SQLite: 3.41.2
Other relevant details: System running on ARM
Expected behaviour
Login Successful
Actual behaviour
Error Message: Username or password is incorrect. Try again
Troubleshooting data
Logs when logging in
EDIT: Support String
@Rubn86 commented on GitHub (Nov 12, 2023):
I am having the exact same issue after updating to version 1.29.2 (was working correctly on version 1.28.2)
The Bitwarden windows client is working without any issues but logging in through the Vault Web is giving me incorrect username/password.
@BlackDex commented on GitHub (Nov 12, 2023):
You can login using username and password on the desktop? Not unlock, but really login? Because if that works, then your username and password are correct, and stored correctly in the database.
@BlackDex commented on GitHub (Nov 12, 2023):
Also, what if you go back to 1.29.2?
@Rubn86 commented on GitHub (Nov 12, 2023):
I didn't realize it was just an unlock, but you are correct, I am unlocking the vault not logging in.
After signing out, I get the same incorrect username/password message.
The previous version I used was 1.28.1 (correction to my previously mentioned version 1.28.2)
I tried rolling back to 1.28.1, with the same result.
@Rubn86 commented on GitHub (Nov 12, 2023):
You can ignore my report, I think there was an issue with the redirected /data folder not picking up the correct path.
I have recreated my container and it is working correctly now, login works.
I will try this for version 1.30 now.
@Rubn86 commented on GitHub (Nov 12, 2023):
Still working correctly after updating to 1.30.0.
QNAP Container station's newest interface had me create the wrong storage mapping.
I noticed that none of the files in the /data folder were being updated after I upgraded.
Should've tested it further.
@BlackDex commented on GitHub (Nov 12, 2023):
@soberhofer, is something like this maybe the same in your case?
Since:
Do you have some more information?
@soberhofer commented on GitHub (Nov 13, 2023):
i did indeed have a default value in the domain value. Changed it to the proper value, issue still persists, but the diagnostics page now shows a green badge with "HTTPS" and "Match"
I checked the database and both users are present (mine which is not working and the other one which is)
Since i can login by authorizing the login via device, i think the user is in fact there and available.
@BlackDex commented on GitHub (Nov 13, 2023):
It could be something is wrong with the database still, or something else strange happened.
The update doesn't change anything.
@BlackDex commented on GitHub (Nov 13, 2023):
@BlackDex commented on GitHub (Nov 14, 2023):
So, if you say you can unlock using your master password in other clients, and use
Login with deviceto access your vault, then maybe this is a way to get it to work again.Create a new Vaultwarden instance which is totally separated from your production, for example run it on your computer. Use the exact same database type and server kdf settlings if you have changed them.
Create an account on that separate instance using the exact same email and password, and also make sure you configure the exact same user kdf settings. Make sure you can login in that instance. If that works, then you should copy the password hash from the temp instance database to your production and replace it. Make sure you backup before you die this of course.
That should fix your login.
@soberhofer commented on GitHub (Nov 14, 2023):
@BlackDex
sqlite> pragma integrity_check;returnsokRegarding your suggestion on how to recover: Thanks i think i might be able to pull off an even simpler solution.
Since i can login via device i should be able to create a new user and import my vault into that user. Since i'm logged in on my devices and they still sync i should be able to export my vault.
I wanted to avoid that, since I find it really strange, that this just started happening (be it from an update or not) so i hoped to be able to solve the root cause
@soberhofer commented on GitHub (Nov 14, 2023):
FYI: I just tried changing my Password via the web interface. That also gave me an
Invalid Passwordmessage and I saw a400 Bad Requestin the Vaultwarden LogsEDIT: Another Data Point. With the Master Password I'm still able to open items which require re-authentication, and I was also able to export my Vault to a .json which needs the master password. So it really is the correct one, it just is not accepted for login...
@BlackDex commented on GitHub (Nov 14, 2023):
The login and change uses the exact same logic, so that is the expected result.
So, the only thing i can think of is that your master-password hash stored in the database is corrupted in some way.
Why and how is not clear as other users on the same instance have no issues at all.
The only thing i can think of is that something during a KDF change (either server or client side) caused something to happen.
I'm not sure from which previous version exactly you updated to v1.30.0, but if it is from v1.28.0, then nothing has changed on that part as far as i know. It was v1.28.0 which increased the default KDF to 600_000 on the server side which should have caused issue right back then if that was the case i think.
Could you share your client kdf settings (
Account settings>Security>Keys)?@soberhofer commented on GitHub (Nov 15, 2023):
I use PBKDF2 SHA-256 with 700_000 iterations
@Electricz1337 commented on GitHub (Nov 16, 2023):
My Instance on Ubuntu 22.04 is also broken after update.
I also thought, that login in the bitwarden app seems to work, but this was just the case because of the caching. After i logged out and tried to log back in it also showed invalid username password.
I can register a new account with my "existing" email and after the login everything is empty.
I am starting the container like so:
sudo docker run --env-file /home/administrator/vaultwarden.env -d --name vaultwarden -v /vw-data/:/data/ -p 8080:80 vaultwarden/server:latest
With SMTP (i will leave this out) and this configured:
DOMAIN=https://srv-vaultwarden
ORG_EVENTS_ENABLED=true
If no solution will be found i will rollback my server since i have backups, so thats not the problem.
@BlackDex commented on GitHub (Nov 16, 2023):
If you have backups, i would love the know the differences for your specific user record in the database between the working and not working.
@Electricz1337 commented on GitHub (Nov 16, 2023):
How do i do it? I would love to provide it to you
@soberhofer commented on GitHub (Nov 16, 2023):
I have compared the state of the database on different timestamps. Before the update i set the KDF Iterations to 700_000 (quite a while ago).
The first backup that was done after the upgrade (and every subsequent one) shows 600_000
password_iterationsfor my user in the users table.I would assume this is the issue. The Web GUI still shows the 700k (as i set them) but the db has 600k.
The other user on my instance has not changed from the default 100k, i assume that's why that one still works.
Also interesting: Although the
password_iterationsinteger was changed to 600k, theclient_kdf_typestill shows 700k in the current db. Is it possible some db migration changed this?@Electricz1337 commented on GitHub (Nov 16, 2023):
After you rolled back, were you able to start and login successfully? I updated yesterday and it broke.
Then i rolled back to tuesday backup, started the version 1.29.2 (as it was running before) and still cant login.
@soberhofer commented on GitHub (Nov 16, 2023):
I didn't roll back yet. Still hoping that staying on this "broken" state might help find and resolve the root cause.
@Electricz1337 commented on GitHub (Nov 16, 2023):
I have both states in my VM Infrastructure. The broken one and a rolled back one.
Database entries in the users table are the same
@Electricz1337 commented on GitHub (Nov 16, 2023):
It somehow seems that the database isnt bound to the docker container:
I can create a new user with the mail adress i previously used. also after checking the database after i created a new user and entry nothing changed in the /vw-data/
@BlackDex commented on GitHub (Nov 16, 2023):
If you can create a user with the exact same mall address, then you are using a different database, because that just isn't possible. Email addresses are unique identifiers and can't exist multiple times.
@Electricz1337 commented on GitHub (Nov 16, 2023):
I figured out one of my problems. Somhow the mounting of the local path failed.
I tested it with a new directory and it didnt created the database there
After completely uninstalling docker.io and reinstalling it, it worked again under the Version 1.29.2
@soberhofer commented on GitHub (Nov 16, 2023):
Just to clarify: In my case i can't invite a user with the same mail. i get the error that the user already exists
@BlackDex commented on GitHub (Nov 16, 2023):
@soberhofer to clarify what this all is.
password_iterationsis the amount of iterations done by the server on the master-password-hash received from the clients, which is the same asPASSWORD_ITERATIONSin the config for Vaultwarden it self.client_kdf_iteris the amount of iterations done by the clients before they generate the master-password-hash which then is sent to the server.So, for me to be clear. For your account, when you were on v1.29.2 it was
600_000for thepassword_iterationsfield, and700_000on theclient_kdf_iter? And this is still the same after the upgrade right?If that is the case, then i really do wonder if the
password_hashorsaltis different between the active database and the backups.Because if those do not differ in any way, and you only updated the server (which also means an updated web-client) it could be something strange there. But if I'm correct you also mentioned that revering back to a previous Vaultwarden didn't resolved your issue right?
@soberhofer commented on GitHub (Nov 16, 2023):
@BlackDex here are the relevant entries:
Before upgrade (08. November):
My User:
password_iterations: 700_000, client_kdf_iter: 700_000Other User:
password_iterations: 100_000, client_kdf_iter: 100_000After upgrade (09. November):
My User:
password_iterations: 600_000, client_kdf_iter: 700_000Other User:
password_iterations: 100_000, client_kdf_iter: 100_000As for the hashes and salts: as they are BLOB's they produce weird outputs when reading them with
select fromin sqlite. How would i easily compare them across two database files?EDIT: BTW I did not revert anything, i think that was @Electricz1337
@Electricz1337 commented on GitHub (Nov 16, 2023):
in the users table in the database there wasn't any difference
Mathijs van Veluw @.***> schrieb am Do., 16. Nov. 2023,
09:14:
@soberhofer commented on GitHub (Nov 16, 2023):
@BlackDex:
I converted
password_hashandsaltto hex to compare them.After the upgrade the
password_hashwas different compared to before the upgrade on my user. Thesaltremained the same.The other user (with 100k PBKDF2 iterations) has unchanged values for
password_hashas well assalt.@BlackDex commented on GitHub (Nov 16, 2023):
@soberhofer.
Thanks for the details. It looks like there is something strange going on there.
Ill have to recheck this my self. The strange thing is that the
password_iterationsshould match what is in your Vaultwarden config, and looking the your first post that states600_000, so how that has become700_000is a mystery too me (as i have not yet looked at the code, but it could still be a mystery after that too haha).@soberhofer commented on GitHub (Nov 16, 2023):
@BlackDex thanks for that.
I'll just leave the setup as it is, as the Password manager stuff works perfectly and i have a good backup strategy in place. I would prefer if we were able to do an inplace fix (maybe copy the password hash and the
password_iterationsvalue from the backup to the production db or something like that :) )@BlackDex commented on GitHub (Nov 16, 2023):
That is the proper solution indeed.
@BlackDex commented on GitHub (Nov 16, 2023):
I am checking the code, but i can not see how those can be mixed in any way at all.
I'm also wrapping my head around how this could have happened at all.
I also find it strange that all the other users still have
100_000as therepassword_iterations, that means nobody logged in at all using there password since v1.28.0 (2023-03-26).Also, if your setting was
700_000somehow, you needed to have set that your self either viaenvor via the admin interface, and after that logged in using your password. But that still doesn't explain how the master-password-hash could be different in some way.The only thing i can think of is that it has something to do with the platform ARM, but since it's an aarch64, 32bit is ruled out.
I probably need to test the back-then build versions of aarch64 and see what happens if i switch those.
Can you check/verify if you ever have changed the
password_iterationsas config for Vaultwarden?@soberhofer commented on GitHub (Nov 20, 2023):
@BlackDex from searching through old versions of my
config.jsoni found out, thatpassword_iterationsin the config file was changed to700_000around September 4th.There is only one other user on this instance besides mine. That user still had
100_000iterations after the upgrade in november. It is almost certain that they did not log in with their password for a long time (no new devices, so no login was needed)If i look at the db now (the user logged in with their password a week ago, i asked them to) they have:
password_iterations: 600_000client_kdf_iter: 100_000EDIT: so if I understand it correctly, the weird thing is that i have
password_iterationsset to700_000in the config (for the last ~2.5 months) but after the upgrade (and maybe subsequent login) the value in the db was changed to600_000for my users@BlackDex commented on GitHub (Nov 20, 2023):
Well yes.
If the value in the
config.jsonwas700_000then it should be for all users, which it did for you, since that seemed to have worked.But, if i look at the config you posted it states
600_000, that is not something changed by Vautlwarden if you have that set your self.Also, even if it was changed in as a config item, and your account already used the
700_000but the server now thinks it is600_000, then it would update the hash. But for that to work it first validates the previous hash with the value from the database, which would be700_000in your case, and then converts that to a600_000hashed value and will use that the next time.I'm not sure which strange quirks have happened here to have caused this, since.
700_000, how is that now600_000?I actually do not think we will figure this out, at least from a code perspective.
The only thing i can think of is that your password somehow didn't updated and the
password_iterationsdid which caused the issue. If that is the case then the only thing i can think of is some kind of database corruption, or storage issue.I would suggest to run the actions describe above, make sure the password hash, and iterations are exactly the same in the current production database as they were in a working backup. But first, stop Vaultwarden, adjust the database, start Vaultwarden again, and see if Vaultwarden will change the
700_000to600_000and make it work on your first password login.@soberhofer commented on GitHub (Nov 20, 2023):
I will do the steps you outlined and report back. thank you very much for your help in the meantime
@soberhofer commented on GitHub (Nov 20, 2023):
@BlackDex It looks like it worked. I have updated
password_iterationsto700_000and set the hash from before the upgrade. I then restarted vaultwarden, and now I am able to login.Interestingly, vaultwarden has updated the
password_iterationsto600_000for my user again (although it is still set to700_000in myconfig.json. I guess it also recalculated the hash for600_000as it's hex value is different again.Is that expected?
@BlackDex commented on GitHub (Nov 20, 2023):
According to your first post, it isn't set to
700_000.So maybe there is something strange is going on there, correct location etc..
@soberhofer commented on GitHub (Nov 20, 2023):
Hmm that's weird. I tried changing the Organization Invitation name via the Admin panel, and the new value showed up in the
config.json, so the mapping appears to be correct.I then tried it the other way around: Changing the Organization invitation name in
config.jsonand restarting the container. The value now also shows in the admin panel.Interestingly the
password_iterationsconfiguration now shows600_000both in the admin interface andconfig.jsonSo i guess somehow the changes didn't really take before.
@ampersandru commented on GitHub (Nov 21, 2023):
this is a pretty bad issue for a mission-critical service - just experienced it today trying to log into a new device on 1.30.1
Reverted back to 1.28.1 and that didn't fix it. Read through this thread and I can't figure out how to fix this, any ideas? I can log in using "log in with device" from my mobile app but I don't see that option on the desktop or web vault
@BlackDex commented on GitHub (Nov 22, 2023):
@ampersandru, same solution as i mentioned for @soberhofer, extract the old password hash from a backup and replace the one in your live database. Also check the iterations if that is different.
If you can somehow trigger this again with a backup, then i would be very interested in what happens. Same btw for @soberhofer.
Can you trigger a same scenario again?
Else its almost impossible for us to mimic
@ampersandru commented on GitHub (Nov 22, 2023):
Additional steps
Thanks for the response - is there a place that walks me through how to extract the old password hash? I do have backups via synology snapshot and hyperbackup
@BlackDex commented on GitHub (Nov 22, 2023):
I have no clue how your backup solution works, or which database type you use. So there is no way for me to provide any guide at all.
Only best option i can give for now is, revert the version and restore a backup from before the change is done. And then check the settings what they are.
I still want to do some testing, but it is pretty hard to debug this since this issue shouldn't happen at all. But i have not dived deeper into this yet.
@ampersandru commented on GitHub (Nov 22, 2023):
I assume its sqlite3 that is default with Vaultwarden.
The only thing that has changed (other than updating VW) since the last time I logged in was changing the SSL Cert from Letsencrypt to Cloudflare (mainly because acme renew stopped working on my synology). I assume that shouldn't change anything
Either way, I figured out how to reset my password (as long as you still are logged in with another device and "log in with device" is enabled) (this will also reset all 2FA on the account):
@BlackDex commented on GitHub (Nov 22, 2023):
That is a way indeed.
It will not help me hunting down this strange thing.
@Electricz1337 commented on GitHub (Nov 22, 2023):
I can stop, delete and pull a new container with Version 1.29.2 without any issues.
But if i pull 1.30.1 a new login is immediately impossible
@BlackDex commented on GitHub (Nov 22, 2023):
@Electricz1337 that would be even more strange.
There must be something else that makes you not able to login.
But without any logs and checks done via the admin interface to see if the user is still there and on the diagnostics page if all is ok.
Also, caching reverse proxies/browsers can have some influence if they still serve old data in some way.
@Electricz1337 commented on GitHub (Nov 22, 2023):
if you tell me what to provide you and from where i can totally do that. I can snapshot the as is state in vsphere and try it out for you
@ampersandru commented on GitHub (Nov 22, 2023):
I forgot to mention that before resetting my master password with the takeover technique, I tried creating a new docker container instance with 1.28.1 and restored data from September and I still could not log in. I created a new reverse proxy for it using the same cloudflare cert as my primary VW instance.
So unless I did completely forget my master password, something is causing the master password to just not work. As stated before, the only change (other than VW updates) was changing my SSL cert from LetsEncrypt to cloudflare a couple months ago. The last time I had to login to a new device was in August
@BlackDex commented on GitHub (Nov 22, 2023):
@Electricz1337 well, that is a nice setup to test with.
What would be needed is the state before it broke, so, in your case that would be the 1.29.2 version then.
And the state details i would need are:
/admin/diagnostics>Generate support stringHave all the items done above on the previous working environment, then update, try to login, and provide the exact same information as above and the items below.
So, did you logged-out before the upgrade, and logged-in directly afterwards or whatever.
LOG_LEVEL=debug(Don't forget to anonymize it)I need to know the state before and after, and hopefully it will give any clue.
@BlackDex commented on GitHub (Nov 22, 2023):
If you restored a backup and used a Vaultwarden version from that same period, that should just work fine.
If not, then either something already went wrong earlier. v1.28.0 (Released in March 2023) was the version which updated the default password iteration value from
100_000to600_000. So, it might be that it went wrong from there already, but that would mean you never did a new login, and only unlocked, and never did any action which required a password validation at the server side.Which would be strange of course. An other option could be a database corruption which isn't fully visible, but that would also be strange.
@Electricz1337 commented on GitHub (Nov 22, 2023):
I cant explain it, but somehow this time it is running..
I updated and tried to login, but it failed - i then recognized that the database wasnt bound to the container.
I reinstalled docker, because i couldnt figure out what was the issue and afterwards it all worked.
@lumpsoid commented on GitHub (Nov 25, 2023):
One possible reason that someone lost their data is that when you run vault through docker, the volume on the computer is created in the root
/folder, where thevw-datafolder is created, which is written in the docker run as/vw-data/:/data/.I missed this at the time, and thought that my data was saved to the folder I run docker-compose.yml from, not the root folder.
and when I moved the system, the data stayed in the root, not what I backed up.
maybe someone else did the same thing when upgrading?
@ampersandru commented on GitHub (Nov 25, 2023):
From what I can tell, no one in this thread lost data - we just couldn't login with our master passwords. Going to our vault admin page still showed all users and number of items still intact
@Security-Bits commented on GitHub (Dec 1, 2023):
Simple & Stupid, but:
Maybe change the path on https://github.com/dani-garcia/vaultwarden/wiki/Updating-the-vaultwarden-image
from
docker run -d --name vaultwarden -v /vw-data/:/data/ -p 80:80 vaultwarden/server:latest
to
docker run -d --name vaultwarden -v $PATH_TO_STORAGE:/data/ -p 80:80 vaultwarden/server:latest
It's obviously an OSI layer 8 problem, and not the projects scope to make sure people read the instructions properly, but it might just save a little bit of time and issues
@BlackDex commented on GitHub (Dec 15, 2023):
Closing this as stale and no clear indication Vaultwarden did any deletions by it self at random.
If you have more information and still think this is a Vaultwarden issue, please reopen with more details
@alexdelprete commented on GitHub (Jan 7, 2024):
Suddenly I couldn't login with my master password...and I never changed it, it was complex and also noted down just in case I forgot it. So something strange happened to that.
After hours of tries, since I was logged in with my mobile phone and fingerprint, I was able to come up with the same exact procedure, even though I had a minor issue in the last step, when changing the master password for the locked out admin user, I was using MS Edge and even though if the master password respected the policy, it always gave me the "Your new master password does not meet the policy requirements". So I switched to Firefox, and changed it there and no error whatsoever. I was then able to login again with my original admin account and the updated master password.
I don't know what happened, but something certainly did, and the lesson is: create an emergency account and have at least one device with device authentication enabled, just in case.
@ampersandru commented on GitHub (Jan 7, 2024):
glad you figured it out. Its just extremely bizarre this is randomly happening to people where there are no log events attributing to it. Its an extremely terrible thing to happen since BW is a mission-critical service.
Did you by any chance change SSL certs or anything? Thats the one thing that I did change within the past few months. I went from LetsEncrypt to Cloudflare certs
@alexdelprete commented on GitHub (Jan 8, 2024):
I must admit I was astonished, because it's been rock-solid for the latest 2y.
The problem is that it's difficult to understand when it actually happened, because I almost always use device login type (fingerprint, passkey, etc.), so I rarely have to input the master password. So I can't say for how long I had this issue.
I use Traefik + Cloudflared, and Traefik manages the certificates and the renewals. I've always used cloudflare certs, and my domain is on cloudflare. So I would exclude that as the cause, at least in my case.
I hope it doesn't happen again, because this will be very difficult to troubleshoot/debug.
@mrclschstr commented on GitHub (Mar 7, 2024):
Many thanks for the workaround. One of my users noticed today that the login via the web vault no longer works with exactly the same symptoms as described by the others. We weren't sure how long this had been the case, but the steps described helped.
To be on the safe side, we have now set up emergency contacts. Let's hope the problem doesn't occur again...
@Cytomax55 commented on GitHub (May 15, 2024):
Wow i just found your post, thank you for the instructions but i found something else that worked for me instead
I changed my ports in the docker compose file from
ports:
- 10234:80
- 3012:3012
to
ports:
- 10234:80
- 445:443
and it magically started working again.....