mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2025-12-09 09:13:02 +03:00
New device registration awaits SMTP which results in bitwarden client timeouts. #144
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @mattpr on GitHub.
Vaultwarden Support String
Everything is in the discussion
Vaultwarden Build Version
v1.33.2
Deployment method
Manually Extracted from Container Image
Custom deployment method
not relevant.
Reverse Proxy
nginx (not relevant)
Host/Server Operating System
Linux
Operating System Version
Ubuntu 22 (not relevant)
Clients
Android
Client Version
2025.4.0 (20100) and 2025.5.0 (debug/standard)
Steps To Reproduce
Discussion here (just see last message).
Issue is when a new device is logged into to vaultwarden server AND smtp is enabled... the code
awaitsthe email to be sent (to check for SMTP error)...meanwhile blocking the HTTP response.If SMTP is slow (it often is) this significantly delays the HTTP response. Client timeouts (android client empirical testing) are 10 seconds and they hang up resulting in unexpected errors for users when vaultwarden is slow (because of slow SMTP). This is also bad user experience because when they first use vaultwarden on a new device they experience very slow logins unless vaultwarden is using a very fast SMTP server.
The fix is that this email sending should be moved into an async function that is not awaited itself by the new device code (so we don't block http response) but if there is an SMTP error it can still be caught and logged within the new async function.
Expected Result
vaultwarden should not block new device http response waiting on sending SMTP notification.
Actual Result
vaultwarden blocks new device HTTP response until SMTP is complete which can take a long time with slow SMTP, leading to 18 second http responses which lead to bitwarden client timeouts (10 seconds) and unexpected errors...as well as slow first logins which makes users think vaultwarden is slow.
Logs
Screenshots or Videos
No response
Additional Context
No response
@BlackDex commented on GitHub:
We would need to spawn it on a different thread, it isn't as simple as just removing the
.await.@mattpr commented on GitHub:
Just to make sure there wasn't some misunderstanding here... There is no timeout happening from the reverse proxy.
The timeout is from the Bitwarden android client.
The nginx reverse proxy records the bitwarden client disconnect as a
499.vaultwarden does not record anything unless you enable trace in which case you see one line that indicates the client hungup before vaultwarden responded.
From empirical testing, the bitwarden (android) client has a apparent 10 second timeout. So anything that causes vaultwarden to take more than 10 seconds to respond will result in client showing user an error (e.g.
This is not a recognized Bitwarden server. You may need to check with your provider or update your server.).If preventing SMTP sending from blocking HTTP response is quite complicated, it would at least be good to surface these potential issues in the logs as a
WARN. This way a reported client error has corresponding server logging explaining what the issue is. Not sure how many people want to leave servers running at INFO or TRACE...and as this case can result in client errors, I think a warning is appropriate.There are a few edge cases that would be useful to have log warnings about here...
@BlackDex commented on GitHub:
I doubt that adding a warn on all disconnects would be very useful, that would probably clutter the logs too much in my opinion. If users experiences issues, then setting the log level to trace or debug is probably the best option and trigger the issue and check the resulting logs then.
@BlackDex commented on GitHub:
My question would be, why is your mail server so slow?
I sometimes have something similar when testing, but never this long of a timeout. And my reverse proxy doesn't cut the connection either that quick.
While trying to send the mail in the background or other thread might be an option, it isn't that trivial.
@mattpr commented on GitHub:
There are lots of reasons an SMTP server can be slow including milters/filters. And yes if the SMTP server was faster this problem would not appear. However SMTP speed (or speed of any API/service/server) is not something that can be guaranteed. So counting on that as a design requirement for vaultwarden seems like this will just come up with different users again and again.
What is guaranteed is the 10 second bitwarden client timeout and that vaultwarden server logs do not make it easy to see that the client hungup before response (unless you enable trace and know what you are looking for) or why vaultwarden is so slow to respond to the client.
Not sure why vaultwarden would want to block an HTTP response while waiting for SMTP notification to go out unless
require_device_emailis enabled. Especially since there is a hard timeout of 10 seconds on the bitwarden clients and so anything unnecessary that slows down http responses creates a confusing user experience if there are random errors that show up and disappear as an SMTP servers speeds up or slows down.Admittedly I don't know enough about rust async/await to say anything regarding the complexity here. But I would expect you should be able to conditionally say I either want to
awaitthe smtp result (or not) depending on whetherrequire_device_emailis enabled. ie, allow SMTP sending to finish (or log error) async/AFTER the http response has been sent in the case whererequire_device_emailis disabled (default).If we don't stop SMTP from blocking http responses when not needed, another improvement would be to improve server-side (vaultwarden) logging around these types of events...so at least the vaultwarden logs make it obvious there is probably a client error and why. Will reduce support discussions from folks like me in the future.
The client reports the error (only useful if you can reproduce the issue and have logging on the client).
The server doesn't report anything in this case unless you enable
tracelogging and look for it in which case it tells you the client disconnected before the http response was sent... which implies the client probably shows a timeout error. But of course unless you enable trace and are looking for this little hint you don't see anything wrong on the server side. So a different improvement would be to surface the "client disappeared" before http response at a warn level (instead of trace) with a bit of context to help others troubleshoot this more easily when they inevitably run into the same problem (e.g. warn message could look something like:client disconnected before http response, possible network problem or client timeout because we took too long to respond).The only other tell in vaultwarden's logs is the long delay between request/response but you have to notice that in the timestamps, the slow response is not called out by vaultwarden in the logs. So while at it, could add a general warning when vaultwarden is slow to respond (time between request and response is more than X seconds).
If SMTP sending is going to block critical request/response path then adding a log warning about a SMTP server would help vaultwarden administrator see what exactly is causing the slow response which in turn is causing the client timeout.
@dislazy commented on GitHub:
ok
@BlackDex commented on GitHub:
Also, This has a bit of a relation to #4693 i think.
@dislazy commented on GitHub:
When I log in to a new device recently, I receive multiple emails during the email verification process.
@mattpr commented on GitHub:
Doesn't sound related to this issue.
@mattpr commented on GitHub:
Could be. I don't know how often a client disconnects before vaultwarden can respond, but as it will generally raise a client-side error (because request can't be completed from client's point of view) it does seem useful to have corresponding server-side warning.
In my case because I experienced the error and could reproduce it, I was able to turn on trace logging and spot the disconnect there as well. For another user reporting an error, they may not be able or available to reproduce the issue.
Even though I was able to reproduce, the issue was not obvious to me either. I eventually spotted the 499 on the nginx access log side. But it wasn't super obvious at first look that there was anything wrong on the server side, only the client error. Then noticing the 499 on nginx led me to look at the vaultwarden timestamps and realize how slow the response was. Seems like these cases could be made more obvious in the server side logs to aid in troubleshooting rather than guess and checking.