mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2025-12-10 01:10:09 +03:00
Docker image crashes once in a while #1741
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @neteroster on GitHub.
Subject of the issue
Hi!
I'm using docker to run bitwarden_rs. But every ~10Hours to ~2Days it crashes (no longer accessable), with nothing seems to be wrong in the log.
Is there any way I can trace the problem down?
Your environment
Steps to reproduce
I just start it with
Expected behaviour
Bitwarden_rs should not crash.
Actual behaviour
Bitwarden_rs crashes.
Relevant logs
@jjlin commented on GitHub:
What's the output of
docker inspect bitwarden(when this problem happens)?@jjlin commented on GitHub:
I suspect Docker is having issues on your system. What OS/version and Docker version are you running?
Check if you can upgrade to the latest (or at least recent) version of Docker. You might also consider disabling healthchecks and seeing if that helps. You can do that using
docker run --no-healthcheck ...or by addingto your service definition if you use Docker Compose.
@neteroster commented on GitHub:
@jjlin I'm using Ubuntu 16.04
Docker Engine is the latest version (19.03.8)
I'll try your suggestion, thank you!
@neteroster commented on GitHub:
@jjlin Thank you! I'll do it when the problem happens next time.
@neteroster commented on GitHub:
@jjlin It happened again, here is the output:
@neteroster commented on GitHub:
The problem happened again. Disabling health-check doesn't help.
@neteroster commented on GitHub:
@jjlin Thanks.
curl -kv https://127.0.0.1:80/aliveis:ps -efgives:strace -fp $(pgrep bitwarden_rs)gives:journalctl -u dockerseems to give nothing useful, except for a strange info:I have never started any container whose ID is
7f998a652f2708a1c0abf1ef4f764f4a1b4e6f0bc8fb99940e4c46c674fc6421.@jjlin commented on GitHub:
Your
stracelog shows 10 threads (presumably the 10 Rocket workers) in the middle of receiving data, so I suspect that's why you can't make any new connections. The question is why they're stuck. Try running@jjlin commented on GitHub:
A few things to try:
docker exec -it bitwarden bashand see if you can get any response from the server, e.g. by runningcurl -kv https://localhost:80/alive. Check ifps -efshows anything unusual.strace -fp $(pgrep bitwarden_rs)from the Docker host and see if there's any activity when you try to interact with the server.journalctl -u dockerfrom the Docker host and see if there's anything that looks unusual.@cwammerl commented on GitHub:
I have exactly the same issue on CentOS 8 with latest image
@neteroster commented on GitHub:
I believe that the problem is solved. Thanks again.
@neteroster commented on GitHub:
@jjlin
I can't recognize most of these IP address, so maybe the service is being scanned by some automatic program from all over the world?
But it seems that some connections were not closed properly (they are always in
CLOSE_WAITstatus). (I'm not sure when these connections were made. If they were made after the problem, that's probably reasonable because the service can't deal with the connections after it stuck.)@neteroster commented on GitHub:
@jjlin
I have put Caddy configured as a reverse proxy in front of bitwarden_rs. And now it no longer responds to the requests of IP address. I will see if this helps.
I truly appreciate your assistance all the time!
@jjlin commented on GitHub:
Running
whoison a sample of these IPs shows they're mostly Chinese, and it looks like you're running on Alibaba Cloud, so maybe this is "normal"? I'd be more concerned about theESTABLISHEDconnections; I don't know of a legitimate reason why these connections should be held open. It has the effect of a Slowloris-style attack, though I doubt it's intentional, since they could take you down much more quickly than every few hours or days if that was their goal.In your original log, you can see messages like
This means someone is connecting to your bitwarden_rs by IP address rather than hostname. There are obviously people/companies out there continually scanning IPs to record what services are running and such, and typically this isn't a huge issue, but it's basically creating a denial of service in this case.
You could try increasing the number of Rocket workers (e.g.,
ROCKET_WORKERS=100). If there end up not being 100 of these scanners or whatever holding connections open, then this should work fine, but otherwise it's just delaying the problem.Better would be to cut off a lot of these connections by putting a reverse proxy in front of bitwarden_rs and configuring it such that it only responds to the hostname, not the IP address (personally, I use HAProxy with the
strict-snioption). In addition, you might consider putting your instance behind Cloudflare.