mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2025-12-09 09:13:02 +03:00
bitwarden_rs appears to be randomly silently crashing on raspberry pi #2208
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @v0idp on GitHub.
So I'm using bitwarden on my machine in a docker container. I use traefik for my domain stuff. It seems to work fine for a few hours but after some time I can't visit the web vault anymore. it just gives me an 404 page not found error. visiting the web vault with ip:port seems to work tho. But Traefik works just fine with every container and I can't tell why it seems to fail after some time for bitwarden.
@mprasil commented on GitHub:
I wonder if there isn't some conflict in configuration that makes traefik drop the back end for a while?
Does traefik show that it updated configuration before this happens?
@mprasil commented on GitHub:
Also do you have some health checks enabled? Maybe they are failing for whatever reason.
@v0idp commented on GitHub:
Yes indeed this is returned by Traefik when no backend is served. There was no error log by traefik tho. Interestingly enough for some reason, a day later the web tresor seems to work again. So I can't really tell why this is happening. I will check the web tresor again later and let you know if it happens again.
@mprasil commented on GitHub:
I think Traefik returns that when it has no backend to serve the request. Check Traefik logs and see if you spot any changes when it breaks. If you have the admin dashboard enabled, do you see bitwarden_rs in the backends?
@v0idp commented on GitHub:
It happened again and no errors or warnings in traefik or bitwarden docker.
@v0idp commented on GitHub:
bitwarden is running normally according to portainer and traefik has no logs for updated configurations
@v0idp commented on GitHub:
I updated traefik to the newest version and it seems there has been no 404 page not found errors anymore. I will close this, if it should happen again I'll come back. thank you!
@v0idp commented on GitHub:
just the web vault randomly stops and traefik says 404 page not found. using my vault with chrome extension or android app still works tho.
@v0idp commented on GitHub:
Sadly I have to reopen this issue since it's still persisting. restarting the bitwarden container just temporary fixes the issue for half a day until it happens again
@BlackDex commented on GitHub:
@v0idp, can you enable some debug logging on traefik?
Also, i don't exactly know how traefik works, but i think it proxies to an internal ip, are you able to do a lynx|links|wget|curl to that internal ip? and does that work? or also returns a 404? what does the bitwarden log report during those 404 requests?
@v0idp commented on GitHub:
I'll set my traefik log level to DEBUG and see if I get any results on testing. bitwarden outputs no logs at all when the web vault stops working. I bet everything I have it has something to do with the bitwarden web vault itself. I suspect it's crashing for whatever reason and that's why traefik gives me a 404 page not found message. Another note: I am using the raspberry build
@v0idp commented on GitHub:
So traefik in debug mode gave no useful information. it still redirects correctly to the container. So it must be something wrong with the web vault. The only thing I was able to see in the bitwarden logs was this:
@mprasil commented on GitHub:
@v0idp I assume 09:00:48 was around the time you tried to unlock the vault?
@BlackDex commented on GitHub:
Are you able to use wget, curl or even Browsh to access the container's ip?
@BlackDex commented on GitHub:
Also, check if you have the latest version of the docker image running. I see that it has been updated 3 days ago.
@v0idp commented on GitHub:
@BlackDex I actually forgot trying to access the web vault via IP. I'm stupid. If it shows me 404 error again I should try accessing it from the local network.
@mprasil Actually no. I just tried half an hour ago. I was still sleeping at 9 AM so no idea why it did that. But yesterday it still worked.
@v0idp commented on GitHub:
I just checked the webpanel in traefik and the frontend and backend of bitwarden disappeared, no logs on traefik or bitwarden. upon restarting bitwarden it's back until it disappears again
@mohammed90 commented on GitHub:
I'm not using traefik and been getting 404 for the /admin page, even with ADMIN_TOKEN set. Only the /admin page gives 404 for me.
@v0idp commented on GitHub:
I can connect to the web vault with the ip and port. it seems like traefik is failing to properly link to it and I have no idea why.
@mprasil commented on GitHub:
Next time you try to unlock, try to open the web console and switch to network tab before unlocking. I wonder if there were any failed requests this time.
@mprasil commented on GitHub:
Check dmesg for any weird stuff like processes getting killed. Also do you limit your container in terms of CPU or memory?
@BlackDex commented on GitHub:
Are there any duplicate ports used by the containers?
Could you try to set the internal bitwarden port to something else say 9080 or something and see if that helps?
@v0idp commented on GitHub:
exactly it just disappears, weirdly enough it appears back in randomly after some time again. it just randomly disappears and appears back as frontend/backend to traefik it seems. and no it's not restarting to my knowledge, just checked it. i have 4 other containers running with the traefik labels for subdomains and they all just work fine without any issue. it's only with bitwarden.
@mprasil commented on GitHub:
I'm using traefik as well and can't reproduce this at all. So you're saying that the container still runs, but for some reason it disappears from traefik backends and frontends? Could that be somehow related to traefik version or docker version?
Also is bitwarden_rs container restarted at any time when it disappears from traefik?
@v0idp commented on GitHub:
Bitwarden is using port 8888 on my end, there are quite some containers which need port 80 why I used different ports for all of them. I also checked the status again and it seems the bitwarden container seemed to restart itself 4 hours ago. so yes it's restarting itself. right now it's showing 404 page not found again. Sidenote: restarting traefik also temporary fixes the problem until it happens again, same with bitwarden_rs container.
@v0idp commented on GitHub:
Directly with docker.
@v0idp commented on GitHub:
I'm running Traefik version 1.7.8
@mprasil commented on GitHub:
Are you using docker-compose or running container directly with docker?
@v0idp commented on GitHub:
I can't see anything unusual in dmesg and I'm not limiting anything no.
@mprasil commented on GitHub:
Can you share your docker run command? (mask the secret stuff)
@v0idp commented on GitHub:
this is the log when restarting
[2019-02-11 13:24:13][_][INFO] GET /notifications/hub (websockets_err), [2019-02-11 13:24:13][rocket::fairing::fairings][INFO] Fairings:, [2019-02-11 13:24:13][_][INFO] 1 launch: Unofficial Warning, [2019-02-11 13:24:13][_][INFO] 1 response: Application Headers, [2019-02-11 13:24:13][bitwarden_rs][WARN] /--------------------------------------------------------------------\, [2019-02-11 13:24:13][bitwarden_rs][WARN] | This is an *unofficial* Bitwarden implementation, DO NOT use the |, [2019-02-11 13:24:13][bitwarden_rs][WARN] | official channels to report bugs/features, regardless of client. |, [2019-02-11 13:24:13][bitwarden_rs][WARN] | Report URL: https://github.com/dani-garcia/bitwarden_rs/issues/new |, [2019-02-11 13:24:13][bitwarden_rs][WARN] \--------------------------------------------------------------------/, [2019-02-11 13:24:13][launch][INFO] Rocket has launched from http://0.0.0.0:80and here the container:

@mprasil commented on GitHub:
And what about your bitwarden_rs docker setup? I find it weird, that the service would restart couple times a day. do you have any logs just before the restart?
@mprasil commented on GitHub:
Any logs before it restarts? I am just wondering why would it restart at all..
@v0idp commented on GitHub:
And to confirm this. Yes bitwarden_rs does restart itself atleast once or twice a day @mprasil
@v0idp commented on GitHub:
I already provided the logs from before and while restarting. there is no other useful logs before that.
@v0idp commented on GitHub:
the bitwarden_rs container seems to restart. nothing else. and then traefik isn't properly linking to it anymore.
@BlackDex commented on GitHub:
do you see some action in dmesg or syslog/messages on the docker host during the same time as the restarts?
@v0idp commented on GitHub:
nothing in syslog or dmesg regarding docker or bitwarden_rs
@BlackDex commented on GitHub:
and nothing strange or out of the order in any log file during that time, also stuff not directly related to docker or bitwarden? its strange that it restarts the whole container, that is not something bitwarden_rs can do as far as i know.
@BlackDex commented on GitHub:
and is it the whole container, or only the bitwarden application it self within the container?
@BlackDex commented on GitHub:
just a brainfart here, could you try to lower the rocket_workers to 4 or something?
@v0idp commented on GitHub:
maybe it's crashing at some point without proper logs and that's where it restarts itself ? because I have the restart policy set to always. but no there are absolutely no useful legs anywhere to be found which just confuses me.
@BlackDex commented on GitHub:
and anything useful from the
docker logscommand??
@mprasil commented on GitHub:
Maybe one more thing to check - see if dmesg contains any issues reported. On raspberry
Under-voltage detectedmight point to some power issues. I know other containers remain stable, but perhaps they aren't using CPU extensively enough to be affected. (also bitwarden_rs image is usingarm7hfbinary, some might only usearm7binaries and not be affected)@v0idp commented on GitHub:
same logs as I posted up there
@v0idp commented on GitHub:
@mprasil commented on GitHub:
Any interesting logs from the docker daemon?
journalctl -u dockershould show some logs.I think the main concern here is the restarting bitwarden_rs. The fact that it restarts without any logs whatsoever is kinda strange, I haven't seen it crash in this way - ever. Rust gives some decent guarantees to limit the number of reasons why service could crash and I'd expect all of these to at least log something..
If it drops down due to some weird reason, this might actually kinda break the docker daemon where it's no longer sure about the state of the container. This would explain why traefik misbehaves as it gets the information from docker daemon. So try to dig some logs from around docker.
@BlackDex commented on GitHub:
Ok, this seems more related to host networking issues then the bitwarden container it self i think.
Also, check
docker network ls --filter driver=bridgeand use the ID in the following commanddocker network inspect <ID>What is the subnet and gateway configured for docker?
Does this overlap with your real/host network somewhere?
Check your host network with
ip addr ls@v0idp commented on GitHub:
the only thing in my dmesg which repeats all the time is this
and
nothing related to the CPU.
@BlackDex commented on GitHub:
Also, please check your power supply for the raspberry.
Make sure it is at least 5volts, better would be 5.1v and also make sure you have a power supply which can deliver 2.5A especially if it is a rpi 3.
@v0idp commented on GitHub:
maybe something in my vault is causing it to restart ? Just found this error maybe it has something to do with it ? Note: I imported my vault from lastpass
@mprasil commented on GitHub:
One think that is slightly different from my configuration is the
traefik.frontend.redirect.entryPoint=httpslabel. Isn't that generating redirect loop? I think it shouldn't honestly, but at this stage I'm out of ideas. Can you maybe remove that label for a test? (also maybe share relevant parts oftraefik.tomlfile.@v0idp commented on GitHub:
it's a log directly from the bitwarden_rs container
@mprasil commented on GitHub:
Is that using the Vault that comes with
bitwarden_rsor some upstream app. The API call looks like nothing I saw before. (but it shouldn't crashbitwarden_rs)@BlackDex commented on GitHub:
Well, I'm clueless atm with all these facts.
only thing i can think of now is update docker, update the bitwarden image and try again. really no idea at this point.
@BlackDex commented on GitHub:
Seems like a call from/to an other application.
https://github.com/EOSIO/eos/issues/3372
But it indeed should not crash it.
Also, it doesn't seem it does, it just returns a 500 error
@v0idp commented on GitHub:
There are bots running to steal coins but searching for bitwarden vaults and it's doing api calls to them ? Any way I can disable this ?
Anyways still wasn't able to find any other logs related to this. maybe one of you wants to look into this via my portainer webpanel.
@mprasil commented on GitHub:
No, the bots are actually trying to exploit completely different service. They just happen to usually run on the same port as your bitwarden service. Nothing to worry about.
@mprasil commented on GitHub:
Ah it looks like the
/v1/wallet/list_keyscall was caused by some bot trying to steal some coins. (see https://github.com/EOSIO/eos/issues/3372 - I suppose your bitwarden_rs server is exposed on port 8888?) It has nothing to do withbitwarden_rshence why I was so confused.Anyways, as I said I don't think the call should crash bitwarden_rs. It just responds with
500like usual when it receives unknown request.Edit: Indeed @BlackDex, looks like. I didn't notice your reply before replying myself.
@v0idp commented on GitHub:
I see thanks. If I have any new info regarding this bug I will guys inform you. Maybe I'll find out. If you guys wanna look for yourself I can give you access to the server, maybe you can find something
@Brice187 commented on GitHub:
Same Problem here on Docker running on QNAP TS-251 (x86)
@mprasil commented on GitHub:
@v0idp I think I'm going to close this as the problems seem to be very specific to your instance and we haven't found a way to reproduce it. If you have more info and wan to look into this further, feel free to reopen.
@mprasil commented on GitHub:
To be honest I think you provided all the information I would check myself. To me it looks like some instability issue, so I'd try swapping the power supply, make sure the pi is cooled properly, etc..
What would really help is getting some other user with the same issue. I've updated the title to be more specific. Hopefully if there's any other user with the same issue, they will comment here.
@mprasil commented on GitHub:
@Brice187 that was a problem with healthcheck, which was added long after this issue was submitted, so that's probably not related.