mirror of
https://github.com/immich-app/immich.git
synced 2026-02-05 00:30:57 +03:00
[BUG] Request failed with status code 503 #898
Closed
opened 2026-02-04 23:25:13 +03:00 by OVERLORD
·
33 comments
No Branch/Tag Specified
main
feat/asset-file-apis
chore/translations
fix/web-switch-label-clickable
fix/web-people-hidden-state
renovate/typescript-projects
release/next
fix/timezones
fix/time-zone-upserts
midzelis/wip
push-zpwsovysllvn
push-nwxlpmyzkyrl
push-nvnkszuqwppm
renovate/github-actions
push-smstsuupsowp
refactor/adaptive_image
push-olwpzvrxnomt
push-lmxsupnmxspl
renovate/machine-learning
feat/web-chromecast-video-looping
feat/use-native-clients
renovate/flutter
fix/create-face-edited
fix/mobile-ios-mtls
docs/contributing
docs/mise-mobile
renovate/grafana-monorepo
feature/bottom-buttons-order
feat/immich-mobile-ui-showcase
refactor/consolidate-image-requests
renovate/connectivity_plus-7.x
renovate/major-vitest-monorepo
renovate/pypi-python-multipart-vulnerability
fix/mobile-people-query
sqlite_thumbs
feat/html-text
chore/no-macro-validation
refactor/purchase-store
uhthomas/mobile-fix-app-bar-fade
uhthomas/mobile-fix-asset-jump
feat/pano-ocr
feat/shared-link-login
fix/database-backup-db-names
fix-keep-correct-ios-shared-album-asset
fix-memory-generation-and-display
feat/verify-permissions
refactor/album-service-small-tests
fix/ml-rocm-build
fix/flipped-dimensions-mobile
push-vpxwmwwxwnvw
fix-migration-width-height
refactor/more-queries
revert/prettier-translations
refactor/asset-service-queries
fix/locale-settings-desc
chore/add-debug-log
feat/edit-filters
shared-deep-link-handler
feat/mobile-editing
feat/thumbnail-native-clients
feat/platform-clients
feat/integrity-checks-izzy
fix/foreground-cloud-sync
feat/dynamic-layout
filter-by-person
feat/csp
refactor/sidebar
fix/disable-editing
fix/view-timeline-deeplink
image-zoom-on-slow-connection
fix-consider-dar-for-video-dimension
fix/merged-edited-assets
perf/optimize-album-sort
open-api-fix
feat/create-job-with-dto
use-toast-primary
feat/vitest-4
feat/ios-fastlane-match
match-signing
fix-update-time-update-timeline
chore/translation-keys
feat/modal-routes
feat/panorama-tiles
feature/mobile-view-asset-owner
feat/system-settings
feature/show-activity-count
better-info-in-asset-viewer
fix/all-people-count
feat/location-favorites
feature/rearrange-buttons-2
fix/download-storage-template
feat/kb-shortcuts-mobile
fix/people-count
push-qolzzzzxrvvn
chore/originals-in-asset-files
feat/asset-size-columns
ben/tree-a11y
new-search-filter-ui
refactor/expectSelectedReadonly
refactor/mobile-grdb
push-qvuktpxmkknu
feat/mobile-native-local-sync
refactor/timeline_ops
fix/scrubber_end
feat/version.txt
feat/context-menus
feat/server-chunked-uploads
refactor/virtualsegment
refactor/rename_daymonth_groups
fix/restrict-android-bg-worker
feat/android-periodic-worker
fix-remote-sync-clean-up
refactor/timeline_move_ops
renovate/mapbox-mapbox-gl-rtl-text-0.x
fix/timeline_split_selectable
feat/keyboard_actions_help_modal
feat/static_frontend
feat/notification-warnign-android
feat/plugins2
feat/plugins
test/create-workflow-token-action
fix/docs-force
debug/search-result-similarity
debug/cf-chunked-uploads
feat/eslint_rule
feat/search-filter-album/web
refactor/timeline_photostream
refactor/timelineasset_asset
feat/session-permissions
feat/timeline_photostream_assetnav
feat/timeline_minor_optimize
feat/timeline_perf_nocomp
feat/timeline_search_results_actions
feat/timeline_search_results_page
fix/timeline_padding
fix/timeline_search_reactivity_warnings
feat/timeline_scrollbar
feat/timeline_stream_withviewer
fix/timeline_back_forth_nav
refactor/timeline_photostream_component
fix/generated-files-checks
fix/locate-button-local
chore/base-image-mimalloc
refactor/timeline_assetlayout
refactor/timeline_selectable
refactor/timeline_aware_actions
refactor/timeline_monthsegment
feat/remove-old-pages
chore/deps-gradle
tmp_photostream
tmp/lcms
feat/mobile-dynamic-thumbnails
fix/mobile-finer-thumbnail-concurrency
refactor/timeline1
refactor/extract_photostream
refactor/rename_load_api
refactor/timeline2
refactor/timeline3
feat/multi-select-asset-viewer
feat-no-thumbhash-cache
refactor/asset_grid
feat/faster-access-checks
fix/18991
fix/19543
chore/temp-remove
fix/21419
feat/mobile-hdr-images
chore/update-mise-lockfile
feat/mise-server-checks
feat/mise-ci
feat/windows-2025
feat/dev_cli
refactor/mobile-migrate-clients
fix/map-theme
fix/require-checkbox
chore/use_swc
feat/efficient-thumbnail-decoding
refactor/mobile-thumbhash
refactor/mobile-thumbhash-new
fix/mobile-uncached-zoom
feat/beta-background-upload
fix/beta-timeline-memories-setting
fix/failed-uploads-not-removed
feat/mobile-shared-album
feat/groups
drift-map-page
drift-auth-user-sync
fix/disable-memory
feat/add-to-album-action
edit-date-time-action
drift-people-page
sqlite-remove-isIn
feat/inline-storage-columns
chore/required-reviewers
refact/asset-manager
fix/folder-sort
pnpm
feat/widget-multiple-server-urls
chore/medium-tests-dbname
fix/web-no-iterator-find
fix/map-pan-interruption
track-livephotos
timeline_events
chore/oxlint-migration
feat/maintenance-worker
feat/dav
chore/demo-snapshot
refactor/server-side-dedupe
feat/integrity-checks
dev/recognition-eval
lighter_buckets_test
perf/postgres-queue
postgres-queue
focus_rings
refactor/web-stores-1
refactor/add-to-taken
feat/sort-places
feat/sidecar-asset-file
vet
tmp/demo-snapshot-preview
fix/server-migration-file-extension
refactor/mobile-v2
fix/asset-update-race-condition
rknn-toolkit-lite2
refactor/mobile-split-up-search-page
feature/Add-rocm-support-for-machine-learning
feat/rocm
chore/async-hash-file
feat/shared-link-view-count
feat/rotation
feat/graphql
feat/job-ids
feat/ignore-library-permission-error
feat/docker-compose-builder
feat/kysely-typeorm
mobile/onboarding
no-video-player
fix/server-qsv-output-format
chore/server-geodata-tweaks
mobile/native-video-player-no-hero
feat/xxhash
fix/docs-concurrency
feat/preload-ml-textual-model
feat/local-tileserver
refactor/exif-orientation
original-path-infix
refactor/mobile/login-form-1
feat/server-editor-endpoints
fix/server-qsv-vbr
fix-mobile-db-problems
feat/ml-armnn-conversion
feat/mobile/backup-with-album-info
feat/fast-initial-sync-1
chore/handle-output_dims
feat/server-more-robust-generation
feat/unassign-faces
feat/shortcuts-on-asset-grid
feat/background-upload
feat/capacitor-mobile-app-poc
feat/server-nvenc-hw-decoding
release/v1.105
fix/mobile-fetch-non-archive
feat/fine-grained-access-controls
web/automation-ui
feat/mobile-server-endpoint-save-dropdown
feat/blurhash-thumbnail
object-storage
feat/memories-animations
dev/metrics
ml/tflite
feat/ml-export-cli
v2.5.3
v2.5.2
v2.5.1
v2.5.0
v2.4.1
v2.4.0
v2.3.1
v2.3.0
v2.2.3
v2.2.2
v2.2.1
v2.2.0
v2.1.0
v2.0.1
v2.0.0
v1.144.1
v1.144.0
v1.143.1
v1.143.0
v1.142.1
v1.142.0
v1.141.1
v1.141.0
v1.140.1
v1.140.0
v1.139.4
v1.139.3
v1.139.2
v1.139.1
v1.139.0
v1.138.1
v1.138.0
v1.137.3
v1.137.2
v1.137.1
v1.137.0
v1.136.0
v1.135.3
v1.135.2
v1.135.1
v1.135.0
v1.134.0
v1.133.1
v1.133.0
v1.132.3
v1.132.2
v1.132.1
v1.132.0
v1.131.3
v1.131.2
v1.131.1
v1.131.0
v1.130.3
v1.130.2
v1.130.1
v1.130.0
v1.129.0
v1.128.0
v1.127.0
v1.126.1
v1.126.0
v1.125.7
v1.125.6
v1.125.5
v1.125.4
v1.125.3
v1.125.2
v1.125.1
v1.125.0
v1.124.2
v1.124.1
v1.124.0
v1.123.0
v1.122.3
v1.122.2
v1.122.1
v1.122.0
v1.121.0
v1.120.2
v1.120.1
v1.120.0
v1.119.1
v1.119.0
v1.118.2
v1.118.1
v1.118.0
v1.117.0
v1.116.2
v1.116.1
v1.116.0
v1.115.0
v1.114.0
v1.113.1
v1.113.0
v1.112.1
v1.112.0
v1.111.0
v1.110.0
v1.109.2
v1.109.1
v1.109.0
v1.108.0
v1.107.2
v1.107.1
v1.107.0
v1.106.4
v1.106.3
v1.106.2
v1.106.1
v1.106.0
v1.105.1
v1.105.0
v1.104.0
v1.103.1
v1.103.0
v1.102.3
v1.102.2
v1.102.1
v1.102.0
v1.101.0
v1.100.0
v1.99.0
v1.98.2
v1.98.1
v1.98.0
v1.97.0
v1.96.0
v1.95.1
v1.95.0
v1.94.1
v1.94.0
v1.93.3
v1.93.2
v1.93.1
v1.93.0
v1.92.1
v1.92.0
v1.91.4
v1.91.3
v1.91.2
v1.91.1
v1.91.0
v1.90.2
v1.90.1
v1.90.0
v1.89.0
v1.88.2
v1.88.1
v1.88.0
v1.87.0
v1.86.0
v1.85.0
v1.84.0
v1.83.0
v1.82.1
v1.82.0
v1.81.1
v1.81.0
v1.80.0
v1.79.1
v1.79.0
v1.78.1
v1.78.0
v1.77.0
v1.76.1
v1.76.0
v1.75.2
v1.75.1
v1.75.0
v1.74.0
v1.73.0
v1.72.2
v1.72.1
v1.72.0
v1.71.0
v1.70.0
v1.69.0
v1.68.0
v1.67.2
v1.67.1
v1.67.0
v1.66.1
v1.66.0
v1.65.0
v1.64.0
v1.63.2
v1.63.1
v1.63.0
v1.62.1
v1.62.0
v1.61.0
v1.60.0
v1.59.1
v1.59.0
v1.58.0
v1.57.1
v1.57.0
v1.56.2
v1.56.1
v1.56.0
v1.55.1
v1.55.0
v1.54.1
v1.54.0
v1.53.0
v1.52.1
v1.52.0
v1.51.2
v1.51.1
v1.51.0
v1.50.1
v1.50.0
v1.49.0
v1.48.1
v1.48.0
v1.47.3
v1.47.2
v1.47.1
v1.47.0
v1.46.1
v1.46.0
v1.45.0
v1.44.0
v1.43.1
v1.43.0
v1.42.0_65-dev
v1.41.1_64-dev
v1.41.0_64-dev
v1.40.1_63-dev
v1.40.0_63-dev
v1.39.0_61-dev
v1.38.2_60-dev
v1.38.1_60-dev
v1.38.0_60-dev
v1.37.0_58-dev
v1.36.2_56-dev
v1.36.1_55-dev
v1.36.0_55-dev
v1.35.0_54-dev
v1.34.0_53-dev
v1.33.1_52-dev
v1.33.0_52-dev
v1.32.1_51-dev
v1.32.0_50-dev
v1.31.1_49-dev
v1.31.0_49-dev
v1.30.2_48-dev
v1.30.0_46-dev
v1.29.6_45-dev
v1.29.6_44-dev
v1.29.5_44-dev
v1.29.4_44-dev
v1.29.3_43-dev
v1.29.2_43-dev
v1.29.1_43-dev
v1.29.0_42-dev
v1.28.4_41-dev
v1.28.4_42-dev
v1.28.3_41-dev
v1.28.2_40-dev
v1.28.1_39-dev
v1.28.0_38-dev
v1.27.0_37-dev
v1.26.0_36-dev
v1.25.0_35-dev
v1.24.0_34-dev
v1.23.0_33-dev
v1.22.0_32-dev
v1.21.1_31-dev
v1.21.0_31-dev
v1.20.3_30-dev
v1.20.2_30-dev
v1.20.1_30-dev
v1.20.0_30-dev
v1.19.1_29-dev
v1.19.0_29-dev
v1.18.0_27-dev
v1.17.0_25-dev
v1.16.0_23-dev
v1.15.1_21-dev
v1.15.0_21-dev
v1.14.0_21-dev
v1.13.0_20-dev
v1.12.0_18-dev
v1.11.0_17-dev
v1.10.0_15-dev
v1.9.1_14-dev
v1.9.0_13-dev
v1.8.0_12-dev
v1.7.0_11-dev
v1.6.0_10-dev
v1.5.1+9-dev
v1.5.0+8-dev
v1.4.0+7-dev
v1.4.0+6-dev
v1.4.0-dev
v1.3.0-dev
v1.3.1-dev
v0.6-dev
v0.5-dev
v0.4-dev
v0.3-dev
v0.2-dev
first-android-release
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: immich-app/immich#898
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zenithyr on GitHub (May 30, 2023).
The bug
The application worked perfectly fine in docker until yesterday when I upgraded to the latest with the added typesense feature. After
docker pull & docker compose up -d, the web portal now complains "Request failed with status code 503".Tried to use the latest yaml and .env files with no luck. Commenting out typesense from docker-compose.yml and .env completely resulted the same error.
UPLOAD_LOCATION does point to a folder filled with data from previously. Not sure if it's related.
Please find container logs in the sections below.
The OS that Immich Server is running on
OpenMediaVault 6.1.0-0.deb11.7-amd64
Version of Immich Server
ghcr.io/immich-app/immich-server:release
Version of Immich Mobile App
not used here
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
3.
docker compose up -dcurl http://127.0.0.1:2283 # or from remotely by http://<ip>:2283server
microservers
web
typesense
@zenithyr commented on GitHub (May 30, 2023):
curl http://<ip>:2283/api/server-info/statsreturns@alextran1502 commented on GitHub (May 30, 2023):
Can you put in the
.envfileLOG_LEVEL=debugand see if you can pull additional information from the server container? It looks like the server cannot start up because, somehow the docker network in your stack cannot communicate with Typesense.Is it your standard procedure to perform
docker compose downthendocker volume prune?@alextran1502 commented on GitHub (May 30, 2023):
Is it the same data though?
@zenithyr commented on GitHub (May 30, 2023):
Thanks for the quick turnaround.
echo "LOG_LEVEL=debug" >> .env. Then ensured the line is present in the file.tail -n5 .envdocker compose down; docker compose up -ddocker ps | grep immich-serverdocker logs 3bc421247963seems DEBUG level isn't working. Anything I was missing?
No, just this once in order to have a clean volume. I usually do
docker compose downfollowed bydocker compose up -d@dhanadhan commented on GitHub (May 30, 2023):
Same, well not really. I am getting 502 bad gateway after upgrading to 1.58.0. Re-pulled the new image and changed the docker-compose lines as mentioned in the release note, but now the web page won't load nor the android app would load my server url :(
Edit: my status page correctly shows the content info, but not loading the web page even with localip:2283
My issue is probably isolated, but not sure what it is after upgrading to 1.58 from 1.57.
Edit 2: solved it by adding the line 45 "entrypoint: ["/bin/sh", "./entrypoint.sh"]" back in to docker-compose.
@zenithyr commented on GitHub (May 30, 2023):
Yes. But in case it's the culprit, I changed it to a new path, restarted the container, and got the same error :( After restart I can see the new directory being created with nothing in it.
Please instruct how to print out more logs, thanks.
@alextran1502 commented on GitHub (May 30, 2023):
@dhanadhan Have you read the release note to make the appropriate changes to the docker-compose file since we changed a few things in it?
@dhanadhan commented on GitHub (May 30, 2023):
Yes, there were 3 changes. It didn't load the web page. But I have just resolved it by adding the line 45 back in.
@alextran1502 commented on GitHub (May 30, 2023):
@dhanadhan It wasn't the intention to keep that line and have the container working, it looks like you haven't pulled the new version?
@zenithyr commented on GitHub (May 30, 2023):
In case it helps, this is the list of containers. They all look normal.
@alextran1502 commented on GitHub (May 30, 2023):
@zenithyr Can you remove the
loggingdirective in the typesense container in the docker-compose file? Your typesense container has the issue.FYI, after using
docker compose downit could remove all the containers so that when you performdocker volume prune.This doesn't look normal as the typesense and redis container doesn't get restart with the same timestamp as other containers. Please bring all the containers down, and make sure that typesense and redis get brought down as well so when you bring up the whole stack, they would be able to communicate with each other.
@dhanadhan commented on GitHub (May 30, 2023):
Pretty sure i pulled the new image. Web page shows v1.58.0 correctly on the bottom left and working as of now with having line 45 "entrypoint: ["/bin/sh", "./entrypoint.sh"]" in the docker compose.
Do I keep this change? It will probably stop working if I remove it.
@zenithyr commented on GitHub (May 30, 2023):
The issue persists :(.
typesense logs looks like below
@alextran1502 commented on GitHub (May 30, 2023):
@zenithyr What is the log now for the server? Maybe try restart your system?
@zenithyr commented on GitHub (May 30, 2023):
Same as before
I then tried to set
TYPESENSE_ENABLED=falsein the.envfile, and commented out typesense from docker-compose completely. Down the stack and up again. I saw no error from the serverBut this time, while I still got the same 503 error by
curl http://<ip>:2283, I got some response oncurl http://<ip>:2283/api/server-info/stats:{"statusCode":401,"message":"Authentication required","error":"Unauthorized"}So is it safe to assume the 503 isn't related to typesense?
@alextran1502 commented on GitHub (May 30, 2023):
Okay, why don't we do this, please follow exactly the commands below.
Then let me know if all the logs look good.
Ah I notice you did
docker pullwhich is not the correct command to pull updates for all containers. It has to bedocker compose pull@zenithyr commented on GitHub (May 30, 2023):
Based on the previous configurations, with typesense disabled:
user@omv ~/d/immich> docker compose down
user@omv ~/d/immich> docker container prune
user@omv ~/d/immich> docker compose pull
Pull. It failed the first time so I retried.
user@omv ~/d/immich> docker compose up
And then refresh the webpage or curl, got nothing but that status code 503...
@jrasm91 commented on GitHub (May 30, 2023):
Your
immich-serveris starting up fine. Looking at the logs it looks like theimmich-proxyis started beforeimmich-serveris ready to accept connections, which is why you get a503. Just restart the proxy container (after everything else has started) and you should be good to go.@zenithyr commented on GitHub (May 30, 2023):
Thanks for jumping in. I tried the following.
docker psdocker container restart dc12bee5c3f5docker psSo the proxy is indeed restarted. And that 503 still didn't go away....
@jrasm91 commented on GitHub (May 30, 2023):
What are the latest immich-web logs and what hardware are you running this on?
@zenithyr commented on GitHub (May 30, 2023):
I am running this on top of the OpenMediaVault 6.1.0-0.deb11.7-amd64, which sits on PromoxVE. Memory usage 8.7G/24G. The UPLOAD_LOCATION is on a RAID-0 ZFS drive.
There are some other docker containers like qBitTorrent, PhotoView running fine so far.
Things I can think of:
One thing to note is that I have set up http proxy for docker server
But even if I commented out above settings and run
sudo systemctl daemon-reload;sudo systemctl restart docker. The 503 error is still there.Before the version upgrade (I forgot the old version), everything went on pretty well. And there hasn't been any hardware changes in between.
@alextran1502 commented on GitHub (May 30, 2023):
Can you please try to setup a new instance of Immich using stock setup? I have a feeling your proxy settings cause the issue
@zenithyr commented on GitHub (May 30, 2023):
Sure. I dropped all proxy entries in the http-proxy.conf file.
restarted by docker compose down and up -d. Then
Seems nobody else ran into similar issues that I am having. I will try to revert to an older version later. Will keep you posted.
Please also let me know if you have anything else to try. Thanks.
@jrasm91 commented on GitHub (May 30, 2023):
Where is the 503 error coming from? You should see it in the web logs, proxy logs, or server logs. If those are clean then it is coming from something in front of immich-proxy.
@zenithyr commented on GitHub (May 30, 2023):
Older version didn't work either. So I dropped everything including images, and start refresh from
docker compose pull.Then I follow the guidance by
docker logs immich_server --follow.After the whole stack is up, even before I make any web request, the server keeps printing out errors, and ends with something below
I wonder if it's possible to find what that mystery request is so I can reproduce easier by go into the running container. Any hint is welcome.
@jrasm91 commented on GitHub (May 30, 2023):
This isn't a mystery request, this is the server trying to connect to typesense on start up.
@zenithyr commented on GitHub (May 31, 2023):
I tested the same set up on a brand new virtual host and there is no problem. Closing the ticket for now.
@zenithyr commented on GitHub (Jun 1, 2023):
OK I found it. @alextran1502 is correct, the 503 comes from the proxy.
~/.docker/config.json/etc/systemd/system/docker.service.d/http-proxy.confThanks to @jrasm91 who helped look into it!
@qrzbing commented on GitHub (Oct 9, 2023):
I met same problem too, and immich works after disabled proxy. How can I let proxy and immich work together?
@AndyXheli commented on GitHub (Nov 22, 2023):
Hey @alextran1502 Im starting to see this on v1.88.2 i updated from v1.88.1 and now seem to take a bit for th server to come up. Then after a 1 or 2 web ui starts to work
@AndyXheli commented on GitHub (Nov 22, 2023):
Heres what i see after starting up the docker
@alextran1502 commented on GitHub (Nov 22, 2023):
@AndyXheli Typesense on start-up would index the library to serve the quick search purpose, which might take a few minutes depending on the size of your gallery
@AndyXheli commented on GitHub (Nov 22, 2023):
Ah okay! yes i have about 300GB worth of data