immich_postgres server hanging on INSERT (not SELECT) #6436

New Issue

OVERLORD · 2026-02-05T12:17:25+03:00

OVERLORD commented

2026-02-05 12:17:25 +03:00

Originally created by @pjrobertson on GitHub (Jul 6, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

Yes

The bug

Been using the standard docker-compose file for a while, and upgrading normally with docker compose pull && docker compose up -d

Now, the immich server container cannot start and doesn't seem to be listening on 2283 at all (I've tried multiple other ports). I've debugged this down to a hang on the postgres database on INSERT.

If I bypass the handleVersionCheck() code with just:

return JobStatus.SKIPPED

it bypasses the the version check (postgres SET) and the nest server starts, but of course it throws errors:

immich_postgres          | 2025-07-07 08:03:47.700 UTC [60] ERROR:  MultiXactId 252 has not been created yet -- apparent wraparound
immich_postgres          | 2025-07-07 08:03:47.700 UTC [60] STATEMENT:  select "users"."id" from "users" where "users"."isAdmin" = $1 and "users"."deletedAt" is null
immich_server            | Query failed : {
immich_server            |   durationMs: 11.193733999998585,
immich_server            |   error: PostgresError: MultiXactId 252 has not been created yet -- apparent wraparound
immich_server            |       at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
immich_server            |       at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
immich_server            |       at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich_server            |       at Socket.emit (node:events:518:28)
immich_server            |       at addChunk (node:internal/streams/readable:561:12)
immich_server            |       at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich_server            |       at Readable.push (node:internal/streams/readable:392:5)
immich_server            |       at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich_server            |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich_server            |     severity_local: 'ERROR',
immich_server            |     severity: 'ERROR',
immich_server            |     code: 'XX000',
immich_server            |     file: 'multixact.c',
immich_server            |     line: '1305',
immich_server            |     routine: 'GetMultiXactIdMembers'
immich_server            |   },
immich_server            |   sql: 'select "users"."id" from "users" where "users"."isAdmin" = $1 and "users"."deletedAt" is null',
immich_server            |   params: [ true ]
immich_server            | }

The OS that Immich Server is running on

Ubuntu 24.04

Version of Immich Server

immich-server:release

Version of Immich Mobile App

N/A

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

#
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:fec42f399876eb6faf9e008570597741c87ff7662a54185593e74b09ce83d177
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/London

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
IMMICH_LOG_LEVEL=verbose

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=XXXXXXXXX

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

Running immich for 6+ months, upgrading with docker compose pull && docker compose up -d
One day could not connect to immich server
Debugged for 2+ hours. Postgres is running fine (DB is OK), redis is running fine (ping -> PONG).
Seems that the server just 'hangs' whilst starting up
...

Relevant log output

root@nl:~/immich-app# docker compose up
[+] Running 4/4
 ✔ Container immich_redis             Created                                                                                                                                                                    0.0s 
 ✔ Container immich_machine_learning  Created                                                                                                                                                                    0.0s 
 ✔ Container immich_postgres          Created                                                                                                                                                                    0.0s 
 ✔ Container immich_server            Created                                                                                                                                                                    0.0s 
Attaching to immich_machine_learning, immich_postgres, immich_redis, immich_server
immich_postgres          | Using SSD storage
immich_redis             | 1:M 06 Jul 2025 10:48:50.371 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 10:48:50.371 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 10:48:50.371 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 10:48:50.371 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 10:48:50.373 * Increased maximum number of open files to 10032 (it was originally set to 1024).
immich_redis             | 1:M 06 Jul 2025 10:48:50.374 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 10:48:50.374 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 10:48:50.375 * Server initialized
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * RDB age 240 seconds
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * RDB memory usage when created 1.47 Mb
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * Done loading RDB, keys loaded: 16, keys expired: 8.
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * DB loaded from disk: 0.000 seconds
immich_redis             | 1:M 06 Jul 2025 10:48:50.377 * Ready to accept connections tcp
immich_postgres          | 
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          | 
immich_postgres          | 2025-07-06 10:48:50.503 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 10:48:50.503 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 10:48:50.540 UTC [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres          | 2025-07-06 10:48:50.542 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres          | 2025-07-06 10:48:50.542 UTC [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres          | 2025-07-06 10:48:50.546 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres          | 2025-07-06 10:48:50.551 UTC [30] LOG:  database system was interrupted; last known up at 2025-07-06 10:43:33 UTC
immich_postgres          | 2025-07-06 10:48:50.699 UTC [30] LOG:  database system was not properly shut down; automatic recovery in progress
immich_postgres          | 2025-07-06 10:48:50.704 UTC [30] LOG:  redo starts at 0/5AF402C8
immich_postgres          | 2025-07-06 10:48:50.704 UTC [30] LOG:  invalid record length at 0/5AF40300: wanted 24, got 0
immich_postgres          | 2025-07-06 10:48:50.704 UTC [30] LOG:  redo done at 0/5AF402C8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
immich_postgres          | 2025-07-06 10:48:50.734 UTC [1] LOG:  database system is ready to accept connections
immich_server            | Initializing Immich v1.135.3
immich_server            | DEBUG: cgroup v2 detected.
immich_server            | DEBUG: No CPU limits set.
immich_server            | Detected CPU Cores: 4
immich_machine_learning  | [07/06/25 11:48:51] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [07/06/25 11:48:51] INFO     Listening at: http://0.0.0.0:3003 (8)              
immich_machine_learning  | [07/06/25 11:48:51] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [07/06/25 11:48:51] INFO     Booting worker with pid: 9                         
immich_machine_learning  | [07/06/25 11:48:52] DEBUG    Could not load ANN shared libraries, using ONNX:   
immich_machine_learning  |                              libmali.so: cannot open shared object file: No such
immich_machine_learning  |                              file or directory                                  
immich_machine_learning  | [07/06/25 11:48:52] DEBUG    RKNN is not available                              
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_machine_learning  | [07/06/25 11:48:53] INFO     Started server process [9]                         
immich_machine_learning  | [07/06/25 11:48:53] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 11:48:53] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 11:48:53] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 11:48:53] INFO     Application startup complete.                      
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:EventRepository] Initialized websocket server
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:DatabaseRepository] targetLists=1, current=1 for clip_index of 998 rows
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:DatabaseRepository] targetLists=1, current=1 for face_index of 5988 rows
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:EventRepository] Initialized websocket server
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:MetadataService] Bootstrapping metadata service
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:MetadataService] Initializing metadata service
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:MapRepository] Initializing metadata repository
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:MetadataService] Initialized local reverse geocoder
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:ServerService] Feature Flags: {
immich_server            |   "smartSearch": true,
immich_server            |   "facialRecognition": true,
immich_server            |   "duplicateDetection": true,
immich_server            |   "map": true,
immich_server            |   "reverseGeocoding": true,
immich_server            |   "importFaces": false,
immich_server            |   "sidecar": true,
immich_server            |   "search": true,
immich_server            |   "trash": true,
immich_server            |   "oauth": false,
immich_server            |   "oauthAutoLaunch": false,
immich_server            |   "passwordLogin": true,
immich_server            |   "configFile": false,
immich_server            |   "email": false
immich_server            | }
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}}
immich_server            | [Nest] 7  - 07/06/2025, 11:48:55 AM     LOG [Microservices:StorageService] Successfully verified system mount folder checks
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:DatabaseRepository] targetLists=1, current=1 for clip_index of 998 rows
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:DatabaseRepository] targetLists=1, current=1 for face_index of 5988 rows
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:DatabaseRepository] Running migrations, this may take a while
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:ServerService] Feature Flags: {
immich_server            |   "smartSearch": true,
immich_server            |   "facialRecognition": true,
immich_server            |   "duplicateDetection": true,
immich_server            |   "map": true,
immich_server            |   "reverseGeocoding": true,
immich_server            |   "importFaces": false,
immich_server            |   "sidecar": true,
immich_server            |   "search": true,
immich_server            |   "trash": true,
immich_server            |   "oauth": false,
immich_server            |   "oauthAutoLaunch": false,
immich_server            |   "passwordLogin": true,
immich_server            |   "configFile": false,
immich_server            |   "email": false
immich_server            | }
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}}
immich_server            | [Nest] 17  - 07/06/2025, 11:48:55 AM     LOG [Api:StorageService] Successfully verified system mount folder checks
// THEN NOTHING... no matter how long I leave it, no further logs

Additional information

Here's some other relevant info:

docker ps:

CONTAINER ID   IMAGE                                                            COMMAND                  CREATED         STATUS                                 PORTS                                         NAMES
c7f0dd665bf6   ghcr.io/immich-app/immich-server:release                         "tini -- /bin/bash s…"   6 minutes ago   Up About a minute (health: starting)   0.0.0.0:2283->2283/tcp, [::]:2283->2283/tcp   immich_server
a3fb5293c16c   valkey/valkey:8-bookworm                                         "docker-entrypoint.s…"   6 minutes ago   Up About a minute (healthy)            6379/tcp                                      immich_redis
49ea768432b8   ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0   "/usr/local/bin/immi…"   6 minutes ago   Up About a minute (healthy)            5432/tcp                                      immich_postgres
30dfe464d1ae   ghcr.io/immich-app/immich-machine-learning:release               "tini -- python -m i…"   6 minutes ago   Up About a minute (healthy)                                                          immich_machine_learning

docker exec -it immich_server /bin/bash:

$ netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.11:44941        0.0.0.0:*               LISTEN      -    

$ curl -v http://localhost:2283
curl: (7) Failed to connect to localhost port 2283 after 0 ms: Couldn't connect to server

docker exec -it immich_server ps aux (after installing ps) -> shows that the two processes are running :/

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2572  1280 ?        Ss   10:42   0:00 tini -- /bin/bash start.sh
root           6  0.7  4.6 12308080 277972 ?     Sl   10:42   0:07 immich
root          17  0.3  3.1 11669172 186668 ?     Sl   10:42   0:04 immich-api
root          40  0.0  0.0   4408  3584 pts/0    Ss   10:42   0:00 /bin/bash

Originally created by @pjrobertson on GitHub (Jul 6, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug Been using the standard docker-compose file for a while, and upgrading normally with `docker compose pull && docker compose up -d` Now, the immich server container cannot start and doesn't seem to be listening on 2283 at all (I've tried multiple other ports). I've debugged this down to a hang on the postgres database on INSERT. If I bypass the handleVersionCheck() code with just: ``` return JobStatus.SKIPPED ``` it bypasses the the version check (postgres SET) and the nest server starts, but of course it throws errors: ``` immich_postgres | 2025-07-07 08:03:47.700 UTC [60] ERROR: MultiXactId 252 has not been created yet -- apparent wraparound immich_postgres | 2025-07-07 08:03:47.700 UTC [60] STATEMENT: select "users"."id" from "users" where "users"."isAdmin" = $1 and "users"."deletedAt" is null immich_server | Query failed : { immich_server | durationMs: 11.193733999998585, immich_server | error: PostgresError: MultiXactId 252 has not been created yet -- apparent wraparound immich_server | at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26) immich_server | at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6) immich_server | at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) immich_server | at Socket.emit (node:events:518:28) immich_server | at addChunk (node:internal/streams/readable:561:12) immich_server | at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) immich_server | at Readable.push (node:internal/streams/readable:392:5) immich_server | at TCP.onStreamRead (node:internal/stream_base_commons:189:23) immich_server | at TCP.callbackTrampoline (node:internal/async_hooks:130:17) { immich_server | severity_local: 'ERROR', immich_server | severity: 'ERROR', immich_server | code: 'XX000', immich_server | file: 'multixact.c', immich_server | line: '1305', immich_server | routine: 'GetMultiXactIdMembers' immich_server | }, immich_server | sql: 'select "users"."id" from "users" where "users"."isAdmin" = $1 and "users"."deletedAt" is null', immich_server | params: [ true ] immich_server | } ``` ### The OS that Immich Server is running on Ubuntu 24.04 ### Version of Immich Server immich-server:release ### Version of Immich Mobile App N/A ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose # # Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - '2283:2283' depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env restart: always healthcheck: disable: false redis: container_name: immich_redis image: docker.io/valkey/valkey:8-bookworm@sha256:fec42f399876eb6faf9e008570597741c87ff7662a54185593e74b09ce83d177 healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs # DB_STORAGE_TYPE: 'HDD' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data restart: always volumes: model-cache: ``` ### Your .env content ```Shell # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The location where your database files are stored DB_DATA_LOCATION=./postgres # To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List TZ=Europe/London # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release IMMICH_LOG_LEVEL=verbose # Connection secret for postgres. You should change it to a random password # Please use only the characters `A-Za-z0-9`, without special characters or spaces DB_PASSWORD=XXXXXXXXX # The values below this line do not need to be changed ################################################################################### DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` ### Reproduction steps 1. Running immich for 6+ months, upgrading with docker compose pull && docker compose up -d 2. One day could not connect to immich server 3. Debugged for 2+ hours. Postgres is running fine (DB is OK), redis is running fine (ping -> PONG). 4. Seems that the server just 'hangs' whilst starting up ... ### Relevant log output ```shell root@nl:~/immich-app# docker compose up [+] Running 4/4 ✔ Container immich_redis Created 0.0s ✔ Container immich_machine_learning Created 0.0s ✔ Container immich_postgres Created 0.0s ✔ Container immich_server Created 0.0s Attaching to immich_machine_learning, immich_postgres, immich_redis, immich_server immich_postgres | Using SSD storage immich_redis | 1:M 06 Jul 2025 10:48:50.371 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 10:48:50.371 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 10:48:50.371 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 10:48:50.371 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 10:48:50.373 * Increased maximum number of open files to 10032 (it was originally set to 1024). immich_redis | 1:M 06 Jul 2025 10:48:50.374 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 10:48:50.374 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 10:48:50.375 * Server initialized immich_redis | 1:M 06 Jul 2025 10:48:50.377 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 10:48:50.377 * RDB age 240 seconds immich_redis | 1:M 06 Jul 2025 10:48:50.377 * RDB memory usage when created 1.47 Mb immich_redis | 1:M 06 Jul 2025 10:48:50.377 * Done loading RDB, keys loaded: 16, keys expired: 8. immich_redis | 1:M 06 Jul 2025 10:48:50.377 * DB loaded from disk: 0.000 seconds immich_redis | 1:M 06 Jul 2025 10:48:50.377 * Ready to accept connections tcp immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2025-07-06 10:48:50.503 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 10:48:50.503 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 10:48:50.540 UTC [1] LOG: starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2025-07-06 10:48:50.542 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2025-07-06 10:48:50.542 UTC [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2025-07-06 10:48:50.546 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2025-07-06 10:48:50.551 UTC [30] LOG: database system was interrupted; last known up at 2025-07-06 10:43:33 UTC immich_postgres | 2025-07-06 10:48:50.699 UTC [30] LOG: database system was not properly shut down; automatic recovery in progress immich_postgres | 2025-07-06 10:48:50.704 UTC [30] LOG: redo starts at 0/5AF402C8 immich_postgres | 2025-07-06 10:48:50.704 UTC [30] LOG: invalid record length at 0/5AF40300: wanted 24, got 0 immich_postgres | 2025-07-06 10:48:50.704 UTC [30] LOG: redo done at 0/5AF402C8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s immich_postgres | 2025-07-06 10:48:50.734 UTC [1] LOG: database system is ready to accept connections immich_server | Initializing Immich v1.135.3 immich_server | DEBUG: cgroup v2 detected. immich_server | DEBUG: No CPU limits set. immich_server | Detected CPU Cores: 4 immich_machine_learning | [07/06/25 11:48:51] INFO Starting gunicorn 23.0.0 immich_machine_learning | [07/06/25 11:48:51] INFO Listening at: http://0.0.0.0:3003 (8) immich_machine_learning | [07/06/25 11:48:51] INFO Using worker: immich_ml.config.CustomUvicornWorker immich_machine_learning | [07/06/25 11:48:51] INFO Booting worker with pid: 9 immich_machine_learning | [07/06/25 11:48:52] DEBUG Could not load ANN shared libraries, using ONNX: immich_machine_learning | libmali.so: cannot open shared object file: No such immich_machine_learning | file or directory immich_machine_learning | [07/06/25 11:48:52] DEBUG RKNN is not available immich_server | Starting api worker immich_server | Starting microservices worker immich_machine_learning | [07/06/25 11:48:53] INFO Started server process [9] immich_machine_learning | [07/06/25 11:48:53] INFO Waiting for application startup. immich_machine_learning | [07/06/25 11:48:53] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 11:48:53] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 11:48:53] INFO Application startup complete. immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:EventRepository] Initialized websocket server immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:DatabaseRepository] targetLists=1, current=1 for clip_index of 998 rows immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:DatabaseRepository] targetLists=1, current=1 for face_index of 5988 rows immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:EventRepository] Initialized websocket server immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:MetadataService] Bootstrapping metadata service immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:MetadataService] Initializing metadata service immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:MapRepository] Initializing metadata repository immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:MetadataService] Initialized local reverse geocoder immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:ServerService] Feature Flags: { immich_server | "smartSearch": true, immich_server | "facialRecognition": true, immich_server | "duplicateDetection": true, immich_server | "map": true, immich_server | "reverseGeocoding": true, immich_server | "importFaces": false, immich_server | "sidecar": true, immich_server | "search": true, immich_server | "trash": true, immich_server | "oauth": false, immich_server | "oauthAutoLaunch": false, immich_server | "passwordLogin": true, immich_server | "configFile": false, immich_server | "email": false immich_server | } immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}} immich_server | [Nest] 7 - 07/06/2025, 11:48:55 AM LOG [Microservices:StorageService] Successfully verified system mount folder checks immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:DatabaseRepository] targetLists=1, current=1 for clip_index of 998 rows immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:DatabaseRepository] targetLists=1, current=1 for face_index of 5988 rows immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:DatabaseRepository] Running migrations, this may take a while immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:ServerService] Feature Flags: { immich_server | "smartSearch": true, immich_server | "facialRecognition": true, immich_server | "duplicateDetection": true, immich_server | "map": true, immich_server | "reverseGeocoding": true, immich_server | "importFaces": false, immich_server | "sidecar": true, immich_server | "search": true, immich_server | "trash": true, immich_server | "oauth": false, immich_server | "oauthAutoLaunch": false, immich_server | "passwordLogin": true, immich_server | "configFile": false, immich_server | "email": false immich_server | } immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}} immich_server | [Nest] 17 - 07/06/2025, 11:48:55 AM LOG [Api:StorageService] Successfully verified system mount folder checks // THEN NOTHING... no matter how long I leave it, no further logs ``` ### Additional information Here's some other relevant info: docker ps: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c7f0dd665bf6 ghcr.io/immich-app/immich-server:release "tini -- /bin/bash s…" 6 minutes ago Up About a minute (health: starting) 0.0.0.0:2283->2283/tcp, [::]:2283->2283/tcp immich_server a3fb5293c16c valkey/valkey:8-bookworm "docker-entrypoint.s…" 6 minutes ago Up About a minute (healthy) 6379/tcp immich_redis 49ea768432b8 ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 "/usr/local/bin/immi…" 6 minutes ago Up About a minute (healthy) 5432/tcp immich_postgres 30dfe464d1ae ghcr.io/immich-app/immich-machine-learning:release "tini -- python -m i…" 6 minutes ago Up About a minute (healthy) immich_machine_learning ``` `docker exec -it immich_server /bin/bash`: ``` $ netstat -tlnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.11:44941 0.0.0.0:* LISTEN - $ curl -v http://localhost:2283 curl: (7) Failed to connect to localhost port 2283 after 0 ms: Couldn't connect to server ``` `docker exec -it immich_server ps aux` (after installing ps) -> shows that the two processes are running :/ ``` USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2572 1280 ? Ss 10:42 0:00 tini -- /bin/bash start.sh root 6 0.7 4.6 12308080 277972 ? Sl 10:42 0:07 immich root 17 0.3 3.1 11669172 186668 ? Sl 10:42 0:04 immich-api root 40 0.0 0.0 4408 3584 pts/0 Ss 10:42 0:00 /bin/bash ```

OVERLORD closed this issue

2026-02-05 12:17:25 +03:00

OVERLORD commented

2026-02-05 12:17:26 +03:00

@pjrobertson commented on GitHub (Jul 7, 2025):

OK, so this was a PITA to debug because setting the debug level in either the .env or immich.json file didn't help. I ended up having to go into the immich_server docker file and mess around with the files in main.js.

Anyway, I managed to pinpoint the hang down to the handleVersionCheck() call in VersionService. Turns out that the this.systemMetadataRepository.set() call was hanging...

The database is hanging on INSERTs but not SELECTs - they are working fine. This is the same if I do psql -U postgres -W immich and then run insert into system_metadata values('version-check-state', '{ "checkedAt": "2025-07-07T07:29:46.311Z", "releaseVersion": "v1.135.3" }'); - it just hangs.

Still trying to debug what exactly is wrong with the database.

@pjrobertson commented on GitHub (Jul 7, 2025): OK, so this was a PITA to debug because setting the debug level in either the .env or immich.json file didn't help. I ended up having to go into the immich_server docker file and mess around with the files in main.js. Anyway, I managed to pinpoint the hang down to the `handleVersionCheck()` call in `VersionService`. Turns out that the `this.systemMetadataRepository.set()` call was hanging... The database is hanging on INSERTs but not SELECTs - they are working fine. This is the same if I do `psql -U postgres -W immich` and then run `insert into system_metadata values('version-check-state', '{ "checkedAt": "2025-07-07T07:29:46.311Z", "releaseVersion": "v1.135.3" }');` - it just hangs. Still trying to debug what exactly is wrong with the database.

OVERLORD referenced this issue

2026-02-05 14:34:58 +03:00

[PR #6436] [MERGED] chore(deps): pin dependencies #11187

Sign in to join this conversation.

Branches Tags

main

chore/translations

fix/partner-ownership-contention

renovate/machine-learning

renovate/pypi-pillow-vulnerability

renovate/pypi-pydantic-settings-vulnerability

renovate/java-25.x

renovate/mobile

chore/harden-mobile-openapi-gen

renovate/java-21.x

fix/min-faces

refactor/asset-update

feat/merge-people-mobile

chore/face-cluster-abstraction

refactor/search-v3/pr-2a

fcast

fix/android-local-album-date

fix/24545-sync-retry-cx

fix/android-locked-folder-delete

make-sure-memory-widget-render

chore/freezed-example

fix/clamp-local-date-time-to-valid-value

fix/datetime-max-limit

fix/viewer-long-image-aspect

feat/native-core

feat/yucca-integration

refactor/mobile-actions

fix/ws-reconnect-battery

renovate/sharp

renovate/pigeon-27.x

refactor/auto-trash-sync

feat/release-source-branch

refactor/session-repository

fix/library-map-live-bounds-updates

renovate/permission_handler-12.x

renovate/node-24.x

fix/mobile-scroll-velocity-placeholders

fix/library-map-loading-flash

agg23/appbar-experiments

renovate/pep621-lock-file-maintenance-machine-learning

refactor/undo-actions

refactor/actions

feat/restore-action

feat/delete-action

fix/29853-thumbnail-decode-size

bugfix/live-photo-stuck

feature/load-previews

chore/pub-upgrade

custom-app-logo

fix/failed-sync-blocks-resume

feat/ml-hwaccel-testing

feat/workflow-logs

feat/server-chunked-uploads

feat/upload-action

fix-29727

feat/download-tag-action

feat/share-action

feat/album-actions

fix/android-cr3-save

feat/browser-profile-action

feat/cast-slideshow-action

feat/locked-folder-action

fix/unauthorized-memory-creation

feat/context-actions

refactor/action-class

feat/memories-view

refactor/asset-action-filter

feat/archive-action

feat/stack-action

feat/mobile-hls-player

refactor/stack-actions

feat/workflow-asset-tags

fix/timeline-drag-select-stops

fix/featured-photo-thumbnail-cache

fix/album-timeline-date-ordering

chore/album-remove-assets

feat/partner-table-permissions

fix/ocr-after-edit

fix/re-enable-pruning

refactor/delete-restore-actions

apk-build-deployment

edit-photo-stacking

fix/mise-lockfile

feat/web-hls-hover

refactor/store-cleanup

fix/hash-mismatch

refactor/app-metadata

refactor/advanced-settings

refactor/current-user

fix/dedupe-integrity-missing-paths

chore/albums-select

fix/fdroid-meta

refactor/per-asset-backup-flag

fix/av1-threads

fix/pass-secrets

fix/download-progress-stuck

fix/negative-backup-remainder

feat/map-asset-number

test-bump-prerelease

fix/handle-unlinking-motion-photos

fix/date-range-formatting

fix/mobile-album-action-msg

asset-add-to-album-trigger

refactor/face-editor-decoupling

feat/zoom-aware-face-editing

refactor/adaptive-image-dimensions

fix-face-editor-video-preview

fix-stack-face-tag-selection

fix/face-boxes-edit-panel

workflow-webhook-step

feat/hero_view_transitions

fix/map-sidepanel-queries

feat/dart-openapi-generator

claude/share-quality-5d7Gn

drag-drop-ghost-target

refactor/workflow-page

refactor/mobile-upload

fix-scroll-flicker-2

update-mise-lock-file

feat/kysely-0.29

release-candidate-flow-1

fix/mobile-22522-sqlite-2067

fix/job-lock

release-candidate-flow-3

release-candidate-flow-2

show-in-timeline-toggle

fix/mise-windows

plan-local-image-display

fix/remote-thumbnail-fallback

chore/riverpod-update

album-order-per-user

fix/web-timeline-dom-node-retention

chore/svelte-enable-state-referenced-locally

feat/favorite-albums

feat/patch-release-from-branch

fix/oauth-linking

feat/timeline_scroll_buffer

panorama-face-overlay

push-txxyuusptoru

push-zpwsovysllvn

chore/queue-endpoint-migration

fix/remove-from-album

chore/thumbnail-histogram

fix/video-thumbnail-bindthis-regression

fix/scrubber-drag-edge-cases

fix/timeline_layout_row_height

refactor-cancellable-task

uhthomas/fix-mobile-use-timer

uhthomas/fix-mobile-video-controls-key

uhthomas/chore-mobile-search-declarative

owner-role-album-user

feat/livephoto-editing

favorite-albums

uhthomas/fix-mobile-isolate-auth

fix/library-watcher-excludes

chore/library-e2e-to-medium

push-ynllunsvrpxy

fix/dont-delete-offline-library-files

push-zvwwxmvkrspv

push-kywsnvvkwzts

push-xxnxokokzrou

push-voozlrlkxsnu

view-in-timeline-transition

push-uymvmlknxpmx

push-qslzsplnusoz

fix/mobile-video-stall

uhthomas/draft-mobile-image-cache-cancel

push-yquostpqnkxp

feat/vitest-4

claude/offload-icloud-hashing-bI3GZ

midzelis/wip

uhthomas/fix-server-fsync-all

feat/mobile-widget-shared-client

push-lnyuzsutzkqq

push-vqqqxxvkunru

push-wvnmqoswptww

push-nwxlpmyzkyrl

fix/map-webgl-error

fix/mobile-player-fit

feat/panorama-tiles

csp-policy

push-rsywxvptwxuv

refactor/restores-file-interceptor

postgres-socketio

claude/auto-screenshot-web-changes-Y7efI

visual-review/pr-26535

push-lvyturrtwkrq

feat/library-offline-count

fix/mobile-video-aspect-ratio

uhthomas/fix-mobile-search-results

uhthomas/feat-sort-smart-search

uhthomas/chore-mobile-maplibre

uhthomas/mobile-fix-asset-details-album-pop

feat/crawl-wrapper

push-skvzqoozqkpl

feat/edit-filters

feat/pg-queue

refactor/asset-upload

better-project-structure

uhthomas/mobile-feat-asset-viewer-details

fix/ml-rocm-build

feat/asset-file-apis

feature/bottom-buttons-order

sqlite_thumbs

fix-keep-correct-ios-shared-album-asset

push-vpxwmwwxwnvw

fix-migration-width-height

shared-deep-link-handler

feat/thumbnail-native-clients

feat/platform-clients

fix/foreground-cloud-sync

filter-by-person

refactor/sidebar

fix/merged-edited-assets

feat/create-job-with-dto

feat/ios-fastlane-match

match-signing

fix-update-time-update-timeline

feat/modal-routes

feature/mobile-view-asset-owner

feat/system-settings

feature/show-activity-count

feat/location-favorites

feature/rearrange-buttons-2

fix/download-storage-template

chore/originals-in-asset-files

ben/tree-a11y

new-search-filter-ui

refactor/expectSelectedReadonly

refactor/mobile-grdb

feat/mobile-native-local-sync

refactor/timeline_ops

fix/scrubber_end

refactor/virtualsegment

refactor/rename_daymonth_groups

fix-remote-sync-clean-up

debug/cf-chunked-uploads

feat/search-filter-album/web

feat/session-permissions

feat/mobile-dynamic-thumbnails

refactor/extract_photostream

refactor/rename_load_api

refactor/timeline2

refactor/timeline3

feat-no-thumbhash-cache

feat/mobile-hdr-images

feat/beta-background-upload

fix/beta-timeline-memories-setting

fix/failed-uploads-not-removed

feat/groups

drift-map-page

feat/add-to-album-action

track-livephotos

feat/maintenance-worker

refactor/server-side-dedupe

feat/integrity-checks

dev/recognition-eval

lighter_buckets_test

perf/postgres-queue

tmp/demo-snapshot-preview

fix/server-migration-file-extension

rknn-toolkit-lite2

feature/Add-rocm-support-for-machine-learning

chore/async-hash-file

feat/shared-link-view-count

feat/graphql

no-video-player

fix/server-qsv-output-format

chore/server-geodata-tweaks

mobile/native-video-player-no-hero

feat/local-tileserver

feat/ml-armnn-conversion

chore/handle-output_dims

feat/capacitor-mobile-app-poc

feat/server-nvenc-hw-decoding

web/automation-ui

object-storage

ml/tflite

feat/ml-export-cli

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: immich-app/immich#6436