Jobs hangs the system after a short while #6433

Closed
opened 2026-02-05 12:17:21 +03:00 by OVERLORD · 1 comment
Owner

Originally created by @off112a on GitHub (Jul 6, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

I have a external library with about 20 000 photos. I have jobs running. But after a short while my whole system hangs and are not reachable from either the web gui or ssh. The only way is to reboot the VM.

I have 8 GB of memory on the VM.

The OS that Immich Server is running on

Proxmox VM debian 12

Version of Immich Server

v1.135.3

Version of Immich Mobile App

latest

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/nextclouddisk/data/arild/files/Photos:/mnt/media/arild:rw
      - /mnt/nextclouddisk/data/maria/files/Photos:/mnt/media/maria:rw
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:fec42f399876eb6faf9e008570597741c87ff7662a54185593e74b09ce83d177
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: --data-checksums
      TZ: Europe/Stockholm
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
      - /etc/localtime:/etc/localtime:ro
    restart: always
volumes:
  model-cache: null
networks: {}

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The location where your database files are stored. Network shares are not supported for the database
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
 TZ=Europe/Stockholm

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=XXXXXXXXX

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

  1. Start the VM
  2. Wait
  3. It hangs.
    ...

Relevant log output

immich_redis             | 7717:C 06 Jul 2025 06:26:10.821 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
immich_redis             | 1:M 06 Jul 2025 06:26:10.830 * Background saving terminated with success
immich_redis             | 1:M 06 Jul 2025 06:34:57.421 * 100 changes in 300 seconds. Saving...
immich_redis             | 1:M 06 Jul 2025 06:35:15.390 * Background saving started by pid 7740
immich_redis             | 7740:C 06 Jul 2025 06:41:11.950 * DB saved on disk
immich_redis             | 7740:C 06 Jul 2025 06:41:11.950 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
immich_redis             | 1:M 06 Jul 2025 06:41:12.022 * Background saving terminated with success
immich_redis             | 1:M 06 Jul 2025 06:46:13.090 * 100 changes in 300 seconds. Saving...
immich_redis             | 1:M 06 Jul 2025 06:46:13.091 * Background saving started by pid 7807
immich_redis             | 7807:C 06 Jul 2025 06:46:13.189 * DB saved on disk
immich_redis             | 7807:C 06 Jul 2025 06:46:13.190 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
immich_redis             | 1:M 06 Jul 2025 06:46:13.192 * Background saving terminated with success
immich_redis             | 1:M 06 Jul 2025 06:51:47.941 * 100 changes in 300 seconds. Saving...
immich_redis             | 1:M 06 Jul 2025 06:51:48.859 * Background saving started by pid 7844
immich_redis             | 7844:C 06 Jul 2025 06:52:24.363 * DB saved on disk
immich_redis             | 7844:C 06 Jul 2025 06:52:27.592 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
immich_redis             | 1:M 06 Jul 2025 06:52:33.358 * Background saving terminated with success
immich_redis             | 1:M 06 Jul 2025 06:57:53.637 * 100 changes in 300 seconds. Saving...
immich_redis             | 1:M 06 Jul 2025 06:57:56.623 * Background saving started by pid 7886
immich_redis             | 7886:C 06 Jul 2025 06:59:34.892 * DB saved on disk
immich_redis             | 7886:C 06 Jul 2025 06:59:38.779 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
immich_redis             | 1:M 06 Jul 2025 06:59:44.550 * Background saving terminated with success
immich_redis             | 1:M 06 Jul 2025 09:18:41.534 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 09:18:41.534 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 09:18:41.534 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 09:18:41.534 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 09:18:41.535 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 09:18:41.540 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 09:18:41.540 * Server initialized
immich_redis             | 1:M 06 Jul 2025 09:18:41.544 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 09:18:41.544 * RDB age 8437 seconds
immich_redis             | 1:M 06 Jul 2025 09:18:41.544 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 09:18:41.714 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 09:18:41.714 * DB loaded from disk: 0.174 seconds
immich_redis             | 1:M 06 Jul 2025 09:18:41.714 * Ready to accept connections tcp
immich_redis             | 1:M 06 Jul 2025 09:42:19.467 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 09:42:19.467 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 09:42:19.467 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 09:42:19.467 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 09:42:19.468 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 09:42:19.475 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 09:42:19.475 * Server initialized
immich_redis             | 1:M 06 Jul 2025 09:42:19.480 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 09:42:19.480 * RDB age 9855 seconds
immich_redis             | 1:M 06 Jul 2025 09:42:19.480 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 09:42:19.664 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 09:42:19.664 * DB loaded from disk: 0.190 seconds
immich_redis             | 1:M 06 Jul 2025 09:42:19.664 * Ready to accept connections tcp
immich_redis             | 1:M 06 Jul 2025 11:41:05.164 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 11:41:05.164 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 11:41:05.164 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 11:41:05.164 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 11:41:05.165 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 11:41:05.171 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 11:41:05.172 * Server initialized
immich_redis             | 1:M 06 Jul 2025 11:41:05.177 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 11:41:05.177 * RDB age 16981 seconds
immich_redis             | 1:M 06 Jul 2025 11:41:05.177 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 11:41:05.225 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 11:41:05.225 * DB loaded from disk: 0.053 seconds
immich_redis             | 1:M 06 Jul 2025 11:41:05.225 * Ready to accept connections tcp
immich_redis             | 1:M 06 Jul 2025 11:50:29.440 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 11:50:29.440 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 11:50:29.440 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 11:50:29.440 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 11:50:29.442 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 11:50:29.448 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 11:50:29.448 * Server initialized
immich_redis             | 1:M 06 Jul 2025 11:50:29.456 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 11:50:29.457 * RDB age 17545 seconds
immich_redis             | 1:M 06 Jul 2025 11:50:29.457 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 11:50:29.501 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 11:50:29.501 * DB loaded from disk: 0.053 seconds
immich_redis             | 1:M 06 Jul 2025 11:50:29.501 * Ready to accept connections tcp
immich_redis             | 1:M 06 Jul 2025 12:13:04.041 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 12:13:04.041 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 12:13:04.041 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 12:13:04.041 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 12:13:04.043 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 12:13:04.109 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 12:13:04.110 * Server initialized
immich_redis             | 1:M 06 Jul 2025 12:13:04.116 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 12:13:04.117 * RDB age 18900 seconds
immich_redis             | 1:M 06 Jul 2025 12:13:04.117 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 12:13:04.360 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 12:13:04.360 * DB loaded from disk: 0.251 seconds
immich_redis             | 1:M 06 Jul 2025 12:13:04.361 * Ready to accept connections tcp
immich_redis             | 1:M 06 Jul 2025 12:19:10.069 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 06 Jul 2025 12:19:10.069 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
immich_redis             | 1:M 06 Jul 2025 12:19:10.069 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:M 06 Jul 2025 12:19:10.069 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf
immich_redis             | 1:M 06 Jul 2025 12:19:10.134 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 06 Jul 2025 12:19:10.137 * Running mode=standalone, port=6379.
immich_redis             | 1:M 06 Jul 2025 12:19:10.137 * Server initialized
immich_redis             | 1:M 06 Jul 2025 12:19:10.143 * Loading RDB produced by Valkey version 8.1.2
immich_redis             | 1:M 06 Jul 2025 12:19:10.143 * RDB age 19266 seconds
immich_redis             | 1:M 06 Jul 2025 12:19:10.143 * RDB memory usage when created 10.91 Mb
immich_redis             | 1:M 06 Jul 2025 12:19:10.334 * Done loading RDB, keys loaded: 12656, keys expired: 2.
immich_redis             | 1:M 06 Jul 2025 12:19:10.334 * DB loaded from disk: 0.197 seconds
immich_redis             | 1:M 06 Jul 2025 12:19:10.334 * Ready to accept connections tcp
immich_machine_learning  | [07/06/25 11:06:11] INFO     Booting worker with pid: 143                       
immich_machine_learning  | [07/06/25 11:06:20] INFO     Started server process [143]                       
immich_machine_learning  | [07/06/25 11:06:20] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 11:06:20] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 11:06:20] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 11:06:20] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 11:06:25] INFO     Loading detection model 'buffalo_l' to memory      
immich_machine_learning  | [07/06/25 11:06:25] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 11:06:26] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [07/06/25 11:06:26] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 11:14:05] CRITICAL WORKER TIMEOUT (pid:143)                           
immich_machine_learning  | [07/06/25 11:17:39] ERROR    Worker (pid:143) was sent code 134!                
immich_machine_learning  | [07/06/25 11:17:40] INFO     Booting worker with pid: 214                       
immich_machine_learning  | [07/06/25 13:41:07] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [07/06/25 13:41:07] INFO     Listening at: http://[::]:3003 (8)                 
immich_machine_learning  | [07/06/25 13:41:07] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [07/06/25 13:41:07] INFO     Booting worker with pid: 9                         
immich_machine_learning  | [07/06/25 13:41:11] INFO     Started server process [9]                         
immich_machine_learning  | [07/06/25 13:41:11] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 13:41:11] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 13:41:11] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 13:41:11] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 13:41:15] INFO     Loading detection model 'buffalo_l' to memory      
immich_machine_learning  | [07/06/25 13:41:15] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 13:41:16] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [07/06/25 13:41:16] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 13:50:31] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [07/06/25 13:50:31] INFO     Listening at: http://[::]:3003 (8)                 
immich_machine_learning  | [07/06/25 13:50:31] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [07/06/25 13:50:31] INFO     Booting worker with pid: 9                         
immich_machine_learning  | [07/06/25 13:50:35] INFO     Started server process [9]                         
immich_machine_learning  | [07/06/25 13:50:35] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 13:50:35] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 13:50:35] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 13:50:35] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 13:50:39] INFO     Loading detection model 'buffalo_l' to memory      
immich_machine_learning  | [07/06/25 13:50:39] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 13:50:40] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [07/06/25 13:50:40] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 11:55:50] CRITICAL WORKER TIMEOUT (pid:9)                             
immich_machine_learning  | [07/06/25 11:56:26] ERROR    Worker (pid:9) was sent SIGKILL! Perhaps out of    
immich_machine_learning  |                              memory?                                            
immich_machine_learning  | [07/06/25 11:56:26] INFO     Booting worker with pid: 40                        
immich_machine_learning  | [07/06/25 11:59:09] INFO     Started server process [40]                        
immich_machine_learning  | [07/06/25 11:59:09] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 11:59:10] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 11:59:10] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 11:59:10] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 14:13:10] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [07/06/25 14:13:10] INFO     Listening at: http://[::]:3003 (8)                 
immich_machine_learning  | [07/06/25 14:13:10] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [07/06/25 14:13:10] INFO     Booting worker with pid: 9                         
immich_machine_learning  | [07/06/25 14:13:22] INFO     Started server process [9]                         
immich_machine_learning  | [07/06/25 14:13:22] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 14:13:22] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 14:13:22] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 14:13:22] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 14:13:29] INFO     Loading detection model 'buffalo_l' to memory      
immich_machine_learning  | [07/06/25 14:13:29] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 14:13:30] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [07/06/25 14:13:30] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 14:19:15] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [07/06/25 14:19:15] INFO     Listening at: http://[::]:3003 (7)                 
immich_machine_learning  | [07/06/25 14:19:15] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [07/06/25 14:19:15] INFO     Booting worker with pid: 8                         
immich_machine_learning  | [07/06/25 14:19:19] INFO     Started server process [8]                         
immich_machine_learning  | [07/06/25 14:19:19] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/06/25 14:19:19] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/06/25 14:19:19] INFO     Initialized request thread pool with 4 threads.    
immich_machine_learning  | [07/06/25 14:19:19] INFO     Application startup complete.                      
immich_machine_learning  | [07/06/25 14:19:23] INFO     Loading detection model 'buffalo_l' to memory      
immich_machine_learning  | [07/06/25 14:19:23] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_machine_learning  | [07/06/25 14:19:24] INFO     Loading recognition model 'buffalo_l' to memory    
immich_machine_learning  | [07/06/25 14:19:24] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of   
immich_machine_learning  |                              preference                                         
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions/:id, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions/:id/lock, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] SharedLinkController {/api/shared-links}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/me, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, PATCH} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] StackController {/api/stacks}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] SyncController {/api/sync}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/full-sync, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/delta-sync, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/stream, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/ack, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/ack, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/ack, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] SystemConfigController {/api/system-config}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config/defaults, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] SystemMetadataController {/api/system-metadata}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/reverse-geocoding-state, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/version-check-state, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] TagController {/api/tags}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/assets, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] TimelineController {/api/timeline}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/timeline/buckets, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/timeline/bucket, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] TrashController {/api/trash}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/empty, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/restore, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/restore/assets, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] UserAdminController {/api/admin/users}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/statistics, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/restore, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] UserController {/api/users}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/license, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/license, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/license, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, PUT} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/:id, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, POST} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, DELETE} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/users/:id/profile-image, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RoutesResolver] ViewController {/api/view}:
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/view/folder/unique-paths, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:RouterExplorer] Mapped {/api/view/folder, GET} route
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:NestApplication] Nest application successfully started
immich_server            | [Nest] 17  - 07/06/2025, 2:19:23 PM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.135.3] [production] 
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:SystemConfigService] LogLevel=log (set via system config)
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:NestFactory] Starting Nest application...
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] ClsModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] ClsCommonModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] KyselyModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] KyselyCoreModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] ClsRootModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:NestApplication] Nest application successfully started
immich_server            | [Nest] 7  - 07/06/2025, 2:19:23 PM     LOG [Microservices:Bootstrap] Immich Microservices is running [v1.135.3] [production] 
immich_postgres          | 2025-07-06 10:25:17.052 CEST [91] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 10:28:14.939 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 10:34:51.751 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 10:36:53.110 CEST [94] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 10:44:32.506 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 10:45:12.821 CEST [96] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 10:49:11.073 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 10:49:23.208 CEST [97] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 10:51:35.059 CEST [98] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 10:59:39.639 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 11:02:26.048 CEST [119] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 11:04:37.617 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 11:06:01.013 CEST [120] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 11:06:03.356 CEST [111] LOG:  unexpected EOF on client connection with an open transaction
immich_postgres          | 2025-07-06 11:16:46.273 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 11:16:53.319 CEST [162] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 11:23:33.408 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 11:24:46.376 CEST [179] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | Using SSD storage
immich_postgres          | 
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          | 
immich_postgres          | 2025-07-06 11:41:05.607 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 11:41:05.607 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 13:41:05.723 CEST [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres          | 2025-07-06 13:41:05.723 CEST [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres          | 2025-07-06 13:41:05.723 CEST [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres          | 2025-07-06 13:41:05.736 CEST [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres          | 2025-07-06 13:41:05.756 CEST [33] LOG:  database system was interrupted; last known up at 2025-07-06 11:17:42 CEST
immich_postgres          | 2025-07-06 13:41:05.958 CEST [33] LOG:  database system was not properly shut down; automatic recovery in progress
immich_postgres          | 2025-07-06 13:41:05.963 CEST [33] LOG:  redo starts at 0/697B19D8
immich_postgres          | 2025-07-06 13:41:05.963 CEST [33] LOG:  invalid record length at 0/697B1AC0: wanted 24, got 0
immich_postgres          | 2025-07-06 13:41:05.963 CEST [33] LOG:  redo done at 0/697B1A88 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
immich_postgres          | 2025-07-06 13:41:06.045 CEST [1] LOG:  database system is ready to accept connections
immich_postgres          | 2025-07-06 11:42:07.404 CEST [40] LOG:  statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.404022+02
immich_postgres          | 2025-07-06 11:42:07.404 CEST [40] LOG:  statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.404244+02
immich_postgres          | 2025-07-06 11:42:07.404 CEST [41] LOG:  stats_timestamp 2025-07-06 13:41:10.301514+02 is later than collector's time 2025-07-06 11:42:07.404212+02 for database 0
immich_postgres          | 2025-07-06 11:42:07.407 CEST [70] LOG:  statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.407792+02
immich_postgres          | 2025-07-06 11:42:07.409 CEST [41] LOG:  stats_timestamp 2025-07-06 13:41:10.301514+02 is later than collector's time 2025-07-06 11:42:07.409508+02 for database 16384
immich_postgres          | Using SSD storage
immich_postgres          | 
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          | 
immich_postgres          | 2025-07-06 11:50:29.819 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 11:50:29.819 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 13:50:29.937 CEST [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres          | 2025-07-06 13:50:29.939 CEST [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres          | 2025-07-06 13:50:29.939 CEST [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres          | 2025-07-06 13:50:29.952 CEST [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres          | 2025-07-06 13:50:29.972 CEST [32] LOG:  database system was interrupted; last known up at 2025-07-06 13:41:06 CEST
immich_postgres          | 2025-07-06 13:50:30.229 CEST [32] LOG:  database system was not properly shut down; automatic recovery in progress
immich_postgres          | 2025-07-06 13:50:30.238 CEST [32] LOG:  redo starts at 0/697B1B38
immich_postgres          | 2025-07-06 13:50:30.246 CEST [32] LOG:  invalid record length at 0/697E1258: wanted 24, got 0
immich_postgres          | 2025-07-06 13:50:30.246 CEST [32] LOG:  redo done at 0/697E1230 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
immich_postgres          | 2025-07-06 13:50:30.328 CEST [1] LOG:  database system is ready to accept connections
immich_postgres          | 2025-07-06 11:51:31.533 CEST [39] LOG:  statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.529581+02
immich_postgres          | 2025-07-06 11:51:31.533 CEST [40] LOG:  stats_timestamp 2025-07-06 13:50:34.559781+02 is later than collector's time 2025-07-06 11:51:31.533923+02 for database 0
immich_postgres          | 2025-07-06 11:51:31.534 CEST [39] LOG:  statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.534868+02
immich_postgres          | 2025-07-06 11:51:31.562 CEST [63] LOG:  statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.562572+02
immich_postgres          | 2025-07-06 11:51:31.562 CEST [40] LOG:  stats_timestamp 2025-07-06 13:50:34.559781+02 is later than collector's time 2025-07-06 11:51:31.562652+02 for database 16384
immich_postgres          | 2025-07-06 11:54:54.874 CEST [39] WARNING:  worker took too long to start; canceled
immich_postgres          | 2025-07-06 11:55:01.137 CEST [64] WARNING:  autovacuum worker started without a worker entry
immich_postgres          | 2025-07-06 11:58:57.811 CEST [61] LOG:  could not receive data from client: Connection reset by peer
immich_postgres          | Using SSD storage
immich_postgres          | 
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          | 
immich_postgres          | 2025-07-06 12:13:05.875 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 12:13:05.875 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 14:13:06.262 CEST [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres          | 2025-07-06 14:13:06.312 CEST [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres          | 2025-07-06 14:13:06.312 CEST [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres          | 2025-07-06 14:13:06.326 CEST [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres          | 2025-07-06 14:13:06.414 CEST [33] LOG:  database system was interrupted; last known up at 2025-07-06 13:50:30 CEST
immich_postgres          | 2025-07-06 14:13:06.585 CEST [33] LOG:  database system was not properly shut down; automatic recovery in progress
immich_postgres          | 2025-07-06 14:13:06.592 CEST [33] LOG:  redo starts at 0/697E12D0
immich_postgres          | 2025-07-06 14:13:06.675 CEST [33] LOG:  invalid record length at 0/69874B60: wanted 24, got 0
immich_postgres          | 2025-07-06 14:13:06.675 CEST [33] LOG:  redo done at 0/69874B38 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.08 s
immich_postgres          | 2025-07-06 14:13:06.980 CEST [1] LOG:  database system is ready to accept connections
immich_postgres          | 2025-07-06 12:14:08.447 CEST [40] LOG:  statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.447607+02
immich_postgres          | 2025-07-06 12:14:08.447 CEST [41] LOG:  stats_timestamp 2025-07-06 14:13:09.737718+02 is later than collector's time 2025-07-06 12:14:08.447765+02 for database 0
immich_postgres          | 2025-07-06 12:14:08.447 CEST [40] LOG:  statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.447814+02
immich_postgres          | 2025-07-06 12:14:08.450 CEST [64] LOG:  statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.45091+02
immich_postgres          | 2025-07-06 12:14:08.451 CEST [41] LOG:  stats_timestamp 2025-07-06 14:13:09.737718+02 is later than collector's time 2025-07-06 12:14:08.451311+02 for database 16384
immich_postgres          | Using SSD storage
immich_postgres          | 
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          | 
immich_postgres          | 2025-07-06 12:19:11.711 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 12:19:11.712 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
immich_postgres          | 2025-07-06 14:19:12.019 CEST [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres          | 2025-07-06 14:19:12.076 CEST [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres          | 2025-07-06 14:19:12.076 CEST [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres          | 2025-07-06 14:19:12.090 CEST [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres          | 2025-07-06 14:19:12.173 CEST [32] LOG:  database system was interrupted; last known up at 2025-07-06 14:13:06 CEST
immich_postgres          | 2025-07-06 14:19:12.352 CEST [32] LOG:  database system was not properly shut down; automatic recovery in progress
immich_postgres          | 2025-07-06 14:19:12.366 CEST [32] LOG:  redo starts at 0/69874BD8
immich_postgres          | 2025-07-06 14:19:12.437 CEST [32] LOG:  invalid record length at 0/698814C0: wanted 24, got 0
immich_postgres          | 2025-07-06 14:19:12.437 CEST [32] LOG:  redo done at 0/69881498 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.07 s
immich_postgres          | 2025-07-06 14:19:12.722 CEST [1] LOG:  database system is ready to accept connections
immich_server            | [Nest] 17  - 07/06/2025, 2:19:28 PM     LOG [Api:EventRepository] Websocket Connect:    al4hfgMkzQyuAHQnAAAB
immich_postgres          | 2025-07-06 12:20:14.492 CEST [39] LOG:  statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.49233+02
immich_postgres          | 2025-07-06 12:20:14.492 CEST [40] LOG:  stats_timestamp 2025-07-06 14:19:15.232619+02 is later than collector's time 2025-07-06 12:20:14.492529+02 for database 0
immich_postgres          | 2025-07-06 12:20:14.492 CEST [39] LOG:  statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.492751+02
immich_postgres          | 2025-07-06 12:20:14.556 CEST [63] LOG:  statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.556789+02
immich_postgres          | 2025-07-06 12:20:14.556 CEST [40] LOG:  stats_timestamp 2025-07-06 14:19:15.232619+02 is later than collector's time 2025-07-06 12:20:14.556859+02 for database 16384

Additional information

No response

Originally created by @off112a on GitHub (Jul 6, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug I have a external library with about 20 000 photos. I have jobs running. But after a short while my whole system hangs and are not reachable from either the web gui or ssh. The only way is to reboot the VM. I have 8 GB of memory on the VM. ### The OS that Immich Server is running on Proxmox VM debian 12 ### Version of Immich Server v1.135.3 ### Version of Immich Mobile App latest ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose # # Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - /mnt/nextclouddisk/data/arild/files/Photos:/mnt/media/arild:rw - /mnt/nextclouddisk/data/maria/files/Photos:/mnt/media/maria:rw - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:2283 depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env restart: always healthcheck: disable: false redis: container_name: immich_redis image: docker.io/valkey/valkey:8-bookworm@sha256:fec42f399876eb6faf9e008570597741c87ff7662a54185593e74b09ce83d177 healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: --data-checksums TZ: Europe/Stockholm # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs # DB_STORAGE_TYPE: 'HDD' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data - /etc/localtime:/etc/localtime:ro restart: always volumes: model-cache: null networks: {} ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The location where your database files are stored. Network shares are not supported for the database DB_DATA_LOCATION=./postgres # To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List TZ=Europe/Stockholm # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password # Please use only the characters `A-Za-z0-9`, without special characters or spaces DB_PASSWORD=XXXXXXXXX # The values below this line do not need to be changed ################################################################################### DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` ### Reproduction steps 1. Start the VM 2. Wait 3. It hangs. ... ### Relevant log output ```shell immich_redis | 7717:C 06 Jul 2025 06:26:10.821 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB immich_redis | 1:M 06 Jul 2025 06:26:10.830 * Background saving terminated with success immich_redis | 1:M 06 Jul 2025 06:34:57.421 * 100 changes in 300 seconds. Saving... immich_redis | 1:M 06 Jul 2025 06:35:15.390 * Background saving started by pid 7740 immich_redis | 7740:C 06 Jul 2025 06:41:11.950 * DB saved on disk immich_redis | 7740:C 06 Jul 2025 06:41:11.950 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB immich_redis | 1:M 06 Jul 2025 06:41:12.022 * Background saving terminated with success immich_redis | 1:M 06 Jul 2025 06:46:13.090 * 100 changes in 300 seconds. Saving... immich_redis | 1:M 06 Jul 2025 06:46:13.091 * Background saving started by pid 7807 immich_redis | 7807:C 06 Jul 2025 06:46:13.189 * DB saved on disk immich_redis | 7807:C 06 Jul 2025 06:46:13.190 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB immich_redis | 1:M 06 Jul 2025 06:46:13.192 * Background saving terminated with success immich_redis | 1:M 06 Jul 2025 06:51:47.941 * 100 changes in 300 seconds. Saving... immich_redis | 1:M 06 Jul 2025 06:51:48.859 * Background saving started by pid 7844 immich_redis | 7844:C 06 Jul 2025 06:52:24.363 * DB saved on disk immich_redis | 7844:C 06 Jul 2025 06:52:27.592 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB immich_redis | 1:M 06 Jul 2025 06:52:33.358 * Background saving terminated with success immich_redis | 1:M 06 Jul 2025 06:57:53.637 * 100 changes in 300 seconds. Saving... immich_redis | 1:M 06 Jul 2025 06:57:56.623 * Background saving started by pid 7886 immich_redis | 7886:C 06 Jul 2025 06:59:34.892 * DB saved on disk immich_redis | 7886:C 06 Jul 2025 06:59:38.779 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB immich_redis | 1:M 06 Jul 2025 06:59:44.550 * Background saving terminated with success immich_redis | 1:M 06 Jul 2025 09:18:41.534 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 09:18:41.534 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 09:18:41.534 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 09:18:41.534 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 09:18:41.535 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 09:18:41.540 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 09:18:41.540 * Server initialized immich_redis | 1:M 06 Jul 2025 09:18:41.544 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 09:18:41.544 * RDB age 8437 seconds immich_redis | 1:M 06 Jul 2025 09:18:41.544 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 09:18:41.714 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 09:18:41.714 * DB loaded from disk: 0.174 seconds immich_redis | 1:M 06 Jul 2025 09:18:41.714 * Ready to accept connections tcp immich_redis | 1:M 06 Jul 2025 09:42:19.467 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 09:42:19.467 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 09:42:19.467 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 09:42:19.467 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 09:42:19.468 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 09:42:19.475 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 09:42:19.475 * Server initialized immich_redis | 1:M 06 Jul 2025 09:42:19.480 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 09:42:19.480 * RDB age 9855 seconds immich_redis | 1:M 06 Jul 2025 09:42:19.480 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 09:42:19.664 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 09:42:19.664 * DB loaded from disk: 0.190 seconds immich_redis | 1:M 06 Jul 2025 09:42:19.664 * Ready to accept connections tcp immich_redis | 1:M 06 Jul 2025 11:41:05.164 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 11:41:05.164 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 11:41:05.164 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 11:41:05.164 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 11:41:05.165 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 11:41:05.171 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 11:41:05.172 * Server initialized immich_redis | 1:M 06 Jul 2025 11:41:05.177 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 11:41:05.177 * RDB age 16981 seconds immich_redis | 1:M 06 Jul 2025 11:41:05.177 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 11:41:05.225 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 11:41:05.225 * DB loaded from disk: 0.053 seconds immich_redis | 1:M 06 Jul 2025 11:41:05.225 * Ready to accept connections tcp immich_redis | 1:M 06 Jul 2025 11:50:29.440 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 11:50:29.440 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 11:50:29.440 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 11:50:29.440 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 11:50:29.442 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 11:50:29.448 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 11:50:29.448 * Server initialized immich_redis | 1:M 06 Jul 2025 11:50:29.456 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 11:50:29.457 * RDB age 17545 seconds immich_redis | 1:M 06 Jul 2025 11:50:29.457 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 11:50:29.501 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 11:50:29.501 * DB loaded from disk: 0.053 seconds immich_redis | 1:M 06 Jul 2025 11:50:29.501 * Ready to accept connections tcp immich_redis | 1:M 06 Jul 2025 12:13:04.041 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 12:13:04.041 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 12:13:04.041 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 12:13:04.041 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 12:13:04.043 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 12:13:04.109 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 12:13:04.110 * Server initialized immich_redis | 1:M 06 Jul 2025 12:13:04.116 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 12:13:04.117 * RDB age 18900 seconds immich_redis | 1:M 06 Jul 2025 12:13:04.117 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 12:13:04.360 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 12:13:04.360 * DB loaded from disk: 0.251 seconds immich_redis | 1:M 06 Jul 2025 12:13:04.361 * Ready to accept connections tcp immich_redis | 1:M 06 Jul 2025 12:19:10.069 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 06 Jul 2025 12:19:10.069 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo immich_redis | 1:M 06 Jul 2025 12:19:10.069 * Valkey version=8.1.2, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:M 06 Jul 2025 12:19:10.069 # Warning: no config file specified, using the default config. In order to specify a config file use valkey-server /path/to/valkey.conf immich_redis | 1:M 06 Jul 2025 12:19:10.134 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 06 Jul 2025 12:19:10.137 * Running mode=standalone, port=6379. immich_redis | 1:M 06 Jul 2025 12:19:10.137 * Server initialized immich_redis | 1:M 06 Jul 2025 12:19:10.143 * Loading RDB produced by Valkey version 8.1.2 immich_redis | 1:M 06 Jul 2025 12:19:10.143 * RDB age 19266 seconds immich_redis | 1:M 06 Jul 2025 12:19:10.143 * RDB memory usage when created 10.91 Mb immich_redis | 1:M 06 Jul 2025 12:19:10.334 * Done loading RDB, keys loaded: 12656, keys expired: 2. immich_redis | 1:M 06 Jul 2025 12:19:10.334 * DB loaded from disk: 0.197 seconds immich_redis | 1:M 06 Jul 2025 12:19:10.334 * Ready to accept connections tcp immich_machine_learning | [07/06/25 11:06:11] INFO Booting worker with pid: 143 immich_machine_learning | [07/06/25 11:06:20] INFO Started server process [143] immich_machine_learning | [07/06/25 11:06:20] INFO Waiting for application startup. immich_machine_learning | [07/06/25 11:06:20] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 11:06:20] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 11:06:20] INFO Application startup complete. immich_machine_learning | [07/06/25 11:06:25] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/06/25 11:06:25] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 11:06:26] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/06/25 11:06:26] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 11:14:05] CRITICAL WORKER TIMEOUT (pid:143) immich_machine_learning | [07/06/25 11:17:39] ERROR Worker (pid:143) was sent code 134! immich_machine_learning | [07/06/25 11:17:40] INFO Booting worker with pid: 214 immich_machine_learning | [07/06/25 13:41:07] INFO Starting gunicorn 23.0.0 immich_machine_learning | [07/06/25 13:41:07] INFO Listening at: http://[::]:3003 (8) immich_machine_learning | [07/06/25 13:41:07] INFO Using worker: immich_ml.config.CustomUvicornWorker immich_machine_learning | [07/06/25 13:41:07] INFO Booting worker with pid: 9 immich_machine_learning | [07/06/25 13:41:11] INFO Started server process [9] immich_machine_learning | [07/06/25 13:41:11] INFO Waiting for application startup. immich_machine_learning | [07/06/25 13:41:11] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 13:41:11] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 13:41:11] INFO Application startup complete. immich_machine_learning | [07/06/25 13:41:15] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/06/25 13:41:15] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 13:41:16] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/06/25 13:41:16] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 13:50:31] INFO Starting gunicorn 23.0.0 immich_machine_learning | [07/06/25 13:50:31] INFO Listening at: http://[::]:3003 (8) immich_machine_learning | [07/06/25 13:50:31] INFO Using worker: immich_ml.config.CustomUvicornWorker immich_machine_learning | [07/06/25 13:50:31] INFO Booting worker with pid: 9 immich_machine_learning | [07/06/25 13:50:35] INFO Started server process [9] immich_machine_learning | [07/06/25 13:50:35] INFO Waiting for application startup. immich_machine_learning | [07/06/25 13:50:35] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 13:50:35] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 13:50:35] INFO Application startup complete. immich_machine_learning | [07/06/25 13:50:39] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/06/25 13:50:39] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 13:50:40] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/06/25 13:50:40] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 11:55:50] CRITICAL WORKER TIMEOUT (pid:9) immich_machine_learning | [07/06/25 11:56:26] ERROR Worker (pid:9) was sent SIGKILL! Perhaps out of immich_machine_learning | memory? immich_machine_learning | [07/06/25 11:56:26] INFO Booting worker with pid: 40 immich_machine_learning | [07/06/25 11:59:09] INFO Started server process [40] immich_machine_learning | [07/06/25 11:59:09] INFO Waiting for application startup. immich_machine_learning | [07/06/25 11:59:10] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 11:59:10] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 11:59:10] INFO Application startup complete. immich_machine_learning | [07/06/25 14:13:10] INFO Starting gunicorn 23.0.0 immich_machine_learning | [07/06/25 14:13:10] INFO Listening at: http://[::]:3003 (8) immich_machine_learning | [07/06/25 14:13:10] INFO Using worker: immich_ml.config.CustomUvicornWorker immich_machine_learning | [07/06/25 14:13:10] INFO Booting worker with pid: 9 immich_machine_learning | [07/06/25 14:13:22] INFO Started server process [9] immich_machine_learning | [07/06/25 14:13:22] INFO Waiting for application startup. immich_machine_learning | [07/06/25 14:13:22] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 14:13:22] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 14:13:22] INFO Application startup complete. immich_machine_learning | [07/06/25 14:13:29] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/06/25 14:13:29] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 14:13:30] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/06/25 14:13:30] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 14:19:15] INFO Starting gunicorn 23.0.0 immich_machine_learning | [07/06/25 14:19:15] INFO Listening at: http://[::]:3003 (7) immich_machine_learning | [07/06/25 14:19:15] INFO Using worker: immich_ml.config.CustomUvicornWorker immich_machine_learning | [07/06/25 14:19:15] INFO Booting worker with pid: 8 immich_machine_learning | [07/06/25 14:19:19] INFO Started server process [8] immich_machine_learning | [07/06/25 14:19:19] INFO Waiting for application startup. immich_machine_learning | [07/06/25 14:19:19] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/06/25 14:19:19] INFO Initialized request thread pool with 4 threads. immich_machine_learning | [07/06/25 14:19:19] INFO Application startup complete. immich_machine_learning | [07/06/25 14:19:23] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/06/25 14:19:23] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/06/25 14:19:24] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/06/25 14:19:24] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sessions, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sessions/:id, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sessions/:id/lock, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] SharedLinkController {/api/shared-links}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/me, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, PATCH} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] StackController {/api/stacks}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/stacks/:id, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] SyncController {/api/sync}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/full-sync, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/delta-sync, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/stream, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/ack, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/ack, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/sync/ack, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] SystemConfigController {/api/system-config}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-config, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-config/defaults, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-config, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] SystemMetadataController {/api/system-metadata}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-metadata/reverse-geocoding-state, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/system-metadata/version-check-state, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] TagController {/api/tags}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/assets, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/:id, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/:id, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/:id, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] TimelineController {/api/timeline}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/timeline/buckets, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/timeline/bucket, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] TrashController {/api/trash}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/trash/empty, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/trash/restore, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/trash/restore/assets, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] UserAdminController {/api/admin/users}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/statistics, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/restore, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] UserController {/api/users}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/license, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/license, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/license, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, PUT} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/me/onboarding, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/:id, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, POST} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, DELETE} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/users/:id/profile-image, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RoutesResolver] ViewController {/api/view}: immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/view/folder/unique-paths, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:RouterExplorer] Mapped {/api/view/folder, GET} route immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:NestApplication] Nest application successfully started immich_server | [Nest] 17 - 07/06/2025, 2:19:23 PM LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.135.3] [production] immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:SystemConfigService] LogLevel=log (set via system config) immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:NestFactory] Starting Nest application... immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] BullModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] ClsModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] ClsCommonModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] KyselyModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] KyselyCoreModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] ClsRootModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] BullModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] BullModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:NestApplication] Nest application successfully started immich_server | [Nest] 7 - 07/06/2025, 2:19:23 PM LOG [Microservices:Bootstrap] Immich Microservices is running [v1.135.3] [production] immich_postgres | 2025-07-06 10:25:17.052 CEST [91] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 10:28:14.939 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 10:34:51.751 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 10:36:53.110 CEST [94] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 10:44:32.506 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 10:45:12.821 CEST [96] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 10:49:11.073 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 10:49:23.208 CEST [97] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 10:51:35.059 CEST [98] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 10:59:39.639 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 11:02:26.048 CEST [119] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 11:04:37.617 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 11:06:01.013 CEST [120] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 11:06:03.356 CEST [111] LOG: unexpected EOF on client connection with an open transaction immich_postgres | 2025-07-06 11:16:46.273 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 11:16:53.319 CEST [162] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 11:23:33.408 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 11:24:46.376 CEST [179] WARNING: autovacuum worker started without a worker entry immich_postgres | Using SSD storage immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2025-07-06 11:41:05.607 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 11:41:05.607 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 13:41:05.723 CEST [1] LOG: starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2025-07-06 13:41:05.723 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2025-07-06 13:41:05.723 CEST [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2025-07-06 13:41:05.736 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2025-07-06 13:41:05.756 CEST [33] LOG: database system was interrupted; last known up at 2025-07-06 11:17:42 CEST immich_postgres | 2025-07-06 13:41:05.958 CEST [33] LOG: database system was not properly shut down; automatic recovery in progress immich_postgres | 2025-07-06 13:41:05.963 CEST [33] LOG: redo starts at 0/697B19D8 immich_postgres | 2025-07-06 13:41:05.963 CEST [33] LOG: invalid record length at 0/697B1AC0: wanted 24, got 0 immich_postgres | 2025-07-06 13:41:05.963 CEST [33] LOG: redo done at 0/697B1A88 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s immich_postgres | 2025-07-06 13:41:06.045 CEST [1] LOG: database system is ready to accept connections immich_postgres | 2025-07-06 11:42:07.404 CEST [40] LOG: statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.404022+02 immich_postgres | 2025-07-06 11:42:07.404 CEST [40] LOG: statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.404244+02 immich_postgres | 2025-07-06 11:42:07.404 CEST [41] LOG: stats_timestamp 2025-07-06 13:41:10.301514+02 is later than collector's time 2025-07-06 11:42:07.404212+02 for database 0 immich_postgres | 2025-07-06 11:42:07.407 CEST [70] LOG: statistics collector's time 2025-07-06 13:41:10.301514+02 is later than backend local time 2025-07-06 11:42:07.407792+02 immich_postgres | 2025-07-06 11:42:07.409 CEST [41] LOG: stats_timestamp 2025-07-06 13:41:10.301514+02 is later than collector's time 2025-07-06 11:42:07.409508+02 for database 16384 immich_postgres | Using SSD storage immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2025-07-06 11:50:29.819 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 11:50:29.819 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 13:50:29.937 CEST [1] LOG: starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2025-07-06 13:50:29.939 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2025-07-06 13:50:29.939 CEST [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2025-07-06 13:50:29.952 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2025-07-06 13:50:29.972 CEST [32] LOG: database system was interrupted; last known up at 2025-07-06 13:41:06 CEST immich_postgres | 2025-07-06 13:50:30.229 CEST [32] LOG: database system was not properly shut down; automatic recovery in progress immich_postgres | 2025-07-06 13:50:30.238 CEST [32] LOG: redo starts at 0/697B1B38 immich_postgres | 2025-07-06 13:50:30.246 CEST [32] LOG: invalid record length at 0/697E1258: wanted 24, got 0 immich_postgres | 2025-07-06 13:50:30.246 CEST [32] LOG: redo done at 0/697E1230 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s immich_postgres | 2025-07-06 13:50:30.328 CEST [1] LOG: database system is ready to accept connections immich_postgres | 2025-07-06 11:51:31.533 CEST [39] LOG: statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.529581+02 immich_postgres | 2025-07-06 11:51:31.533 CEST [40] LOG: stats_timestamp 2025-07-06 13:50:34.559781+02 is later than collector's time 2025-07-06 11:51:31.533923+02 for database 0 immich_postgres | 2025-07-06 11:51:31.534 CEST [39] LOG: statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.534868+02 immich_postgres | 2025-07-06 11:51:31.562 CEST [63] LOG: statistics collector's time 2025-07-06 13:50:34.559781+02 is later than backend local time 2025-07-06 11:51:31.562572+02 immich_postgres | 2025-07-06 11:51:31.562 CEST [40] LOG: stats_timestamp 2025-07-06 13:50:34.559781+02 is later than collector's time 2025-07-06 11:51:31.562652+02 for database 16384 immich_postgres | 2025-07-06 11:54:54.874 CEST [39] WARNING: worker took too long to start; canceled immich_postgres | 2025-07-06 11:55:01.137 CEST [64] WARNING: autovacuum worker started without a worker entry immich_postgres | 2025-07-06 11:58:57.811 CEST [61] LOG: could not receive data from client: Connection reset by peer immich_postgres | Using SSD storage immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2025-07-06 12:13:05.875 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 12:13:05.875 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 14:13:06.262 CEST [1] LOG: starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2025-07-06 14:13:06.312 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2025-07-06 14:13:06.312 CEST [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2025-07-06 14:13:06.326 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2025-07-06 14:13:06.414 CEST [33] LOG: database system was interrupted; last known up at 2025-07-06 13:50:30 CEST immich_postgres | 2025-07-06 14:13:06.585 CEST [33] LOG: database system was not properly shut down; automatic recovery in progress immich_postgres | 2025-07-06 14:13:06.592 CEST [33] LOG: redo starts at 0/697E12D0 immich_postgres | 2025-07-06 14:13:06.675 CEST [33] LOG: invalid record length at 0/69874B60: wanted 24, got 0 immich_postgres | 2025-07-06 14:13:06.675 CEST [33] LOG: redo done at 0/69874B38 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.08 s immich_postgres | 2025-07-06 14:13:06.980 CEST [1] LOG: database system is ready to accept connections immich_postgres | 2025-07-06 12:14:08.447 CEST [40] LOG: statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.447607+02 immich_postgres | 2025-07-06 12:14:08.447 CEST [41] LOG: stats_timestamp 2025-07-06 14:13:09.737718+02 is later than collector's time 2025-07-06 12:14:08.447765+02 for database 0 immich_postgres | 2025-07-06 12:14:08.447 CEST [40] LOG: statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.447814+02 immich_postgres | 2025-07-06 12:14:08.450 CEST [64] LOG: statistics collector's time 2025-07-06 14:13:09.737718+02 is later than backend local time 2025-07-06 12:14:08.45091+02 immich_postgres | 2025-07-06 12:14:08.451 CEST [41] LOG: stats_timestamp 2025-07-06 14:13:09.737718+02 is later than collector's time 2025-07-06 12:14:08.451311+02 for database 16384 immich_postgres | Using SSD storage immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2025-07-06 12:19:11.711 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 12:19:11.712 GMT [1] LOG: skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf" immich_postgres | 2025-07-06 14:19:12.019 CEST [1] LOG: starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2025-07-06 14:19:12.076 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2025-07-06 14:19:12.076 CEST [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2025-07-06 14:19:12.090 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2025-07-06 14:19:12.173 CEST [32] LOG: database system was interrupted; last known up at 2025-07-06 14:13:06 CEST immich_postgres | 2025-07-06 14:19:12.352 CEST [32] LOG: database system was not properly shut down; automatic recovery in progress immich_postgres | 2025-07-06 14:19:12.366 CEST [32] LOG: redo starts at 0/69874BD8 immich_postgres | 2025-07-06 14:19:12.437 CEST [32] LOG: invalid record length at 0/698814C0: wanted 24, got 0 immich_postgres | 2025-07-06 14:19:12.437 CEST [32] LOG: redo done at 0/69881498 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.07 s immich_postgres | 2025-07-06 14:19:12.722 CEST [1] LOG: database system is ready to accept connections immich_server | [Nest] 17 - 07/06/2025, 2:19:28 PM LOG [Api:EventRepository] Websocket Connect: al4hfgMkzQyuAHQnAAAB immich_postgres | 2025-07-06 12:20:14.492 CEST [39] LOG: statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.49233+02 immich_postgres | 2025-07-06 12:20:14.492 CEST [40] LOG: stats_timestamp 2025-07-06 14:19:15.232619+02 is later than collector's time 2025-07-06 12:20:14.492529+02 for database 0 immich_postgres | 2025-07-06 12:20:14.492 CEST [39] LOG: statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.492751+02 immich_postgres | 2025-07-06 12:20:14.556 CEST [63] LOG: statistics collector's time 2025-07-06 14:19:15.232619+02 is later than backend local time 2025-07-06 12:20:14.556789+02 immich_postgres | 2025-07-06 12:20:14.556 CEST [40] LOG: stats_timestamp 2025-07-06 14:19:15.232619+02 is later than collector's time 2025-07-06 12:20:14.556859+02 for database 16384 ``` ### Additional information _No response_
Author
Owner

@bo0tzz commented on GitHub (Jul 6, 2025):

Immich doesn't have any ability to hang your system; if that happens, it's because there is a problem with your OS or hardware. You could try limiting Immich's resource usage to see if that helps: https://immich.app/docs/FAQ/#can-i-limit-cpu-and-ram-usage

@bo0tzz commented on GitHub (Jul 6, 2025): Immich doesn't have any ability to hang your system; if that happens, it's because there is a problem with your OS or hardware. You could try limiting Immich's resource usage to see if that helps: https://immich.app/docs/FAQ/#can-i-limit-cpu-and-ram-usage
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#6433