immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 12 Packages Projects Releases 10 Wiki Activity

[BUG] Connection terminated due to connection timeout when re-running "Extract Metadata" job #1526

New Issue
Closed
opened 2026-02-05 02:14:28 +03:00 by OVERLORD · 10 comments
OVERLORD commented 2026-02-05 02:14:28 +03:00
Owner
Copy Link

Originally created by @florihupf on GitHub (Oct 29, 2023).

The bug

Just upgraded to 1.83.0 but this also happened with 1.82.1: When running the "Extract Metadata" job I get a lot of connection errors (every 1-2s).

If I let the job continue I end up with missing thumbnails.

Example logs:

[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:57:01 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)

I have tons of assets (> 120k). Other than that, everything works fine.

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.83.0

Version of Immich Mobile App

v1.82.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"
name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /mnt/VM/immich/model_cache:/cache
    env_file:
      - immich.env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - immich.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - /mnt/VM/immich/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - immich.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /mnt/VM/immich/db/postgresql/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

# This env file goes into both: docker-compose.yml and the various containers.

DB_HOSTNAME=immich_postgres
DB_USERNAME=xxx
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich

TYPESENSE_API_KEY=some-text
REDIS_HOSTNAME=immich_redis

UPLOAD_LOCATION=/mnt/Data/bilder
LOG_LEVEL=simple

JWT_SECRET=xxx

PUBLIC_LOGIN_PAGE_MESSAGE="Willkommen zum Bilder - Gedoens!"

IMMICH_VERSION=v1.83.0

Reproduction steps

1. Run the Metadata Extraction Job
2. Wait for ~20s: Then errors are logged every 1-2 seconds.

Additional information

Let me know if I can provide additional information. And thanks a LOT for immich. :-)

Originally created by @florihupf on GitHub (Oct 29, 2023). ### The bug Just upgraded to 1.83.0 but this also happened with 1.82.1: When running the "Extract Metadata" job I get a lot of connection errors (every 1-2s). If I let the job continue I end up with missing thumbnails. Example logs: ``` [Nest] 7 - 10/29/2023, 5:56:59 AM ERROR [ExceptionsHandler] Connection terminated due to connection timeout Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 10/29/2023, 5:56:59 AM ERROR [ExceptionsHandler] Connection terminated due to connection timeout Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 10/29/2023, 5:56:59 AM ERROR [ExceptionsHandler] Connection terminated due to connection timeout Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 10/29/2023, 5:57:01 AM ERROR [ExceptionsHandler] Connection terminated due to connection timeout Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) ``` I have tons of assets (> 120k). Other than that, everything works fine. ### The OS that Immich Server is running on Debian ### Version of Immich Server v1.83.0 ### Version of Immich Mobile App v1.82.1 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - immich.env depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - immich.env depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - /mnt/VM/immich/model_cache:/cache env_file: - immich.env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - immich.env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - /mnt/VM/immich/tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - immich.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - /mnt/VM/immich/db/postgresql/pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 depends_on: - immich-server - immich-web restart: always ``` ### Your .env content ```Shell # This env file goes into both: docker-compose.yml and the various containers. DB_HOSTNAME=immich_postgres DB_USERNAME=xxx DB_PASSWORD=xxx DB_DATABASE_NAME=immich TYPESENSE_API_KEY=some-text REDIS_HOSTNAME=immich_redis UPLOAD_LOCATION=/mnt/Data/bilder LOG_LEVEL=simple JWT_SECRET=xxx PUBLIC_LOGIN_PAGE_MESSAGE="Willkommen zum Bilder - Gedoens!" IMMICH_VERSION=v1.83.0 ``` ### Reproduction steps ```bash 1. Run the Metadata Extraction Job 2. Wait for ~20s: Then errors are logged every 1-2 seconds. ``` ### Additional information Let me know if I can provide additional information. And thanks a LOT for immich. :-)
OVERLORD commented 2026-02-05 02:14:40 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Oct 29, 2023):

Hello, what is your server spec? is Immich hosting on an SSD or HDD?

@alextran1502 commented on GitHub (Oct 29, 2023): Hello, what is your server spec? is Immich hosting on an SSD or HDD?
OVERLORD commented 2026-02-05 02:14:42 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 29, 2023):

I am running the actual images on HDD, but the DB and everything else is on a SSD using ZFS.

Only tricky part might be that I use a jail under Truenas Scale to get the full docker compose experience: https://github.com/Jip-Hop/jailmaker

@florihupf commented on GitHub (Oct 29, 2023): I am running the actual images on HDD, but the DB and everything else is on a SSD using ZFS. Only tricky part might be that I use a jail under Truenas Scale to get the full docker compose experience: https://github.com/Jip-Hop/jailmaker
OVERLORD commented 2026-02-05 02:14:49 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Oct 29, 2023):

@florihupf Do you think following this guide to install would be helpful? https://immich.app/docs/install/truenas

@alextran1502 commented on GitHub (Oct 29, 2023): @florihupf Do you think following this guide to install would be helpful? https://immich.app/docs/install/truenas
OVERLORD commented 2026-02-05 02:14:53 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 29, 2023):

Thanks for the pointer, but I prefer a raw docker compose environment due to stability.

I am running immich for a long time with no issues so far.

@florihupf commented on GitHub (Oct 29, 2023): Thanks for the pointer, but I prefer a raw docker compose environment due to stability. I am running immich for a long time with no issues so far.
OVERLORD commented 2026-02-05 02:14:56 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 29, 2023):

Is there a way to find more debug info on what is going on?

@florihupf commented on GitHub (Oct 29, 2023): Is there a way to find more debug info on what is going on?
OVERLORD commented 2026-02-05 02:14:59 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Oct 29, 2023):

From my understanding the issue of not able to write to the database is because disk IO has maxed out. What is the job concurrency do you have for Metadata extraction job? Perhaps lowering it would help solve the issue.

The database mapping from your docker compose, is it on an SSD or HDD?

@alextran1502 commented on GitHub (Oct 29, 2023): From my understanding the issue of not able to write to the database is because disk IO has maxed out. What is the job concurrency do you have for Metadata extraction job? Perhaps lowering it would help solve the issue. The database mapping from your docker compose, is it on an SSD or HDD?
OVERLORD commented 2026-02-05 02:15:01 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 29, 2023):

I had the default concurrency of 5 and the database is on an SSD. Let me try to lower the concurrency.

@florihupf commented on GitHub (Oct 29, 2023): I had the default concurrency of 5 and the database is on an SSD. Let me try to lower the concurrency.
OVERLORD commented 2026-02-05 02:15:09 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 29, 2023):

Lowering the concurrency seems to help, it is still going, will report when things are done / have failed: fingers crossed :-)

@florihupf commented on GitHub (Oct 29, 2023): Lowering the concurrency seems to help, it is still going, will report when things are done / have failed: *fingers crossed* :-)
OVERLORD commented 2026-02-05 02:15:12 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Oct 30, 2023):

Hello, did lower the concurrency help with the issue?

@alextran1502 commented on GitHub (Oct 30, 2023): Hello, did lower the concurrency help with the issue?
OVERLORD commented 2026-02-05 02:15:13 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Oct 30, 2023):

Hi Alex, yeah, looks like it went through with no issues at concurrency "1" after 11h for about 130k assets. Thanks for the helpful pointer!

@florihupf commented on GitHub (Oct 30, 2023): Hi Alex, yeah, looks like it went through with no issues at concurrency "1" after 11h for about 130k assets. Thanks for the helpful pointer!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1526
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?