[BUG] Recognize Faces job fails to progress, 'fetch failed' in logs #1828

Closed
opened 2026-02-05 04:06:08 +03:00 by OVERLORD · 2 comments
Owner

Originally created by @HybridSarcasm on GitHub (Dec 18, 2023).

The bug

Manually starting the "Recognize Faces" job seems to 'stall', with the "Waiting" value always increasing. Docker logs show lots of "Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed" stack errors.

The OS that Immich Server is running on

Docker 24.0.7 on Debian 12.4

Version of Immich Server

1.91.3

Version of Immich Mobile App

N/A

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    labels:
      - traefik.enable=true

      # HTTPS entrypoint
      - traefik.http.routers.photos.entryPoints=https
      - traefik.http.routers.photos.tls=true
      - traefik.http.routers.photos.rule=Host(`photos.domain.home`) # Define hostname for HTTPS entrypoint

      # Define an HTTP entryPoint, for purposes of being re-directed to HTTPS
      - traefik.http.routers.photos-redirect.entryPoints=http
      - traefik.http.routers.photos-redirect.rule=Host(`photos.domain.home`)

      # Actual HTTP -> HTTPS redirect for this service
      # References redirect middleware from dynamic.yml trafeik file
      - traefik.http.routers.photos-redirect.middlewares=https-redirect@file

      # LB definition for non-standard HTTP port
      - traefik.http.services.photos.loadbalancer.server.port=3001
      - traefik.http.services.photos.loadbalancer.passhostheader=true

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    # image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /docker/data/immich/pgdata:/var/lib/postgresql/data
    restart: always

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
#UPLOAD_LOCATION=./library
UPLOAD_LOCATION=/mnt/photos

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Logon to Immich web admin profile
2. Go to "Jobs"
3. Start "Recognize Faces" job
4. Observe docker logs for immich-microservices display the following statement for various objects IDs:
[Nest] 7  - 12/18/2023, 10:47:58 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 12/18/2023, 10:47:58 AM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:246:23)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/18/2023, 10:47:58 AM   ERROR [JobService] Object:

Additional information

No response

Originally created by @HybridSarcasm on GitHub (Dec 18, 2023). ### The bug Manually starting the "Recognize Faces" job seems to 'stall', with the "Waiting" value always increasing. Docker logs show lots of "Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed" stack errors. ### The OS that Immich Server is running on Docker 24.0.7 on Debian 12.4 ### Version of Immich Server 1.91.3 ### Version of Immich Mobile App N/A ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ["start.sh", "immich"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always labels: - traefik.enable=true # HTTPS entrypoint - traefik.http.routers.photos.entryPoints=https - traefik.http.routers.photos.tls=true - traefik.http.routers.photos.rule=Host(`photos.domain.home`) # Define hostname for HTTPS entrypoint # Define an HTTP entryPoint, for purposes of being re-directed to HTTPS - traefik.http.routers.photos-redirect.entryPoints=http - traefik.http.routers.photos-redirect.rule=Host(`photos.domain.home`) # Actual HTTP -> HTTPS redirect for this service # References redirect middleware from dynamic.yml trafeik file - traefik.http.routers.photos-redirect.middlewares=https-redirect@file # LB definition for non-standard HTTP port - traefik.http.services.photos.loadbalancer.server.port=3001 - traefik.http.services.photos.loadbalancer.passhostheader=true immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: container_name: immich_postgres # image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 image: tensorchord/pgvecto-rs:pg14-v0.1.11 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - /docker/data/immich/pgdata:/var/lib/postgresql/data restart: always ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored #UPLOAD_LOCATION=./library UPLOAD_LOCATION=/mnt/photos # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=some-random-text DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. Logon to Immich web admin profile 2. Go to "Jobs" 3. Start "Recognize Faces" job 4. Observe docker logs for immich-microservices display the following statement for various objects IDs: [Nest] 7 - 12/18/2023, 10:47:58 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 12/18/2023, 10:47:58 AM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11730:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:246:23) at async /usr/src/app/dist/domain/job/job.service.js:113:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) [Nest] 7 - 12/18/2023, 10:47:58 AM ERROR [JobService] Object: ``` ### Additional information _No response_
Author
Owner

@bo0tzz commented on GitHub (Dec 18, 2023):

You don't have the machine learning container in your docker-compose.yml, are you using an external one?

@bo0tzz commented on GitHub (Dec 18, 2023): You don't have the machine learning container in your docker-compose.yml, are you using an external one?
Author
Owner

@HybridSarcasm commented on GitHub (Dec 18, 2023):

You don't have the machine learning container in your docker-compose.yml, are you using an external one?

Yes! And my ML settings had the default hostname instead of the remote hostname. Updated with correct hostname and all is well again. Thank you!

@HybridSarcasm commented on GitHub (Dec 18, 2023): > You don't have the machine learning container in your docker-compose.yml, are you using an external one? Yes! And my ML settings had the default hostname instead of the remote hostname. Updated with correct hostname and all is well again. Thank you!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1828