[BUG] Failed to search (500 - Internal Server Error) #1999

Closed
opened 2026-02-05 04:42:41 +03:00 by OVERLORD · 7 comments
Owner

Originally created by @LIHAQ on GitHub (Jan 17, 2024).

The bug

When searching for photos, 500 errors will be generated .

immich_server logs:
[Error: Machine learning request for clip failed with status 500: Internal Server Error
at MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async SearchService.search (/usr/src/app/dist/domain/search/search.service.js:65:35)] Error: Machine learning request for clip failed with status 500: Internal Server Error

The OS that Immich Server is running on

Ubuntu 20.04.1 LTS

Version of Immich Server

1.92.1

Version of Immich Mobile App

1.92.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.yml
      service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1.
2.
3.
...

Additional information

No response

Originally created by @LIHAQ on GitHub (Jan 17, 2024). ### The bug When searching for photos, 500 errors will be generated . immich_server logs: [Error: Machine learning request for clip failed with status 500: Internal Server Error at MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async SearchService.search (/usr/src/app/dist/domain/search/search.service.js:65:35)] Error: Machine learning request for clip failed with status 500: Internal Server Error ### The OS that Immich Server is running on Ubuntu 20.04.1 LTS ### Version of Immich Server 1.92.1 ### Version of Immich Mobile App 1.92.1 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} extends: file: hwaccel.yml service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc restart: always database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. 2. 3. ... ``` ### Additional information _No response_
Author
Owner

@mertalev commented on GitHub (Jan 17, 2024):

Can you share the ML service logs?

@mertalev commented on GitHub (Jan 17, 2024): Can you share the ML service logs?
Author
Owner

@LIHAQ commented on GitHub (Jan 17, 2024):

Can you share the ML service logs?

[01/17/24 07:22:59] INFO Listening at: http://0.0.0.0:3003 (9)
[01/17/24 07:22:59] INFO Using worker: app.config.CustomUvicornWorker
[01/17/24 07:22:59] INFO Booting worker with pid: 13
[01/17/24 07:23:04] INFO Created in-memory cache with unloading after 300s
of inactivity.
[01/17/24 07:23:04] INFO Initialized request thread pool with 4 threads.
[01/17/24 07:28:38] INFO Starting gunicorn 21.2.0
[01/17/24 07:28:38] INFO Listening at: http://0.0.0.0:3003 (9)
[01/17/24 07:28:38] INFO Using worker: app.config.CustomUvicornWorker
[01/17/24 07:28:38] INFO Booting worker with pid: 13
[01/17/24 07:28:44] INFO Created in-memory cache with unloading after 300s
of inactivity.
[01/17/24 07:28:44] INFO Initialized request thread pool with 4 threads.
[01/17/24 07:29:25] INFO Loading clip model 'nllb-clip-base-siglip__v1' to
memory

@LIHAQ commented on GitHub (Jan 17, 2024): > Can you share the ML service logs? [01/17/24 07:22:59] INFO Listening at: http://0.0.0.0:3003 (9) [01/17/24 07:22:59] INFO Using worker: app.config.CustomUvicornWorker [01/17/24 07:22:59] INFO Booting worker with pid: 13 [01/17/24 07:23:04] INFO Created in-memory cache with unloading after 300s of inactivity. [01/17/24 07:23:04] INFO Initialized request thread pool with 4 threads. [01/17/24 07:28:38] INFO Starting gunicorn 21.2.0 [01/17/24 07:28:38] INFO Listening at: http://0.0.0.0:3003 (9) [01/17/24 07:28:38] INFO Using worker: app.config.CustomUvicornWorker [01/17/24 07:28:38] INFO Booting worker with pid: 13 [01/17/24 07:28:44] INFO Created in-memory cache with unloading after 300s of inactivity. [01/17/24 07:28:44] INFO Initialized request thread pool with 4 threads. [01/17/24 07:29:25] INFO Loading clip model 'nllb-clip-base-siglip__v1' to memory
Author
Owner

@CrossNiGao commented on GitHub (Jan 23, 2024):

i got the same error with the same CLIP model. i check the log , the model is loading again and again but cpu load is very low .
may be the model not running actually.
...
[01/23/24 09:22:41] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:43] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:45] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:48] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:50] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:53] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
memory
[01/23/24 09:22:55] INFO Loading clip model 'nllb-clip-large-siglip__v1' to
...
so i try models , results :

error:
nllb-clip-large-siglip__v1,XLM-Roberta-Large-ViT-H-14__frozen_laion5b_s13b_b90k

normal:
LABSE-Vit-L-14 is normal (multilingual, and Chinese search works well)
ViT-B-32__openai
ViT-g-14__laion2b-s12b-b42k

@CrossNiGao commented on GitHub (Jan 23, 2024): i got the same error with the same CLIP model. i check the log , the model is loading again and again but cpu load is very low . may be the model not running actually. ... [01/23/24 09:22:41] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:43] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:45] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:48] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:50] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:53] INFO Loading clip model 'nllb-clip-large-siglip__v1' to memory [01/23/24 09:22:55] INFO Loading clip model 'nllb-clip-large-siglip__v1' to ... so i try models , results : error: nllb-clip-large-siglip__v1,XLM-Roberta-Large-ViT-H-14__frozen_laion5b_s13b_b90k normal: LABSE-Vit-L-14 is normal (multilingual, and Chinese search works well) ViT-B-32__openai ViT-g-14__laion2b-s12b-b42k
Author
Owner

@mariomaz87 commented on GitHub (Jan 25, 2024):

I'm having the same issue on a fresh install

@mariomaz87 commented on GitHub (Jan 25, 2024): I'm having the same issue on a fresh install
Author
Owner

@UntaggedRui commented on GitHub (Jan 27, 2024):

Same for me.


immich_machine_learning  | [01/27/24 13:26:54] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:15] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:19] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:21] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:23] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:24] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_machine_learning  | [01/27/24 13:27:26] INFO     Loading clip model 'nllb-clip-large-siglip__v1' to
immich_machine_learning  |                              memory
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:25 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset c0c9b6b0-adcb-4b7c-bb90-2616d385426f
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:26 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset c0c9b6b0-adcb-4b7c-bb90-2616d385426f
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:26 PM   ERROR [JobService] Unable to run job handler (smartSearch/clip-encode): Error: Machine learning request for clip failed with status 500: Internal Server Error
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:26 PM   ERROR [JobService] Error: Machine learning request for clip failed with status 500: Internal Server Error
immich_microservices  |     at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19)
immich_microservices  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_microservices  |     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:64:31)
immich_microservices  |     at async /usr/src/app/dist/domain/job/job.service.js:116:37
immich_microservices  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_microservices  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:26 PM   ERROR [JobService] Object:
immich_microservices  | {
immich_microservices  |   "id": "339c0302-e315-4a9a-b495-c036a5d10ee5",
immich_microservices  |   "source": "upload"
immich_microservices  | }
immich_microservices  |
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:28 PM   ERROR [JobService] Unable to run job handler (smartSearch/clip-encode): Error: Machine learning request for clip failed with status 500: Internal Server Error
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:28 PM   ERROR [JobService] Error: Machine learning request for clip failed with status 500: Internal Server Error
immich_microservices  |     at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19)
immich_microservices  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_microservices  |     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:64:31)
immich_microservices  |     at async /usr/src/app/dist/domain/job/job.service.js:116:37
immich_microservices  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_microservices  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_microservices  | [Nest] 7  - 01/27/2024, 1:27:28 PM   ERROR [JobService] Object:
immich_microservices  | {
immich_microservices  |   "id": "c0c9b6b0-adcb-4b7c-bb90-2616d385426f",
immich_microservices  |   "source": "upload"
immich_microservices  | }
immich_microservices  |

@UntaggedRui commented on GitHub (Jan 27, 2024): Same for me. ``` immich_machine_learning | [01/27/24 13:26:54] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:15] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:19] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:21] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:23] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:24] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory immich_machine_learning | [01/27/24 13:27:26] INFO Loading clip model 'nllb-clip-large-siglip__v1' to immich_machine_learning | memory ``` ``` immich_microservices | [Nest] 7 - 01/27/2024, 1:27:25 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset c0c9b6b0-adcb-4b7c-bb90-2616d385426f immich_microservices | [Nest] 7 - 01/27/2024, 1:27:26 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset c0c9b6b0-adcb-4b7c-bb90-2616d385426f immich_microservices | [Nest] 7 - 01/27/2024, 1:27:26 PM ERROR [JobService] Unable to run job handler (smartSearch/clip-encode): Error: Machine learning request for clip failed with status 500: Internal Server Error immich_microservices | [Nest] 7 - 01/27/2024, 1:27:26 PM ERROR [JobService] Error: Machine learning request for clip failed with status 500: Internal Server Error immich_microservices | at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19) immich_microservices | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) immich_microservices | at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:64:31) immich_microservices | at async /usr/src/app/dist/domain/job/job.service.js:116:37 immich_microservices | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_microservices | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_microservices | [Nest] 7 - 01/27/2024, 1:27:26 PM ERROR [JobService] Object: immich_microservices | { immich_microservices | "id": "339c0302-e315-4a9a-b495-c036a5d10ee5", immich_microservices | "source": "upload" immich_microservices | } immich_microservices | immich_microservices | [Nest] 7 - 01/27/2024, 1:27:28 PM ERROR [JobService] Unable to run job handler (smartSearch/clip-encode): Error: Machine learning request for clip failed with status 500: Internal Server Error immich_microservices | [Nest] 7 - 01/27/2024, 1:27:28 PM ERROR [JobService] Error: Machine learning request for clip failed with status 500: Internal Server Error immich_microservices | at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:22:19) immich_microservices | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) immich_microservices | at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:64:31) immich_microservices | at async /usr/src/app/dist/domain/job/job.service.js:116:37 immich_microservices | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_microservices | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_microservices | [Nest] 7 - 01/27/2024, 1:27:28 PM ERROR [JobService] Object: immich_microservices | { immich_microservices | "id": "c0c9b6b0-adcb-4b7c-bb90-2616d385426f", immich_microservices | "source": "upload" immich_microservices | } immich_microservices | ```
Author
Owner

@mertalev commented on GitHub (Jan 28, 2024):

Fixed via #6695

@mertalev commented on GitHub (Jan 28, 2024): Fixed via #6695
Author
Owner

@shackrock commented on GitHub (Apr 10, 2024):

For what it's worth I have these errors on fresh install, referenced here: https://github.com/immich-app/immich/discussions/8624

@shackrock commented on GitHub (Apr 10, 2024): For what it's worth I have these errors on fresh install, referenced here: https://github.com/immich-app/immich/discussions/8624
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1999