immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

CUDA cannot be called properly. #7858

New Issue
Closed
opened 2026-02-05 13:21:15 +03:00 by OVERLORD · 1 comment
OVERLORD commented 2026-02-05 13:21:15 +03:00
Owner
Copy Link

Originally created by @bb159 on GitHub (Nov 18, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

After configuring CUDA for immich, I encountered an error.

The OS that Immich Server is running on

ubuntu24

Version of Immich Server

v2.2.3

Version of Immich Mobile App

v2.2.3

Platform with the issue

  • Server
  • Web
  • Mobile

Device make and model

No response

Your docker-compose.yml content

root@bb:/opt/immich-app# cat ./docker-compose.yml                                                                                                                                                   #
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.nju.edu.cn/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: nvenc # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - /1T:/1T
      - /data:/linux
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
      - ./geodata:/build/geodata
      - ./i18n-iso-countries/langs:/usr/src/app/node_modules/i18n-iso-countries/langs
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.nju.edu.cn/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: cuda # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - ./model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image:  docker.io/valkey/valkey:8@sha256:81db6d39e1bba3b3ff32bd3a1b19a6d69690f94a3954ec131277b9a26b95b3aa
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.nju.edu.cn/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/data/pictures

# The location where your database files are stored. Network shares are not supported for the database
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Asia/Shanghai

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v2.2.3

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

文本识别
使用机器学习识别图片中的文本

启用文本识别
如果禁用，则不会对图像编码以用于文本识别。

文本识别模型
服务器模型比移动模型更准确，但需要更长的时间来处理和使用更多的内存。

PP-OCRv5_server
最低检测分数
要检测的文本的最小置信度分数为0-1。较低的值将检测到更多的文本，但可能会导致误报。

0.4
最低识别分数
检测到的文本的最小置信度得分为0-1。较低的值将识别更多的文本，但可能会导致误报。

0.5
最高分辨率
高于此分辨率的预览将调整大小，同时保持纵横比。更高的值更准确，但处理时间更长，占用更多内存。

1280

Relevant log output

immich_machine_learning  | 2025-11-18 15:15:08.065625874 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 2000: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=44be8fb8dcfd ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=382 ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size); 
immich_machine_learning  | 2025-11-18 15:15:08.065644679 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv.2' Status Message: CUDNN failure 2000: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=44be8fb8dcfd ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=382 ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size); 
immich_machine_learning  | [11/18/25 15:15:08] ERROR    Exception in ASGI application
immich_machine_learning  |                                             
immich_machine_learning  |                              ╭─────── Traceback (most recent call last) ───────╮
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:177 in predict       │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   174 │   │   inputs = text                     │
immich_machine_learning  |                              │   175 │   else:                                 │
immich_machine_learning  |                              │   176 │   │   raise HTTPException(400, "Either  │
immich_machine_learning  |                              │ ❱ 177 │   response = await run_inference(inputs │
immich_machine_learning  |                              │   178 │   return ORJSONResponse(response)       │
immich_machine_learning  |                              │   179                                           │
immich_machine_learning  |                              │   180                                           │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:202 in run_inference │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   199 │   │   response[entry["task"]] = output  │
immich_machine_learning  |                              │   200 │                                         │
immich_machine_learning  |                              │   201 │   without_deps, with_deps = entries     │
immich_machine_learning  |                              │ ❱ 202 │   await asyncio.gather(*[_run_inference │
immich_machine_learning  |                              │   203 │   if with_deps:                         │
immich_machine_learning  |                              │   204 │   │   await asyncio.gather(*[_run_infer │
immich_machine_learning  |                              │   205 │   if isinstance(payload, Image):        │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:197 in               │
immich_machine_learning  |                              │ _run_inference                                  │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   194 │   │   │   │   message = f"Task {entry[' │
immich_machine_learning  |                              │       output of {dep}"                          │
immich_machine_learning  |                              │   195 │   │   │   │   raise HTTPException(400,  │
immich_machine_learning  |                              │   196 │   │   model = await load(model)         │
immich_machine_learning  |                              │ ❱ 197 │   │   output = await run(model.predict, │
immich_machine_learning  |                              │   198 │   │   outputs[model.identity] = output  │
immich_machine_learning  |                              │   199 │   │   response[entry["task"]] = output  │
immich_machine_learning  |                              │   200                                           │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/main.py:215 in run           │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │   212 │   if thread_pool is None:               │
immich_machine_learning  |                              │   213 │   │   return func(*args, **kwargs)      │
immich_machine_learning  |                              │   214 │   partial_func = partial(func, *args, * │
immich_machine_learning  |                              │ ❱ 215 │   return await asyncio.get_running_loop │
immich_machine_learning  |                              │   216                                           │
immich_machine_learning  |                              │   217                                           │
immich_machine_learning  |                              │   218 async def load(model: InferenceModel) ->  │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/local/lib/python3.11/concurrent/futures/th │
immich_machine_learning  |                              │ read.py:58 in run                               │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/models/base.py:60 in predict │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │    57 │   │   self.load()                       │
immich_machine_learning  |                              │    58 │   │   if model_kwargs:                  │
immich_machine_learning  |                              │    59 │   │   │   self.configure(**model_kwargs │
immich_machine_learning  |                              │ ❱  60 │   │   return self._predict(*inputs)     │
immich_machine_learning  |                              │    61 │                                         │
immich_machine_learning  |                              │    62 │   @abstractmethod                       │
immich_machine_learning  |                              │    63 │   def _predict(self, *inputs: Any, **mo │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/models/ocr/detection.py:70   │
immich_machine_learning  |                              │ in _predict                                     │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │    67 │   │   w, h = inputs.size                │
immich_machine_learning  |                              │    68 │   │   if w < 32 or h < 32:              │
immich_machine_learning  |                              │    69 │   │   │   return self._empty            │
immich_machine_learning  |                              │ ❱  70 │   │   out = self.session.run(None, {"x" │
immich_machine_learning  |                              │    71 │   │   boxes, scores = self.postprocess( │
immich_machine_learning  |                              │    72 │   │   if len(boxes) == 0:               │
immich_machine_learning  |                              │    73 │   │   │   return self._empty            │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /usr/src/immich_ml/sessions/ort.py:51 in run    │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │    48 │   │   input_feed: dict[str, NDArray[np. │
immich_machine_learning  |                              │    49 │   │   run_options: Any = None,          │
immich_machine_learning  |                              │    50 │   ) -> list[NDArray[np.float32]]:       │
immich_machine_learning  |                              │ ❱  51 │   │   outputs: list[NDArray[np.float32] │
immich_machine_learning  |                              │       run_options)                              │
immich_machine_learning  |                              │    52 │   │   return outputs                    │
immich_machine_learning  |                              │    53 │                                         │
immich_machine_learning  |                              │    54 │   @property                             │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
immich_machine_learning  |                              │ ime/capi/onnxruntime_inference_collection.py:22 │
immich_machine_learning  |                              │ 0 in run                                        │
immich_machine_learning  |                              │                                                 │
immich_machine_learning  |                              │    217 │   │   if not output_names:             │
immich_machine_learning  |                              │    218 │   │   │   output_names = [output.name  │
immich_machine_learning  |                              │    219 │   │   try:                             │
immich_machine_learning  |                              │ ❱  220 │   │   │   return self._sess.run(output │
immich_machine_learning  |                              │    221 │   │   except C.EPFail as err:          │
immich_machine_learning  |                              │    222 │   │   │   if self._enable_fallback:    │
immich_machine_learning  |                              │    223 │   │   │   │   print(f"EP Error: {err!s │
immich_machine_learning  |                              ╰─────────────────────────────────────────────────╯
immich_machine_learning  |                              Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero
immich_machine_learning  |                              status code returned while running FusedConv node.
immich_machine_learning  |                              Name:'Conv.2' Status Message: CUDNN failure 2000:
immich_machine_learning  |                              CUDNN_STATUS_BAD_PARAM ; GPU=0 ;
immich_machine_learning  |                              hostname=44be8fb8dcfd ;
immich_machine_learning  |                              file=/onnxruntime_src/onnxruntime/core/providers/cu
immich_machine_learning  |                              da/nn/conv.cc ; line=382 ;
immich_machine_learning  |                              expr=cudnnFindConvolutionForwardAlgorithmEx(
immich_machine_learning  |                              GetCudnnHandle(context), s_.x_tensor, s_.x_data,
immich_machine_learning  |                              s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor,
immich_machine_learning  |                              s_.y_data, 1, &algo_count, &perf,
immich_machine_learning  |                              algo_search_workspace.get(), max_ws_size);
immich_server            | [Nest] 7  - 11/18/2025, 11:15:08 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed with status 500: Internal Server Error
immich_server            | [Nest] 7  - 11/18/2025, 11:15:08 PM   ERROR [Microservices:{"id":"c0f25065-03a9-4370-9097-ca6beb6e5b7a"}] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr":{"detection":{"modelName":"PP-OCRv5_server","options":{"minScore":0.5,"maxResolution":1280}},"recognition":{"modelName":"PP-OCRv5_server","options":{"minScore":0.8}}}}' failed for all URLs
immich_server            | Error: Machine learning request '{"ocr":{"detection":{"modelName":"PP-OCRv5_server","options":{"minScore":0.5,"maxResolution":1280}},"recognition":{"modelName":"PP-OCRv5_server","options":{"minScore":0.8}}}}' failed for all URLs
immich_server            |     at MachineLearningRepository.predict (/usr/src/app/server/dist/repositories/machine-learning.repository.js:117:15)
immich_server            |     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
immich_server            |     at async MachineLearningRepository.ocr (/usr/src/app/server/dist/repositories/machine-learning.repository.js:150:26)
immich_server            |     at async OcrService.handleOcr (/usr/src/app/server/dist/services/ocr.service.js:52:28)
immich_server            |     at async JobService.onJobRun (/usr/src/app/server/dist/services/job.service.js:199:30)
immich_server            |     at async EventRepository.onEvent (/usr/src/app/server/dist/repositories/event.repository.js:91:13)
immich_server            |     at async /usr/src/app/server/node_modules/.pnpm/bullmq@5.61.2/node_modules/bullmq/dist/cjs/classes/worker.js:528:32

Additional information

No response

Originally created by @bb159 on GitHub (Nov 18, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug After configuring CUDA for immich, I encountered an error. ### The OS that Immich Server is running on ubuntu24 ### Version of Immich Server v2.2.3 ### Version of Immich Mobile App v2.2.3 ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Device make and model _No response_ ### Your docker-compose.yml content ```YAML root@bb:/opt/immich-app# cat ./docker-compose.yml # # WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose # # Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. name: immich services: immich-server: container_name: immich_server image: ghcr.nju.edu.cn/immich-app/immich-server:${IMMICH_VERSION:-release} extends: file: hwaccel.transcoding.yml service: nvenc # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - /1T:/1T - /data:/linux - ${UPLOAD_LOCATION}:/data - /etc/localtime:/etc/localtime:ro - ./geodata:/build/geodata - ./i18n-iso-countries/langs:/usr/src/app/node_modules/i18n-iso-countries/langs env_file: - .env ports: - '2283:2283' depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.nju.edu.cn/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration file: hwaccel.ml.yml service: cuda # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - ./model-cache:/cache env_file: - .env restart: always healthcheck: disable: false redis: container_name: immich_redis image: docker.io/valkey/valkey:8@sha256:81db6d39e1bba3b3ff32bd3a1b19a6d69690f94a3954ec131277b9a26b95b3aa healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: ghcr.nju.edu.cn/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs # DB_STORAGE_TYPE: 'HDD' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data shm_size: 128mb restart: always volumes: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/data/pictures # The location where your database files are stored. Network shares are not supported for the database DB_DATA_LOCATION=./postgres # To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List TZ=Asia/Shanghai # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=v2.2.3 # Connection secret for postgres. You should change it to a random password # Please use only the characters `A-Za-z0-9`, without special characters or spaces DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` ### Reproduction steps 文本识别 使用机器学习识别图片中的文本 启用文本识别 如果禁用，则不会对图像编码以用于文本识别。 文本识别模型 服务器模型比移动模型更准确，但需要更长的时间来处理和使用更多的内存。 PP-OCRv5_server 最低检测分数 要检测的文本的最小置信度分数为0-1。较低的值将检测到更多的文本，但可能会导致误报。 0.4 最低识别分数 检测到的文本的最小置信度得分为0-1。较低的值将识别更多的文本，但可能会导致误报。 0.5 最高分辨率 高于此分辨率的预览将调整大小，同时保持纵横比。更高的值更准确，但处理时间更长，占用更多内存。 1280 ### Relevant log output ```shell immich_machine_learning | 2025-11-18 15:15:08.065625874 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 2000: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=44be8fb8dcfd ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=382 ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size); immich_machine_learning | 2025-11-18 15:15:08.065644679 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv.2' Status Message: CUDNN failure 2000: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=44be8fb8dcfd ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=382 ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size); immich_machine_learning | [11/18/25 15:15:08] ERROR Exception in ASGI application immich_machine_learning | immich_machine_learning | ╭─────── Traceback (most recent call last) ───────╮ immich_machine_learning | │ /usr/src/immich_ml/main.py:177 in predict │ immich_machine_learning | │ │ immich_machine_learning | │ 174 │ │ inputs = text │ immich_machine_learning | │ 175 │ else: │ immich_machine_learning | │ 176 │ │ raise HTTPException(400, "Either │ immich_machine_learning | │ ❱ 177 │ response = await run_inference(inputs │ immich_machine_learning | │ 178 │ return ORJSONResponse(response) │ immich_machine_learning | │ 179 │ immich_machine_learning | │ 180 │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/main.py:202 in run_inference │ immich_machine_learning | │ │ immich_machine_learning | │ 199 │ │ response[entry["task"]] = output │ immich_machine_learning | │ 200 │ │ immich_machine_learning | │ 201 │ without_deps, with_deps = entries │ immich_machine_learning | │ ❱ 202 │ await asyncio.gather(*[_run_inference │ immich_machine_learning | │ 203 │ if with_deps: │ immich_machine_learning | │ 204 │ │ await asyncio.gather(*[_run_infer │ immich_machine_learning | │ 205 │ if isinstance(payload, Image): │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/main.py:197 in │ immich_machine_learning | │ _run_inference │ immich_machine_learning | │ │ immich_machine_learning | │ 194 │ │ │ │ message = f"Task {entry[' │ immich_machine_learning | │ output of {dep}" │ immich_machine_learning | │ 195 │ │ │ │ raise HTTPException(400, │ immich_machine_learning | │ 196 │ │ model = await load(model) │ immich_machine_learning | │ ❱ 197 │ │ output = await run(model.predict, │ immich_machine_learning | │ 198 │ │ outputs[model.identity] = output │ immich_machine_learning | │ 199 │ │ response[entry["task"]] = output │ immich_machine_learning | │ 200 │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/main.py:215 in run │ immich_machine_learning | │ │ immich_machine_learning | │ 212 │ if thread_pool is None: │ immich_machine_learning | │ 213 │ │ return func(*args, **kwargs) │ immich_machine_learning | │ 214 │ partial_func = partial(func, *args, * │ immich_machine_learning | │ ❱ 215 │ return await asyncio.get_running_loop │ immich_machine_learning | │ 216 │ immich_machine_learning | │ 217 │ immich_machine_learning | │ 218 async def load(model: InferenceModel) -> │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/local/lib/python3.11/concurrent/futures/th │ immich_machine_learning | │ read.py:58 in run │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/models/base.py:60 in predict │ immich_machine_learning | │ │ immich_machine_learning | │ 57 │ │ self.load() │ immich_machine_learning | │ 58 │ │ if model_kwargs: │ immich_machine_learning | │ 59 │ │ │ self.configure(**model_kwargs │ immich_machine_learning | │ ❱ 60 │ │ return self._predict(*inputs) │ immich_machine_learning | │ 61 │ │ immich_machine_learning | │ 62 │ @abstractmethod │ immich_machine_learning | │ 63 │ def _predict(self, *inputs: Any, **mo │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/models/ocr/detection.py:70 │ immich_machine_learning | │ in _predict │ immich_machine_learning | │ │ immich_machine_learning | │ 67 │ │ w, h = inputs.size │ immich_machine_learning | │ 68 │ │ if w < 32 or h < 32: │ immich_machine_learning | │ 69 │ │ │ return self._empty │ immich_machine_learning | │ ❱ 70 │ │ out = self.session.run(None, {"x" │ immich_machine_learning | │ 71 │ │ boxes, scores = self.postprocess( │ immich_machine_learning | │ 72 │ │ if len(boxes) == 0: │ immich_machine_learning | │ 73 │ │ │ return self._empty │ immich_machine_learning | │ │ immich_machine_learning | │ /usr/src/immich_ml/sessions/ort.py:51 in run │ immich_machine_learning | │ │ immich_machine_learning | │ 48 │ │ input_feed: dict[str, NDArray[np. │ immich_machine_learning | │ 49 │ │ run_options: Any = None, │ immich_machine_learning | │ 50 │ ) -> list[NDArray[np.float32]]: │ immich_machine_learning | │ ❱ 51 │ │ outputs: list[NDArray[np.float32] │ immich_machine_learning | │ run_options) │ immich_machine_learning | │ 52 │ │ return outputs │ immich_machine_learning | │ 53 │ │ immich_machine_learning | │ 54 │ @property │ immich_machine_learning | │ │ immich_machine_learning | │ /opt/venv/lib/python3.11/site-packages/onnxrunt │ immich_machine_learning | │ ime/capi/onnxruntime_inference_collection.py:22 │ immich_machine_learning | │ 0 in run │ immich_machine_learning | │ │ immich_machine_learning | │ 217 │ │ if not output_names: │ immich_machine_learning | │ 218 │ │ │ output_names = [output.name │ immich_machine_learning | │ 219 │ │ try: │ immich_machine_learning | │ ❱ 220 │ │ │ return self._sess.run(output │ immich_machine_learning | │ 221 │ │ except C.EPFail as err: │ immich_machine_learning | │ 222 │ │ │ if self._enable_fallback: │ immich_machine_learning | │ 223 │ │ │ │ print(f"EP Error: {err!s │ immich_machine_learning | ╰─────────────────────────────────────────────────╯ immich_machine_learning | Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero immich_machine_learning | status code returned while running FusedConv node. immich_machine_learning | Name:'Conv.2' Status Message: CUDNN failure 2000: immich_machine_learning | CUDNN_STATUS_BAD_PARAM ; GPU=0 ; immich_machine_learning | hostname=44be8fb8dcfd ; immich_machine_learning | file=/onnxruntime_src/onnxruntime/core/providers/cu immich_machine_learning | da/nn/conv.cc ; line=382 ; immich_machine_learning | expr=cudnnFindConvolutionForwardAlgorithmEx( immich_machine_learning | GetCudnnHandle(context), s_.x_tensor, s_.x_data, immich_machine_learning | s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, immich_machine_learning | s_.y_data, 1, &algo_count, &perf, immich_machine_learning | algo_search_workspace.get(), max_ws_size); immich_server | [Nest] 7 - 11/18/2025, 11:15:08 PM WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed with status 500: Internal Server Error immich_server | [Nest] 7 - 11/18/2025, 11:15:08 PM ERROR [Microservices:{"id":"c0f25065-03a9-4370-9097-ca6beb6e5b7a"}] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr":{"detection":{"modelName":"PP-OCRv5_server","options":{"minScore":0.5,"maxResolution":1280}},"recognition":{"modelName":"PP-OCRv5_server","options":{"minScore":0.8}}}}' failed for all URLs immich_server | Error: Machine learning request '{"ocr":{"detection":{"modelName":"PP-OCRv5_server","options":{"minScore":0.5,"maxResolution":1280}},"recognition":{"modelName":"PP-OCRv5_server","options":{"minScore":0.8}}}}' failed for all URLs immich_server | at MachineLearningRepository.predict (/usr/src/app/server/dist/repositories/machine-learning.repository.js:117:15) immich_server | at process.processTicksAndRejections (node:internal/process/task_queues:105:5) immich_server | at async MachineLearningRepository.ocr (/usr/src/app/server/dist/repositories/machine-learning.repository.js:150:26) immich_server | at async OcrService.handleOcr (/usr/src/app/server/dist/services/ocr.service.js:52:28) immich_server | at async JobService.onJobRun (/usr/src/app/server/dist/services/job.service.js:199:30) immich_server | at async EventRepository.onEvent (/usr/src/app/server/dist/repositories/event.repository.js:91:13) immich_server | at async /usr/src/app/server/node_modules/.pnpm/bullmq@5.61.2/node_modules/bullmq/dist/cjs/classes/worker.js:528:32 ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 13:21:18 +03:00
Author
Owner
Copy Link

@bb159 commented on GitHub (Nov 18, 2025):

I found that the VRAM recycling mechanism might be faulty, as my VRAM usage is consistently abnormal.

@bb159 commented on GitHub (Nov 18, 2025): I found that the VRAM recycling mechanism might be faulty, as my VRAM usage is consistently abnormal.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#7858
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?