immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

"Failed to smart search" - Internal Server Error since latest version update (upgraded to 1.135.3 from 1.134.0) #6482

New Issue
Closed
opened 2026-02-05 12:19:03 +03:00 by OVERLORD · 3 comments
OVERLORD commented 2026-02-05 12:19:03 +03:00
Owner
Copy Link

Originally created by @tomzorz on GitHub (Jul 22, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

When I try to do a context search, I get an error back since updating the latest version - see the docker log below.

I had the machine learning features working before, but now it's broken as upgraded to 1.135.3 from 1.134.0. I wonder if there might've been something else as I haven't immediately tried a context search before updating. I haven't upgraded my unraid host or my nvidia drivers, so I don't think the issue is coming from there.

The OS that Immich Server is running on

UnRaid 6.12.8

Version of Immich Server

v1.135.3

Version of Immich Mobile App

latest (but probably not relevant)

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

n/a as I am on unraid. 

Running latest `ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0` and `redis` alongside.

Your .env content

Pasting the relevant things from my unraid setup:

Machine learning host: 0.0.0.0
... Port: 3003
... Worker: 1
... Timeout: 120
NVIDIA_DRIVER_CAPABILITIES: all
device: my device ID as reported by unraid

Reproduction steps

  1. Try any context search, e.g. "hedgehog"
  2. See immediate error, both in logs and on the UI

Relevant log output

[Nest] 250  - 07/21/2025, 6:13:16 PM     LOG [Api:EventRepository] Websocket Connect:    QWOk2VRKqatY4yFOAAAB
[07/21/25 18:13:21] INFO     Loading textual model 'ViT-B-32__openai' to memory 
[07/21/25 18:13:21] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-07-21 18:13:22.245116533 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2025-07-21 18:13:22.465823438 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=62beda82f7d5 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc ; line=571 ; expr=cudnnReduceTensor( CudaKernel::GetCudnnHandle(cuda_stream), reduce_desc, indices_cuda.get(), indices_bytes, workspace_cuda.get(), workspace_bytes, &one, input_tensor, reinterpret_cast<const CudaT*>(input.Data<T>()), &zero, output_tensor, p_output); 
2025-07-21 18:13:22.465857568 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running ReduceL2 node. Name:'ReduceL2_1604' Status Message: CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=62beda82f7d5 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc ; line=571 ; expr=cudnnReduceTensor( CudaKernel::GetCudnnHandle(cuda_stream), reduce_desc, indices_cuda.get(), indices_bytes, workspace_cuda.get(), workspace_bytes, &one, input_tensor, reinterpret_cast<const CudaT*>(input.Data<T>()), &zero, output_tensor, p_output); 
[07/21/25 18:13:22] ERROR    Exception in ASGI application                      
                                                                                
                             ╭─────── Traceback (most recent call last) ──���────╮
                             │ /app/immich/machine-learning/immich_ml/main.py: │
                             │ 177 in predict                                  │
                             │                                                 │
                             │   174 │   │   inputs = text                     │
                             │   175 │   else:                                 │
                             │   176 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 177 │   response = await run_inference(inputs │
                             │   178 │   return ORJSONResponse(response)       │
                             │   179                                           │
                             │   180                                           │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/main.py: │
                             │ 200 in run_inference                            │
                             │                                                 │
                             │   197 │   │   response[entry["task"]] = output  │
                             │   198 │                                         │
                             │   199 │   without_deps, with_deps = entries     │
                             │ ❱ 200 │   await asyncio.gather(*[_run_inference │
                             │   201 │   if with_deps:                         │
                             │   202 │   │   await asyncio.gather(*[_run_infer │
                             │   203 │   if isinstance(payload, Image):        │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/main.py: │
                             │ 195 in _run_inference                           │
                             │                                                 │
                             │   192 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   193 │   │   │   │   raise HTTPException(400,  │
                             │   194 │   │   model = await load(model)         │
                             │ ❱ 195 │   │   output = await run(model.predict, │
                             │   196 │   │   outputs[model.identity] = output  │
                             │   197 │   │   response[entry["task"]] = output  │
                             │   198                                           │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/main.py: │
                             │ 213 in run                                      │
                             │                                                 │
                             │   210 │   if thread_pool is None:               │
                             │   211 │   │   return func(*args, **kwargs)      │
                             │   212 │   partial_func = partial(func, *args, * │
                             │ ❱ 213 │   return await asyncio.get_running_loop │
                             │   214                                           │
                             │   215                                           │
                             │   216 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/lib/python3.11/concurrent/futures/thread.p │
                             │ y:58 in run                                     │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/models/b │
                             │ ase.py:61 in predict                            │
                             │                                                 │
                             │    58 │   │   self.load()                       │
                             │    59 │   │   if model_kwargs:                  │
                             │    60 │   │   │   self.configure(**model_kwargs │
                             │ ❱  61 │   │   return self._predict(*inputs, **m │
                             │    62 │                                         │
                             │    63 │   @abstractmethod                       │
                             │    64 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/models/c │
                             │ lip/textual.py:24 in _predict                   │
                             │                                                 │
                             │    21 │                                         │
                             │    22 │   def _predict(self, inputs: str, langu │
                             │    23 │   │   tokens = self.tokenize(inputs, la │
                             │ ❱  24 │   │   res: NDArray[np.float32] = self.s │
                             │    25 │   │   return serialize_np_array(res)    │
                             │    26 │                                         │
                             │    27 │   def _load(self) -> ModelSession:      │
                             │                                                 │
                             │ /app/immich/machine-learning/immich_ml/sessions │
                             │ /ort.py:49 in run                               │
                             │                                                 │
                             │    46 │   │   input_feed: dict[str, NDArray[np. │
                             │    47 │   │   run_options: Any = None,          │
                             │    48 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  49 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    50 │   │   return outputs                    │
                             │    51 │                                         │
                             │    52 │   @property                             │
                             │                                                 │
                             │ /lsiopy/lib/python3.11/site-packages/onnxruntim │
                             │ e/capi/onnxruntime_inference_collection.py:220  │
                             │ in run                                          │
                             │                                                 │
                             │    217 │   │   if not output_names:             │
                             │    218 │   │   │   output_names = [output.name  │
                             │    219 │   │   try:                             │
                             │ ❱  220 │   │   │   return self._sess.run(output │
                             │    221 │   │   except C.EPFail as err:          │
                             │    222 │   │   │   if self._enable_fallback:    │
                             │    223 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero     
                             status code returned while running ReduceL2 node.  
                             Name:'ReduceL2_1604' Status Message: CUDNN failure 
                             5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ;      
                             hostname=62beda82f7d5 ;                            
                             file=/onnxruntime_src/onnxruntime/core/providers/cu
                             da/reduction/reduction_ops.cc ; line=571 ;         
                             expr=cudnnReduceTensor(                            
                             CudaKernel::GetCudnnHandle(cuda_stream),           
                             reduce_desc, indices_cuda.get(), indices_bytes,    
                             workspace_cuda.get(), workspace_bytes, &one,       
                             input_tensor, reinterpret_cast<const               
                             CudaT*>(input.Data<T>()), &zero, output_tensor,    
                             p_output);                                         
[Nest] 250  - 07/21/2025, 6:13:22 PM    WARN [Api:MachineLearningRepository~u9hsyes5] Machine learning request to "http://127.0.0.1:3003" failed with status 500: Internal Server Error
[Nest] 250  - 07/21/2025, 6:13:22 PM   ERROR [Api:ErrorInterceptor~u9hsyes5] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai","options":{"language":"en-US"}}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai","options":{"language":"en-US"}}}}' failed for all URLs
    at MachineLearningRepository.predict (/app/immich/server/dist/repositories/machine-learning.repository.js:98:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async MachineLearningRepository.encodeText (/app/immich/server/dist/repositories/machine-learning.repository.js:121:26)
    at async SearchService.searchSmart (/app/immich/server/dist/services/search.service.js:84:25)

Additional information

No response

Originally created by @tomzorz on GitHub (Jul 22, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug When I try to do a context search, I get an error back since updating the latest version - see the docker log below. I had the machine learning features working before, but now it's broken as upgraded to 1.135.3 from 1.134.0. I wonder if there might've been something else as I haven't immediately tried a context search before updating. I haven't upgraded my unraid host or my nvidia drivers, so I don't think the issue is coming from there. ### The OS that Immich Server is running on UnRaid 6.12.8 ### Version of Immich Server v1.135.3 ### Version of Immich Mobile App latest (but probably not relevant) ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML n/a as I am on unraid. Running latest `ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0` and `redis` alongside. ``` ### Your .env content ```Shell Pasting the relevant things from my unraid setup: Machine learning host: 0.0.0.0 ... Port: 3003 ... Worker: 1 ... Timeout: 120 NVIDIA_DRIVER_CAPABILITIES: all device: my device ID as reported by unraid ``` ### Reproduction steps 1. Try any context search, e.g. "hedgehog" 2. See immediate error, both in logs and on the UI ### Relevant log output ```shell [Nest] 250 - 07/21/2025, 6:13:16 PM LOG [Api:EventRepository] Websocket Connect: QWOk2VRKqatY4yFOAAAB [07/21/25 18:13:21] INFO Loading textual model 'ViT-B-32__openai' to memory [07/21/25 18:13:21] INFO Setting execution providers to ['CUDAExecutionProvider', 'CPUExecutionProvider'], in descending order of preference 2025-07-21 18:13:22.245116533 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message. 2025-07-21 18:13:22.465823438 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=62beda82f7d5 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc ; line=571 ; expr=cudnnReduceTensor( CudaKernel::GetCudnnHandle(cuda_stream), reduce_desc, indices_cuda.get(), indices_bytes, workspace_cuda.get(), workspace_bytes, &one, input_tensor, reinterpret_cast<const CudaT*>(input.Data<T>()), &zero, output_tensor, p_output); 2025-07-21 18:13:22.465857568 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running ReduceL2 node. Name:'ReduceL2_1604' Status Message: CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=62beda82f7d5 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc ; line=571 ; expr=cudnnReduceTensor( CudaKernel::GetCudnnHandle(cuda_stream), reduce_desc, indices_cuda.get(), indices_bytes, workspace_cuda.get(), workspace_bytes, &one, input_tensor, reinterpret_cast<const CudaT*>(input.Data<T>()), &zero, output_tensor, p_output); [07/21/25 18:13:22] ERROR Exception in ASGI application ╭─────── Traceback (most recent call last) ──���────╮ │ /app/immich/machine-learning/immich_ml/main.py: │ │ 177 in predict │ │ │ │ 174 │ │ inputs = text │ │ 175 │ else: │ │ 176 │ │ raise HTTPException(400, "Either │ │ ❱ 177 │ response = await run_inference(inputs │ │ 178 │ return ORJSONResponse(response) │ │ 179 │ │ 180 │ │ │ │ /app/immich/machine-learning/immich_ml/main.py: │ │ 200 in run_inference │ │ │ │ 197 │ │ response[entry["task"]] = output │ │ 198 │ │ │ 199 │ without_deps, with_deps = entries │ │ ❱ 200 │ await asyncio.gather(*[_run_inference │ │ 201 │ if with_deps: │ │ 202 │ │ await asyncio.gather(*[_run_infer │ │ 203 │ if isinstance(payload, Image): │ │ │ │ /app/immich/machine-learning/immich_ml/main.py: │ │ 195 in _run_inference │ │ │ │ 192 │ │ │ │ message = f"Task {entry[' │ │ output of {dep}" │ │ 193 │ │ │ │ raise HTTPException(400, │ │ 194 │ │ model = await load(model) │ │ ❱ 195 │ │ output = await run(model.predict, │ │ 196 │ │ outputs[model.identity] = output │ │ 197 │ │ response[entry["task"]] = output │ │ 198 │ │ │ │ /app/immich/machine-learning/immich_ml/main.py: │ │ 213 in run │ │ │ │ 210 │ if thread_pool is None: │ │ 211 │ │ return func(*args, **kwargs) │ │ 212 │ partial_func = partial(func, *args, * │ │ ❱ 213 │ return await asyncio.get_running_loop │ │ 214 │ │ 215 │ │ 216 async def load(model: InferenceModel) -> │ │ │ │ /usr/lib/python3.11/concurrent/futures/thread.p │ │ y:58 in run │ │ │ │ /app/immich/machine-learning/immich_ml/models/b │ │ ase.py:61 in predict │ │ │ │ 58 │ │ self.load() │ │ 59 │ │ if model_kwargs: │ │ 60 │ │ │ self.configure(**model_kwargs │ │ ❱ 61 │ │ return self._predict(*inputs, **m │ │ 62 │ │ │ 63 │ @abstractmethod │ │ 64 │ def _predict(self, *inputs: Any, **mo │ │ │ │ /app/immich/machine-learning/immich_ml/models/c │ │ lip/textual.py:24 in _predict │ │ │ │ 21 │ │ │ 22 │ def _predict(self, inputs: str, langu │ │ 23 │ │ tokens = self.tokenize(inputs, la │ │ ❱ 24 │ │ res: NDArray[np.float32] = self.s │ │ 25 │ │ return serialize_np_array(res) │ │ 26 │ │ │ 27 │ def _load(self) -> ModelSession: │ │ │ │ /app/immich/machine-learning/immich_ml/sessions │ │ /ort.py:49 in run │ │ │ │ 46 │ │ input_feed: dict[str, NDArray[np. │ │ 47 │ │ run_options: Any = None, │ │ 48 │ ) -> list[NDArray[np.float32]]: │ │ ❱ 49 │ │ outputs: list[NDArray[np.float32] │ │ run_options) │ │ 50 │ │ return outputs │ │ 51 │ │ │ 52 │ @property │ │ │ │ /lsiopy/lib/python3.11/site-packages/onnxruntim │ │ e/capi/onnxruntime_inference_collection.py:220 │ │ in run │ │ │ │ 217 │ │ if not output_names: │ │ 218 │ │ │ output_names = [output.name │ │ 219 │ │ try: │ │ ❱ 220 │ │ │ return self._sess.run(output │ │ 221 │ │ except C.EPFail as err: │ │ 222 │ │ │ if self._enable_fallback: │ │ 223 │ │ │ │ print(f"EP Error: {err!s │ ╰─────────────────────────────────────────────────╯ Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running ReduceL2 node. Name:'ReduceL2_1604' Status Message: CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=62beda82f7d5 ; file=/onnxruntime_src/onnxruntime/core/providers/cu da/reduction/reduction_ops.cc ; line=571 ; expr=cudnnReduceTensor( CudaKernel::GetCudnnHandle(cuda_stream), reduce_desc, indices_cuda.get(), indices_bytes, workspace_cuda.get(), workspace_bytes, &one, input_tensor, reinterpret_cast<const CudaT*>(input.Data<T>()), &zero, output_tensor, p_output); [Nest] 250 - 07/21/2025, 6:13:22 PM WARN [Api:MachineLearningRepository~u9hsyes5] Machine learning request to "http://127.0.0.1:3003" failed with status 500: Internal Server Error [Nest] 250 - 07/21/2025, 6:13:22 PM ERROR [Api:ErrorInterceptor~u9hsyes5] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai","options":{"language":"en-US"}}}}' failed for all URLs Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai","options":{"language":"en-US"}}}}' failed for all URLs at MachineLearningRepository.predict (/app/immich/server/dist/repositories/machine-learning.repository.js:98:15) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async MachineLearningRepository.encodeText (/app/immich/server/dist/repositories/machine-learning.repository.js:121:26) at async SearchService.searchSmart (/app/immich/server/dist/services/search.service.js:84:25) ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 12:19:04 +03:00
Author
Owner
Copy Link

@tomzorz commented on GitHub (Jul 22, 2025):

I'm in the immich discord under tomzorz if someone wants to poke at me directly, happy to provide whatever assistance to debug. (Linked the issue in the unraid channel as well.)

@tomzorz commented on GitHub (Jul 22, 2025): I'm in the immich discord under `tomzorz` if someone wants to poke at me directly, happy to provide whatever assistance to debug. (Linked the issue in the unraid channel as well.)
OVERLORD commented 2026-02-05 12:19:05 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Jul 22, 2025):

@mertalev thoughts?

@bo0tzz commented on GitHub (Jul 22, 2025): @mertalev thoughts?
OVERLORD commented 2026-02-05 12:19:05 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Jul 22, 2025):

It doesn't seem like an Immich issue, more likely in the driver or nvidia container toolkit. If you can't restart docker engine (not just the container), try restarting the server.

@mertalev commented on GitHub (Jul 22, 2025): It doesn't seem like an Immich issue, more likely in the driver or nvidia container toolkit. If you can't restart docker engine (not just the container), try restarting the server.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#6482
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?