immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

CUDA Machine Learning - ERROR Worker was sent SIGSEGV! #4664

New Issue
Closed
opened 2026-02-05 10:46:21 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 10:46:21 +03:00
Owner
Copy Link

Originally created by @Electr0Fi on GitHub (Oct 30, 2024).

The bug

I am trying to use the CUDA ML image with Immich however I constantly get a SIGSEGV error within the docker.io/altran1502/immich-machine-learning:v1.119.0-cuda pod.

As such, none of the Face Detection or Facial Recognition jobs are being completed.

The OS that Immich Server is running on

TrueNAS Scale Dragonfish-24.04.2.3

Version of Immich Server

v1.119.0

Version of Immich Mobile App

v1.118.0 build.163

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

[Default docker-compose.yml file used by the TrueNAS App]

Your .env content

[Default .env file used by the TrueNAS App]

Reproduction steps

  1. Start TrueNAS App container
  2. Have Face Detection or Facial Recognition jobs in the queue

...

Relevant log output

### From docker.io/altran1502/immich-machine-learning:v1.119.0-cuda Pod Log ###

2024-10-30T18:15:38.373020093+01:00  [2;36m[10/30/24 17:15:38] [0m [2;36m  [0m [34mINFO     [0m Booting worker with pid:  [1;36m3062 [0m                      
2024-10-30T18:15:44.037339811+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Started server process  [1m[ [0m [1;36m3062 [0m [1m] [0m                      
2024-10-30T18:15:44.039173475+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Waiting for application startup.                   
2024-10-30T18:15:44.041015897+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Created in-memory cache with unloading after 300s  
2024-10-30T18:15:44.041060719+01:00  [2;36m                     [0m         of inactivity.                                     
2024-10-30T18:15:44.041811530+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Initialized request thread pool with  [1;36m4 [0m threads.    
2024-10-30T18:15:44.042708016+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Application startup complete.                      
2024-10-30T18:15:44.205091691+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Loading detection model  [32m'buffalo_l' [0m to memory      
2024-10-30T18:15:44.205146385+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Setting execution providers to                     
2024-10-30T18:15:44.205159061+01:00  [2;36m                     [0m          [1m[ [0m [32m'CUDAExecutionProvider' [0m,  [32m'CPUExecutionProvider' [0m [1m] [0m, 
2024-10-30T18:15:44.205166901+01:00  [2;36m                     [0m         in descending order of preference                  
2024-10-30T18:15:44.353961568+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [1;31mERROR    [0m Worker  [1m( [0mpi [1;92md:3062 [0m [1m) [0m was sent SIGSEGV!                
2024-10-30T18:15:44.361574078+01:00  [2;36m[10/30/24 17:15:44] [0m [2;36m  [0m [34mINFO     [0m Booting worker with pid:  [1;36m3082 [0m                      
2024-10-30T18:15:49.788180355+01:00  [2;36m[10/30/24 17:15:49] [0m [2;36m  [0m [34mINFO     [0m Started server process  [1m[ [0m [1;36m3082 [0m [1m] [0m                      
2024-10-30T18:15:49.788902237+01:00  [2;36m[10/30/24 17:15:49] [0m [2;36m  [0m [34mINFO     [0m Waiting for application startup.                   
2024-10-30T18:15:49.791517183+01:00  [2;36m[10/30/24 17:15:49] [0m [2;36m  [0m [34mINFO     [0m Created in-memory cache with unloading after 300s  
2024-10-30T18:15:49.791554282+01:00  [2;36m                     [0m         of inactivity.                                     
2024-10-30T18:15:49.792420213+01:00  [2;36m[10/30/24 17:15:49] [0m [2;36m  [0m [34mINFO     [0m Initialized request thread pool with  [1;36m4 [0m threads.    
2024-10-30T18:15:49.793382125+01:00  [2;36m[10/30/24 17:15:49] [0m [2;36m  [0m [34mINFO     [0m Application startup complete.                      
2024-10-30T18:15:50.011234962+01:00  [2;36m[10/30/24 17:15:50] [0m [2;36m  [0m [34mINFO     [0m Loading detection model  [32m'buffalo_l' [0m to memory      
2024-10-30T18:15:50.014607680+01:00  [2;36m[10/30/24 17:15:50] [0m [2;36m  [0m [34mINFO     [0m Setting execution providers to                     
2024-10-30T18:15:50.015807518+01:00  [2;36m                     [0m          [1m[ [0m [32m'CUDAExecutionProvider' [0m,  [32m'CPUExecutionProvider' [0m [1m] [0m, 
2024-10-30T18:15:50.015836072+01:00  [2;36m                     [0m         in descending order of preference                  
2024-10-30T18:15:50.164811791+01:00  [2;36m[10/30/24 17:15:50] [0m [2;36m  [0m [1;31mERROR    [0m Worker  [1m( [0mpi [1;92md:3082 [0m [1m) [0m was sent SIGSEGV!



### From docker.io/altran1502/immich-server:v1.119.0 Pod Log ###

2024-10-30T18:24:00.183428146+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mUnable to run job handler (smartSearch/smart-search): Error: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed [39m
2024-10-30T18:24:00.199604632+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mError: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed
2024-10-30T18:24:00.199658711+01:00     at /usr/src/app/dist/repositories/machine-learning.repository.js:18:19
2024-10-30T18:24:00.199672104+01:00     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:17:21)
2024-10-30T18:24:00.199680307+01:00     at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:41:26)
2024-10-30T18:24:00.199688362+01:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:108:27)
2024-10-30T18:24:00.199695341+01:00     at async /usr/src/app/dist/services/job.service.js:163:36
2024-10-30T18:24:00.199702187+01:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024-10-30T18:24:00.199709240+01:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [39m
2024-10-30T18:24:00.203588748+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mObject: [39m
2024-10-30T18:24:00.203670111+01:00 {
2024-10-30T18:24:00.203684178+01:00   "id": "188c37ba-ed94-47ff-b8d7-78e4a541fbb2"
2024-10-30T18:24:00.203690890+01:00 }
2024-10-30T18:24:00.203714276+01:00 
2024-10-30T18:24:00.203724179+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mUnable to run job handler (smartSearch/smart-search): Error: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed [39m
2024-10-30T18:24:00.203730908+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mError: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed
2024-10-30T18:24:00.203737396+01:00     at /usr/src/app/dist/repositories/machine-learning.repository.js:18:19
2024-10-30T18:24:00.203744122+01:00     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:17:21)
2024-10-30T18:24:00.203750239+01:00     at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:41:26)
2024-10-30T18:24:00.203757069+01:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:108:27)
2024-10-30T18:24:00.203763343+01:00     at async /usr/src/app/dist/services/job.service.js:163:36
2024-10-30T18:24:00.203769483+01:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024-10-30T18:24:00.204213844+01:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [39m
2024-10-30T18:24:00.204235423+01:00  [31m[Nest] 7  -  [39m10/30/2024, 6:24:00 PM  [31m  ERROR [39m  [33m[Microservices:JobService] [39m  [31mObject: [39m
2024-10-30T18:24:00.204243665+01:00 {
2024-10-30T18:24:00.204250693+01:00   "id": "1b04b0bc-2ff6-4b20-b2e8-b35c9cc3a6cf"
2024-10-30T18:24:00.204257618+01:00 }

Additional information

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1660 ...    Off | 00000000:01:00.0 Off |                  N/A |
| 39%   47C    P0              36W / 125W |      0MiB /  6144MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
Originally created by @Electr0Fi on GitHub (Oct 30, 2024). ### The bug I am trying to use the CUDA ML image with Immich however I constantly get a SIGSEGV error within the `docker.io/altran1502/immich-machine-learning:v1.119.0-cuda` pod. As such, none of the Face Detection or Facial Recognition jobs are being completed. ### The OS that Immich Server is running on TrueNAS Scale Dragonfish-24.04.2.3 ### Version of Immich Server v1.119.0 ### Version of Immich Mobile App v1.118.0 build.163 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML [Default docker-compose.yml file used by the TrueNAS App] ``` ### Your .env content ```Shell [Default .env file used by the TrueNAS App] ``` ### Reproduction steps 1. Start TrueNAS App container 2. Have Face Detection or Facial Recognition jobs in the queue 3. ... ### Relevant log output ```shell ### From docker.io/altran1502/immich-machine-learning:v1.119.0-cuda Pod Log ### 2024-10-30T18:15:38.373020093+01:00 [2;36m[10/30/24 17:15:38] [0m [2;36m [0m [34mINFO [0m Booting worker with pid: [1;36m3062 [0m 2024-10-30T18:15:44.037339811+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Started server process [1m[ [0m [1;36m3062 [0m [1m] [0m 2024-10-30T18:15:44.039173475+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Waiting for application startup. 2024-10-30T18:15:44.041015897+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Created in-memory cache with unloading after 300s 2024-10-30T18:15:44.041060719+01:00 [2;36m [0m of inactivity. 2024-10-30T18:15:44.041811530+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Initialized request thread pool with [1;36m4 [0m threads. 2024-10-30T18:15:44.042708016+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Application startup complete. 2024-10-30T18:15:44.205091691+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Loading detection model [32m'buffalo_l' [0m to memory 2024-10-30T18:15:44.205146385+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Setting execution providers to 2024-10-30T18:15:44.205159061+01:00 [2;36m [0m [1m[ [0m [32m'CUDAExecutionProvider' [0m, [32m'CPUExecutionProvider' [0m [1m] [0m, 2024-10-30T18:15:44.205166901+01:00 [2;36m [0m in descending order of preference 2024-10-30T18:15:44.353961568+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [1;31mERROR [0m Worker [1m( [0mpi [1;92md:3062 [0m [1m) [0m was sent SIGSEGV! 2024-10-30T18:15:44.361574078+01:00 [2;36m[10/30/24 17:15:44] [0m [2;36m [0m [34mINFO [0m Booting worker with pid: [1;36m3082 [0m 2024-10-30T18:15:49.788180355+01:00 [2;36m[10/30/24 17:15:49] [0m [2;36m [0m [34mINFO [0m Started server process [1m[ [0m [1;36m3082 [0m [1m] [0m 2024-10-30T18:15:49.788902237+01:00 [2;36m[10/30/24 17:15:49] [0m [2;36m [0m [34mINFO [0m Waiting for application startup. 2024-10-30T18:15:49.791517183+01:00 [2;36m[10/30/24 17:15:49] [0m [2;36m [0m [34mINFO [0m Created in-memory cache with unloading after 300s 2024-10-30T18:15:49.791554282+01:00 [2;36m [0m of inactivity. 2024-10-30T18:15:49.792420213+01:00 [2;36m[10/30/24 17:15:49] [0m [2;36m [0m [34mINFO [0m Initialized request thread pool with [1;36m4 [0m threads. 2024-10-30T18:15:49.793382125+01:00 [2;36m[10/30/24 17:15:49] [0m [2;36m [0m [34mINFO [0m Application startup complete. 2024-10-30T18:15:50.011234962+01:00 [2;36m[10/30/24 17:15:50] [0m [2;36m [0m [34mINFO [0m Loading detection model [32m'buffalo_l' [0m to memory 2024-10-30T18:15:50.014607680+01:00 [2;36m[10/30/24 17:15:50] [0m [2;36m [0m [34mINFO [0m Setting execution providers to 2024-10-30T18:15:50.015807518+01:00 [2;36m [0m [1m[ [0m [32m'CUDAExecutionProvider' [0m, [32m'CPUExecutionProvider' [0m [1m] [0m, 2024-10-30T18:15:50.015836072+01:00 [2;36m [0m in descending order of preference 2024-10-30T18:15:50.164811791+01:00 [2;36m[10/30/24 17:15:50] [0m [2;36m [0m [1;31mERROR [0m Worker [1m( [0mpi [1;92md:3082 [0m [1m) [0m was sent SIGSEGV! ### From docker.io/altran1502/immich-server:v1.119.0 Pod Log ### 2024-10-30T18:24:00.183428146+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mUnable to run job handler (smartSearch/smart-search): Error: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed [39m 2024-10-30T18:24:00.199604632+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mError: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed 2024-10-30T18:24:00.199658711+01:00 at /usr/src/app/dist/repositories/machine-learning.repository.js:18:19 2024-10-30T18:24:00.199672104+01:00 at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:17:21) 2024-10-30T18:24:00.199680307+01:00 at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:41:26) 2024-10-30T18:24:00.199688362+01:00 at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:108:27) 2024-10-30T18:24:00.199695341+01:00 at async /usr/src/app/dist/services/job.service.js:163:36 2024-10-30T18:24:00.199702187+01:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 2024-10-30T18:24:00.199709240+01:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [39m 2024-10-30T18:24:00.203588748+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mObject: [39m 2024-10-30T18:24:00.203670111+01:00 { 2024-10-30T18:24:00.203684178+01:00 "id": "188c37ba-ed94-47ff-b8d7-78e4a541fbb2" 2024-10-30T18:24:00.203690890+01:00 } 2024-10-30T18:24:00.203714276+01:00 2024-10-30T18:24:00.203724179+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mUnable to run job handler (smartSearch/smart-search): Error: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed [39m 2024-10-30T18:24:00.203730908+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mError: Machine learning request to "http://immich-machinelearning:32002" failed with SocketError: other side closed 2024-10-30T18:24:00.203737396+01:00 at /usr/src/app/dist/repositories/machine-learning.repository.js:18:19 2024-10-30T18:24:00.203744122+01:00 at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:17:21) 2024-10-30T18:24:00.203750239+01:00 at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:41:26) 2024-10-30T18:24:00.203757069+01:00 at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:108:27) 2024-10-30T18:24:00.203763343+01:00 at async /usr/src/app/dist/services/job.service.js:163:36 2024-10-30T18:24:00.203769483+01:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 2024-10-30T18:24:00.204213844+01:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [39m 2024-10-30T18:24:00.204235423+01:00 [31m[Nest] 7 - [39m10/30/2024, 6:24:00 PM [31m ERROR [39m [33m[Microservices:JobService] [39m [31mObject: [39m 2024-10-30T18:24:00.204243665+01:00 { 2024-10-30T18:24:00.204250693+01:00 "id": "1b04b0bc-2ff6-4b20-b2e8-b35c9cc3a6cf" 2024-10-30T18:24:00.204257618+01:00 } ``` ### Additional information ``` +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1660 ... Off | 00000000:01:00.0 Off | N/A | | 39% 47C P0 36W / 125W | 0MiB / 6144MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#4664
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?