immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 17 Packages Projects Releases 10 Wiki Activity

[BUG] immich-machine-learning taking up a huge amount of storage space #1769

New Issue
Closed
opened 2026-02-05 03:39:49 +03:00 by OVERLORD · 17 comments
OVERLORD commented 2026-02-05 03:39:49 +03:00
Owner
Copy Link

Originally created by @BlueWolfProduction on GitHub (Dec 11, 2023).

The bug

I have a fresh new setup (like a few hours ago), using the docker compose setup. All settings all default, except a change in the storage template. I have uploaded 626 photos, worth 3.9GB. I've let it run for a bit, all jobs are finished and all images are scanned. As far as I can tell it has done all the facial and object recognition it should've done because I can search for everything as expected.

According to 'docker system df -v', the immich-machine-learning image is currently sitting at 290GB in storage space, which is insane. And despite all jobs finished, it is still growing at a rapid pace.

This is actually my third attempt, as the previous attempts filled up my / entirely and killed my access to the device. I moved the default docker storage to a bigger external storage for the time being but it doesn't appear to stop growing.

Since I started writing ticket, the volume has grown by another 11GB, in the span of about 5 minutes.

The OS that Immich Server is running on

Ubuntu 22.04.3 LTS (On a Odroid N2+)

Version of Immich Server

v1.90.2

Version of Immich Mobile App

none

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:60e49e22fa5706cd8df7d5e0bc50ee9bab7c608039fa653c4d961014237cca46
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:6a0e35296341e676fe6bd8d236c72afffe2dfe3d7eb9c2405c0f3fc04500cd07
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/data/photos

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=[...]

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Install using the docker compose guide
2. Upload some images (I noticed it was happening after I had uploaded about 100 photos)
3. Watch free space get lower over the next hour

Additional information

No response

Originally created by @BlueWolfProduction on GitHub (Dec 11, 2023). ### The bug I have a fresh new setup (like a few hours ago), using the docker compose setup. All settings all default, except a change in the storage template. I have uploaded 626 photos, worth 3.9GB. I've let it run for a bit, all jobs are finished and all images are scanned. As far as I can tell it has done all the facial and object recognition it should've done because I can search for everything as expected. According to '`docker system df -v`', the immich-machine-learning image is currently sitting at 290GB in storage space, which is insane. And despite all jobs finished, it is still growing at a rapid pace. This is actually my third attempt, as the previous attempts filled up my / entirely and killed my access to the device. I moved the default docker storage to a bigger external storage for the time being but it doesn't appear to stop growing. Since I started writing ticket, the volume has grown by another 11GB, in the span of about 5 minutes. ### The OS that Immich Server is running on Ubuntu 22.04.3 LTS (On a Odroid N2+) ### Version of Immich Server v1.90.2 ### Version of Immich Mobile App none ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:60e49e22fa5706cd8df7d5e0bc50ee9bab7c608039fa653c4d961014237cca46 restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:6a0e35296341e676fe6bd8d236c72afffe2dfe3d7eb9c2405c0f3fc04500cd07 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache: tsdata: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/data/photos # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=some-random-text DB_PASSWORD=[...] # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. Install using the docker compose guide 2. Upload some images (I noticed it was happening after I had uploaded about 100 photos) 3. Watch free space get lower over the next hour ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 03:39:54 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Dec 11, 2023):

Have you changed anything relating to your docker's installation/configuration?

@alextran1502 commented on GitHub (Dec 11, 2023): Have you changed anything relating to your docker's installation/configuration?
OVERLORD commented 2026-02-05 03:40:04 +03:00
Author
Owner
Copy Link

@BlueWolfProduction commented on GitHub (Dec 11, 2023):

Have you changed anything relating to your docker's installation/configuration?

Not during my first two attempts. Everything was all default. But it kept filling up my / within a few hours and causing me to lose access to ssh along with a couple of system files that got corrupted along its way.

So for the 3rd attempt I have changed where docker is storing the images and volumes so I can use a much bigger external drive. These were the exact steps I took, which I don't believe should affect how Immich is run.

ps: I reported the OS wrong. I should've said it's running Ubuntu 22.04.3 LTS, which is running on an Odroid N2+

@BlueWolfProduction commented on GitHub (Dec 11, 2023): > Have you changed anything relating to your docker's installation/configuration? Not during my first two attempts. Everything was all default. But it kept filling up my / within a few hours and causing me to lose access to ssh along with a couple of system files that got corrupted along its way. So for the 3rd attempt I have changed where docker is storing the images and volumes so I can use a much bigger external drive. [These were the exact steps I took](https://old.reddit.com/r/docker/comments/11hr57y/change_dockers_default_volume_directory/jay1g0u/), which I don't believe should affect how Immich is run. ps: I reported the OS wrong. I should've said it's running Ubuntu 22.04.3 LTS, which is running on an Odroid N2+
OVERLORD commented 2026-02-05 03:40:11 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 11, 2023):

Is it the model-cache volume that's growing, or another part of the container?

@bo0tzz commented on GitHub (Dec 11, 2023): Is it the `model-cache` volume that's growing, or another part of the container?
OVERLORD commented 2026-02-05 03:40:18 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Dec 11, 2023):

I think something might have gone wrong with the reconfiguration that led to the issue.

@alextran1502 commented on GitHub (Dec 11, 2023): I think something might have gone wrong with the reconfiguration that led to the issue.
OVERLORD commented 2026-02-05 03:40:22 +03:00
Author
Owner
Copy Link

@BlueWolfProduction commented on GitHub (Dec 11, 2023):

So I turned it off for a moment (docker compose down) and switched it on (docker compose up -d) and it appears to have cleared whatever was taking up so much space as the container went back to being 24.4kB in size. It stayed that way and didn't grow. I have uploaded an additional 600 photos and now I can see it has started to grow again slowly.

Is it the model-cache volume that's growing, or another part of the container?

Docker is still kind of new to me, so I am learning as I go. If there is any way to get the exact information you need just let me know.

The full output of 'docker system df -v' is the following right now:

$ docker system df -v
Images space usage:

REPOSITORY                                   TAG       IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
ghcr.io/immich-app/immich-server             release   37c896e8ef80   3 days ago     832MB     97.16MB       734.5MB       2
ghcr.io/immich-app/immich-machine-learning   release   cab5b51c6b45   3 days ago     1.26GB    97.16MB       1.162GB       1
<none>                                       <none>    c1cafc225d92   11 days ago    241MB     7.66MB        233MB         1
<none>                                       <none>    bf5ccad43774   4 weeks ago    27.5MB    7.66MB        19.88MB       1
<none>                                       <none>    53d71694cf5a   8 months ago   935MB     0B            935.3MB       1

Containers space usage:

CONTAINER ID   IMAGE                                                COMMAND                  LOCAL VOLUMES   SIZE      CREATED          STATUS          NAMES
e425d4e3cf40   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   0               0B        35 minutes ago   Up 35 minutes   immich_server
c89ddfb9b45e   ghcr.io/immich-app/immich-server:release             "tini -- /bin/sh sta…"   0               0B        35 minutes ago   Up 35 minutes   immich_microservices
ea8627aaf36d   redis:6.2-alpine                                     "docker-entrypoint.s…"   1               0B        35 minutes ago   Up 35 minutes   immich_redis
e8a5eca266a4   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     1               5.86GB    35 minutes ago   Up 35 minutes   immich_machine_learning
a23b1ee7f85c   postgres:14-alpine                                   "docker-entrypoint.s…"   1               63B       35 minutes ago   Up 35 minutes   immich_postgres
b4d0ffa9b56e   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   1               0B        35 minutes ago   Up 35 minutes   immich_typesense

Local Volumes space usage:

VOLUME NAME                                                        LINKS     SIZE
immich_model-cache                                                 1         903MB
immich_tsdata                                                      1         18.85MB
9ef4d241fea17c2c37430d6b2f544806a498529e13f14b38987047a0b42d3447   0         477.3kB
c6e9e4cbd62509fee01e730f1f80dc6187331d551e02b0b92f321d8909bff034   1         923.6kB
immich_pgdata                                                      1         202.9MB

Build cache usage: 0B

CACHE ID   CACHE TYPE   SIZE      CREATED   LAST USED   USAGE     SHARED

Inside the container it appears these core.## files are slowly accommodating:

root@e8a5eca266a4:/usr/src/app# ls -alh
total 3.6G
drwxr-xr-x 1 root root 4.0K Dec 11 19:00 .
drwxr-xr-x 1 root root 4.0K Dec  1 18:45 ..
-rw-r--r-- 1 root root    0 Dec  8 14:23 __init__.py
-rw-r--r-- 1 root root 1.9K Dec  8 14:23 config.py
-rw-r--r-- 1 root root 1.6K Dec  8 14:23 conftest.py
-rw------- 1 root root 1.7G Dec 11 18:51 core.103
-rw------- 1 root root 2.0G Dec 11 18:54 core.127
-rw------- 1 root root 1.2G Dec 11 18:37 core.15
-rw------- 1 root root 347M Dec 11 18:56 core.160
-rw------- 1 root root 1.9G Dec 11 19:01 core.172
-rw------- 1 root root    0 Dec 11 18:39 core.40
-rw------- 1 root root    0 Dec 11 18:41 core.47
-rw------- 1 root root    0 Dec 11 18:43 core.67
-rw------- 1 root root 211M Dec 11 18:45 core.79
-rw------- 1 root root 246M Dec 11 18:48 core.91
-rw-r--r-- 1 root root  426 Dec  8 14:23 log_conf.json
-rw-r--r-- 1 root root 4.7K Dec  8 14:23 main.py
drwxr-xr-x 2 root root 4.0K Dec  8 14:23 models
-rw-r--r-- 1 root root  984 Dec  8 14:23 schemas.py
-rwxr-xr-x 1 root root  453 Dec  8 14:23 start.sh
-rw-r--r-- 1 root root  11K Dec  8 14:23 test_main.py

I think something might have gone wrong with the reconfiguration that led to the issue.

During what configuration exactly? It started going wrong with default docker settings, which forced me to relocate where docker stores its stuff. It didn't fix the main issue, but at least I can have it run for more than an hour before it crashes my system.

@BlueWolfProduction commented on GitHub (Dec 11, 2023): So I turned it off for a moment (docker compose down) and switched it on (docker compose up -d) and it appears to have cleared whatever was taking up so much space as the container went back to being 24.4kB in size. It stayed that way and didn't grow. I have uploaded an additional 600 photos and now I can see it has started to grow again slowly. > Is it the `model-cache` volume that's growing, or another part of the container? Docker is still kind of new to me, so I am learning as I go. If there is any way to get the exact information you need just let me know. The full output of 'docker system df -v' is the following right now: ``` $ docker system df -v Images space usage: REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS ghcr.io/immich-app/immich-server release 37c896e8ef80 3 days ago 832MB 97.16MB 734.5MB 2 ghcr.io/immich-app/immich-machine-learning release cab5b51c6b45 3 days ago 1.26GB 97.16MB 1.162GB 1 <none> <none> c1cafc225d92 11 days ago 241MB 7.66MB 233MB 1 <none> <none> bf5ccad43774 4 weeks ago 27.5MB 7.66MB 19.88MB 1 <none> <none> 53d71694cf5a 8 months ago 935MB 0B 935.3MB 1 Containers space usage: CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES e425d4e3cf40 ghcr.io/immich-app/immich-server:release "tini -- /bin/sh sta…" 0 0B 35 minutes ago Up 35 minutes immich_server c89ddfb9b45e ghcr.io/immich-app/immich-server:release "tini -- /bin/sh sta…" 0 0B 35 minutes ago Up 35 minutes immich_microservices ea8627aaf36d redis:6.2-alpine "docker-entrypoint.s…" 1 0B 35 minutes ago Up 35 minutes immich_redis e8a5eca266a4 ghcr.io/immich-app/immich-machine-learning:release "tini -- ./start.sh" 1 5.86GB 35 minutes ago Up 35 minutes immich_machine_learning a23b1ee7f85c postgres:14-alpine "docker-entrypoint.s…" 1 63B 35 minutes ago Up 35 minutes immich_postgres b4d0ffa9b56e typesense/typesense:0.24.1 "/opt/typesense-serv…" 1 0B 35 minutes ago Up 35 minutes immich_typesense Local Volumes space usage: VOLUME NAME LINKS SIZE immich_model-cache 1 903MB immich_tsdata 1 18.85MB 9ef4d241fea17c2c37430d6b2f544806a498529e13f14b38987047a0b42d3447 0 477.3kB c6e9e4cbd62509fee01e730f1f80dc6187331d551e02b0b92f321d8909bff034 1 923.6kB immich_pgdata 1 202.9MB Build cache usage: 0B CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED ``` Inside the container it appears these core.## files are slowly accommodating: ``` root@e8a5eca266a4:/usr/src/app# ls -alh total 3.6G drwxr-xr-x 1 root root 4.0K Dec 11 19:00 . drwxr-xr-x 1 root root 4.0K Dec 1 18:45 .. -rw-r--r-- 1 root root 0 Dec 8 14:23 __init__.py -rw-r--r-- 1 root root 1.9K Dec 8 14:23 config.py -rw-r--r-- 1 root root 1.6K Dec 8 14:23 conftest.py -rw------- 1 root root 1.7G Dec 11 18:51 core.103 -rw------- 1 root root 2.0G Dec 11 18:54 core.127 -rw------- 1 root root 1.2G Dec 11 18:37 core.15 -rw------- 1 root root 347M Dec 11 18:56 core.160 -rw------- 1 root root 1.9G Dec 11 19:01 core.172 -rw------- 1 root root 0 Dec 11 18:39 core.40 -rw------- 1 root root 0 Dec 11 18:41 core.47 -rw------- 1 root root 0 Dec 11 18:43 core.67 -rw------- 1 root root 211M Dec 11 18:45 core.79 -rw------- 1 root root 246M Dec 11 18:48 core.91 -rw-r--r-- 1 root root 426 Dec 8 14:23 log_conf.json -rw-r--r-- 1 root root 4.7K Dec 8 14:23 main.py drwxr-xr-x 2 root root 4.0K Dec 8 14:23 models -rw-r--r-- 1 root root 984 Dec 8 14:23 schemas.py -rwxr-xr-x 1 root root 453 Dec 8 14:23 start.sh -rw-r--r-- 1 root root 11K Dec 8 14:23 test_main.py ``` > I think something might have gone wrong with the reconfiguration that led to the issue. During what configuration exactly? It started going wrong with default docker settings, which forced me to relocate where docker stores its stuff. It didn't fix the main issue, but at least I can have it run for more than an hour before it crashes my system.
OVERLORD commented 2026-02-05 03:40:28 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 11, 2023):

Can you post the logs of the ML container as well?

@mertalev could it be that something is coredumping all over the place? If so we should probably disable that.

@bo0tzz commented on GitHub (Dec 11, 2023): Can you post the logs of the ML container as well? @mertalev could it be that something is coredumping all over the place? If so we should probably disable that.
OVERLORD commented 2026-02-05 03:40:31 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 11, 2023):

I think it might be that gunicorn is making core dumps after the ML service does its idle shutdown. It handles different signals differently, so I'll look into optimizing this.

@mertalev commented on GitHub (Dec 11, 2023): I think it might be that gunicorn is making core dumps after the ML service does its idle shutdown. It handles different signals differently, so I'll look into optimizing this.
OVERLORD commented 2026-02-05 03:40:33 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 11, 2023):

Weird that it's happening without any jobs running, though. It only does the shutdown after the service gets its first request after restarting.

@mertalev commented on GitHub (Dec 11, 2023): Weird that it's happening without any jobs running, though. It only does the shutdown after the service gets its first request after restarting.
OVERLORD commented 2026-02-05 03:40:35 +03:00
Author
Owner
Copy Link

@BlueWolfProduction commented on GitHub (Dec 11, 2023):

Can you post the logs of the ML container as well?

@mertalev could it be that something is coredumping all over the place? If so we should probably disable that.

Here is the current full output of 'docker logs immich_machine_learning'

[12/11/23 18:24:09] INFO     Starting gunicorn 21.2.0                                                                                                                                                                                       
[12/11/23 18:24:09] INFO     Listening at: http://0.0.0.0:3003 (9)                                                                                                                                                                          
[12/11/23 18:24:09] INFO     Using worker: uvicorn.workers.UvicornWorker                                                                                                                                                                    
[12/11/23 18:24:09] INFO     Booting worker with pid: 15                                                                                                                                                                                    
[12/11/23 18:24:35] INFO     Created in-memory cache with unloading after 300s                                                                                                                                                              
                             of inactivity.                                                                                                                                                                                                 
[12/11/23 18:24:35] INFO     Initialized request thread pool with 6 threads.                                                                                                                                                                
[12/11/23 18:36:34] INFO     Loading image classification model                                                                                                                                                                             
                             'microsoft/resnet-50'                                                                                                                                                                                          
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.                                                                 
[12/11/23 18:36:34] INFO     ONNX model not found in cache directory for                                                                                                                                                                    
                             'microsoft/resnet-50'.Exporting optimized model for                                                                                                                                                            
                             future use.                                                                                                                                                                                                    
Framework not specified. Using pt to export to ONNX.                                                                                                                                                                                        
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.                                                                 
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.                                                                                                                                                                                                                     
  warnings.warn(                                                                                                                                                                                                                            
Using the export variant default. Available variants are:                                                                                                                                                                                   
        - default: The default ONNX variant.                                                                                                                                                                                                
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.                                                                 
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 18:37:44] ERROR    Worker (pid:15) was sent code 132!                 
[12/11/23 18:37:46] INFO     Booting worker with pid: 40                         
[12/11/23 18:39:46] CRITICAL WORKER TIMEOUT (pid:40)                             
[12/11/23 18:39:47] ERROR    Worker (pid:40) was sent SIGKILL! Perhaps out of   
                             memory?                                             
[12/11/23 18:39:47] INFO     Booting worker with pid: 47                         
[12/11/23 18:41:48] CRITICAL WORKER TIMEOUT (pid:47)                             
[12/11/23 18:41:50] ERROR    Worker (pid:47) was sent SIGKILL! Perhaps out of   
                             memory?                                             
[12/11/23 18:41:51] INFO     Booting worker with pid: 67                         
[12/11/23 18:43:51] CRITICAL WORKER TIMEOUT (pid:67)                             
[12/11/23 18:43:55] ERROR    Worker (pid:67) was sent SIGKILL! Perhaps out of   
                             memory?                                             
[12/11/23 18:43:55] INFO     Booting worker with pid: 79                         
[12/11/23 18:45:56] CRITICAL WORKER TIMEOUT (pid:79)                             
[12/11/23 18:45:59] ERROR    Worker (pid:79) was sent SIGKILL! Perhaps out of   
                             memory?                                             
[12/11/23 18:45:59] INFO     Booting worker with pid: 91                         
[12/11/23 18:47:59] CRITICAL WORKER TIMEOUT (pid:91)                             
[12/11/23 18:48:02] ERROR    Worker (pid:91) was sent SIGKILL! Perhaps out of   
                             memory?                                             
[12/11/23 18:48:02] INFO     Booting worker with pid: 103                        
[12/11/23 18:49:43] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 18:49:43] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 18:49:44] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 18:50:04] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 18:50:04] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 18:51:02] ERROR    Worker (pid:103) was sent code 132!                
[12/11/23 18:51:04] INFO     Booting worker with pid: 127                        
[12/11/23 18:52:45] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 18:52:45] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 18:52:45] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 18:53:06] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 18:53:19] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 18:53:19] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 18:54:49] ERROR    Worker (pid:127) was sent code 132!                
[12/11/23 18:54:50] INFO     Booting worker with pid: 160                        
[12/11/23 18:56:50] CRITICAL WORKER TIMEOUT (pid:160)                            
[12/11/23 18:56:54] ERROR    Worker (pid:160) was sent SIGKILL! Perhaps out of  
                             memory?                                             
[12/11/23 18:56:54] INFO     Booting worker with pid: 172                        
[12/11/23 18:58:41] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 18:58:41] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 18:58:41] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 18:59:23] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 18:59:50] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 18:59:51] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:01:37] ERROR    Worker (pid:172) was sent code 132!                
[12/11/23 19:01:38] INFO     Booting worker with pid: 199                        
[12/11/23 19:02:20] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:02:20] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:02:20] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:02:20] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:02:57] ERROR    Worker (pid:199) was sent code 132!                
[12/11/23 19:02:57] INFO     Booting worker with pid: 222                        
[12/11/23 19:03:23] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:03:23] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:03:23] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:03:25] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:03:26] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:04:08] ERROR    Worker (pid:222) was sent code 132!                
[12/11/23 19:04:08] INFO     Booting worker with pid: 252                        
[12/11/23 19:04:45] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:04:45] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:04:45] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:04:45] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:05:17] ERROR    Worker (pid:252) was sent code 132!                
[12/11/23 19:05:17] INFO     Booting worker with pid: 275                        
[12/11/23 19:05:39] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:05:39] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:05:39] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:05:39] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:06:05] ERROR    Worker (pid:275) was sent code 132!                
[12/11/23 19:06:05] INFO     Booting worker with pid: 298                        
[12/11/23 19:06:34] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:06:34] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:06:34] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:06:36] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:06:36] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:07:18] ERROR    Worker (pid:298) was sent code 132!                
[12/11/23 19:07:18] INFO     Booting worker with pid: 328                        
[12/11/23 19:07:45] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:07:45] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:07:45] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:07:45] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:08:34] ERROR    Worker (pid:328) was sent code 132!                
[12/11/23 19:08:34] INFO     Booting worker with pid: 351                        
[12/11/23 19:08:58] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:08:58] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:08:58] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:09:00] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:09:06] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:09:06] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:10:42] ERROR    Worker (pid:351) was sent code 132!                
[12/11/23 19:10:42] INFO     Booting worker with pid: 382                        
[12/11/23 19:11:15] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:11:15] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:11:15] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:11:18] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:11:18] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:12:25] ERROR    Worker (pid:382) was sent code 132!                
[12/11/23 19:12:25] INFO     Booting worker with pid: 412                        
[12/11/23 19:12:56] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:12:56] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:12:56] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:12:57] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:13:02] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:13:03] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:14:57] CRITICAL WORKER TIMEOUT (pid:412)                            
[12/11/23 19:15:15] ERROR    Worker (pid:412) was sent SIGKILL! Perhaps out of  
                             memory?                                             
[12/11/23 19:15:16] INFO     Booting worker with pid: 443                        
[12/11/23 19:15:46] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:15:46] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:15:46] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:15:46] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:16:39] ERROR    Worker (pid:443) was sent code 132!                
[12/11/23 19:16:39] INFO     Booting worker with pid: 466                        
[12/11/23 19:17:02] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:17:02] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:17:02] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
[12/11/23 19:17:02] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
12/11/23 19:17:31] ERROR    Worker (pid:466) was sent code 132!                
[12/11/23 19:17:31] INFO     Booting worker with pid: 489                        
[12/11/23 19:18:00] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:18:00] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:18:00] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:18:04] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:18:04] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:18:44] ERROR    Worker (pid:489) was sent code 132!                
[12/11/23 19:18:44] INFO     Booting worker with pid: 520                        
[12/11/23 19:19:16] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:19:16] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:19:16] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:19:18] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:19:27] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:19:27] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:21:17] CRITICAL WORKER TIMEOUT (pid:520)                            
[12/11/23 19:21:29] ERROR    Worker (pid:520) was sent code 132!                
[12/11/23 19:21:30] INFO     Booting worker with pid: 551                        
[12/11/23 19:22:08] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:22:08] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:22:08] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:22:10] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:22:10] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:23:14] ERROR    Worker (pid:551) was sent code 132!                
[12/11/23 19:23:14] INFO     Booting worker with pid: 581                        
[12/11/23 19:23:55] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:23:55] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:23:55] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:23:56] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:24:03] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:24:04] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:25:56] CRITICAL WORKER TIMEOUT (pid:581)                            
[12/11/23 19:26:05] ERROR    Worker (pid:581) was sent code 132!                
[12/11/23 19:26:05] INFO     Booting worker with pid: 612                        
[12/11/23 19:26:45] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:26:45] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:26:45] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:26:47] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:26:49] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:27:57] ERROR    Worker (pid:612) was sent code 132!                
[12/11/23 19:27:57] INFO     Booting worker with pid: 642                        
[12/11/23 19:28:31] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:28:31] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:28:31] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:28:34] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:28:34] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:29:36] ERROR    Worker (pid:642) was sent code 132!                
[12/11/23 19:29:36] INFO     Booting worker with pid: 672                        
[12/11/23 19:30:09] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:30:09] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:30:09] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:30:15] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:30:15] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:31:46] ERROR    Worker (pid:672) was sent code 132!                
[12/11/23 19:31:46] INFO     Booting worker with pid: 696                        
[12/11/23 19:32:17] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:32:17] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:32:17] INFO     Loading facial recognition model 'buffalo_l'       
[12/11/23 19:32:18] INFO     Loading clip model 'ViT-B-32__openai'              
[12/11/23 19:32:26] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:32:26] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: d794b736-f75b-4dbb-a242-08103bb6712e)')' thrown while requesting HEAD https://huggingface.co/microsoft/resnet-5
0/resolve/main/config.json
[12/11/23 19:32:38] WARNING  '(ReadTimeoutError("HTTPSConnectionPool(host='huggi
                             ngface.co', port=443): Read timed out. (read       
                             timeout=10)"), '(Request ID:                        
                             d794b736-f75b-4dbb-a242-08103bb6712e)')' thrown    
                             while requesting HEAD                               
                             https://huggingface.co/microsoft/resnet-50/resolve/
                             main/config.json                                    
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:
[12/11/23 19:36:49] ERROR    Worker (pid:696) was sent code 132!                
[12/11/23 19:36:49] INFO     Booting worker with pid: 727                        
[12/11/23 19:37:29] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                      
[12/11/23 19:37:29] INFO     Initialized request thread pool with 6 threads.    
[12/11/23 19:37:29] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                               
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
[12/11/23 19:37:29] INFO     ONNX model not found in cache directory for        
                             'microsoft/resnet-50'.Exporting optimized model for
                             future use.                                         
Framework not specified. Using pt to export to ONNX.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext
ImageProcessor instead.
  warnings.warn(
Using the export variant default. Available variants are:
        - default: The default ONNX variant.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Using framework PyTorch: 2.1.0
/opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th
is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:

Lots of error's with code 132 here, and even a critical error at the beginning, with the suggestion it may have run out of memory? This device has 3.63GB of ram and 2GB of swap. I haven't seen any signs of low memory issues when any other programs that are running, and right now it is quite happily sitting at 50% memory usage.

@BlueWolfProduction commented on GitHub (Dec 11, 2023): > Can you post the logs of the ML container as well? > > @mertalev could it be that something is coredumping all over the place? If so we should probably disable that. Here is the current full output of 'docker logs immich_machine_learning' ``` [12/11/23 18:24:09] INFO Starting gunicorn 21.2.0 [12/11/23 18:24:09] INFO Listening at: http://0.0.0.0:3003 (9) [12/11/23 18:24:09] INFO Using worker: uvicorn.workers.UvicornWorker [12/11/23 18:24:09] INFO Booting worker with pid: 15 [12/11/23 18:24:35] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 18:24:35] INFO Initialized request thread pool with 6 threads. [12/11/23 18:36:34] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 18:36:34] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 18:37:44] ERROR Worker (pid:15) was sent code 132! [12/11/23 18:37:46] INFO Booting worker with pid: 40 [12/11/23 18:39:46] CRITICAL WORKER TIMEOUT (pid:40) [12/11/23 18:39:47] ERROR Worker (pid:40) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:39:47] INFO Booting worker with pid: 47 [12/11/23 18:41:48] CRITICAL WORKER TIMEOUT (pid:47) [12/11/23 18:41:50] ERROR Worker (pid:47) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:41:51] INFO Booting worker with pid: 67 [12/11/23 18:43:51] CRITICAL WORKER TIMEOUT (pid:67) [12/11/23 18:43:55] ERROR Worker (pid:67) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:43:55] INFO Booting worker with pid: 79 [12/11/23 18:45:56] CRITICAL WORKER TIMEOUT (pid:79) [12/11/23 18:45:59] ERROR Worker (pid:79) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:45:59] INFO Booting worker with pid: 91 [12/11/23 18:47:59] CRITICAL WORKER TIMEOUT (pid:91) [12/11/23 18:48:02] ERROR Worker (pid:91) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:48:02] INFO Booting worker with pid: 103 [12/11/23 18:49:43] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 18:49:43] INFO Initialized request thread pool with 6 threads. [12/11/23 18:49:44] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 18:50:04] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 18:50:04] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 18:51:02] ERROR Worker (pid:103) was sent code 132! [12/11/23 18:51:04] INFO Booting worker with pid: 127 [12/11/23 18:52:45] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 18:52:45] INFO Initialized request thread pool with 6 threads. [12/11/23 18:52:45] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 18:53:06] INFO Loading facial recognition model 'buffalo_l' [12/11/23 18:53:19] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 18:53:19] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 18:54:49] ERROR Worker (pid:127) was sent code 132! [12/11/23 18:54:50] INFO Booting worker with pid: 160 [12/11/23 18:56:50] CRITICAL WORKER TIMEOUT (pid:160) [12/11/23 18:56:54] ERROR Worker (pid:160) was sent SIGKILL! Perhaps out of memory? [12/11/23 18:56:54] INFO Booting worker with pid: 172 [12/11/23 18:58:41] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 18:58:41] INFO Initialized request thread pool with 6 threads. [12/11/23 18:58:41] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 18:59:23] INFO Loading facial recognition model 'buffalo_l' [12/11/23 18:59:50] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 18:59:51] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:01:37] ERROR Worker (pid:172) was sent code 132! [12/11/23 19:01:38] INFO Booting worker with pid: 199 [12/11/23 19:02:20] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:02:20] INFO Initialized request thread pool with 6 threads. [12/11/23 19:02:20] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:02:20] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:02:57] ERROR Worker (pid:199) was sent code 132! [12/11/23 19:02:57] INFO Booting worker with pid: 222 [12/11/23 19:03:23] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:03:23] INFO Initialized request thread pool with 6 threads. [12/11/23 19:03:23] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:03:25] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:03:26] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:04:08] ERROR Worker (pid:222) was sent code 132! [12/11/23 19:04:08] INFO Booting worker with pid: 252 [12/11/23 19:04:45] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:04:45] INFO Initialized request thread pool with 6 threads. [12/11/23 19:04:45] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:04:45] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:05:17] ERROR Worker (pid:252) was sent code 132! [12/11/23 19:05:17] INFO Booting worker with pid: 275 [12/11/23 19:05:39] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:05:39] INFO Initialized request thread pool with 6 threads. [12/11/23 19:05:39] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:05:39] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:06:05] ERROR Worker (pid:275) was sent code 132! [12/11/23 19:06:05] INFO Booting worker with pid: 298 [12/11/23 19:06:34] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:06:34] INFO Initialized request thread pool with 6 threads. [12/11/23 19:06:34] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:06:36] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:06:36] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:07:18] ERROR Worker (pid:298) was sent code 132! [12/11/23 19:07:18] INFO Booting worker with pid: 328 [12/11/23 19:07:45] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:07:45] INFO Initialized request thread pool with 6 threads. [12/11/23 19:07:45] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:07:45] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:08:34] ERROR Worker (pid:328) was sent code 132! [12/11/23 19:08:34] INFO Booting worker with pid: 351 [12/11/23 19:08:58] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:08:58] INFO Initialized request thread pool with 6 threads. [12/11/23 19:08:58] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:09:00] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:09:06] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:09:06] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:10:42] ERROR Worker (pid:351) was sent code 132! [12/11/23 19:10:42] INFO Booting worker with pid: 382 [12/11/23 19:11:15] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:11:15] INFO Initialized request thread pool with 6 threads. [12/11/23 19:11:15] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:11:18] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:11:18] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:12:25] ERROR Worker (pid:382) was sent code 132! [12/11/23 19:12:25] INFO Booting worker with pid: 412 [12/11/23 19:12:56] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:12:56] INFO Initialized request thread pool with 6 threads. [12/11/23 19:12:56] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:12:57] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:13:02] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:13:03] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:14:57] CRITICAL WORKER TIMEOUT (pid:412) [12/11/23 19:15:15] ERROR Worker (pid:412) was sent SIGKILL! Perhaps out of memory? [12/11/23 19:15:16] INFO Booting worker with pid: 443 [12/11/23 19:15:46] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:15:46] INFO Initialized request thread pool with 6 threads. [12/11/23 19:15:46] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:15:46] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:16:39] ERROR Worker (pid:443) was sent code 132! [12/11/23 19:16:39] INFO Booting worker with pid: 466 [12/11/23 19:17:02] INFO Created in-memory cache with unloading after 300s of inactivity. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:17:02] INFO Initialized request thread pool with 6 threads. [12/11/23 19:17:02] INFO Loading image classification model 'microsoft/resnet-50' [12/11/23 19:17:02] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: 12/11/23 19:17:31] ERROR Worker (pid:466) was sent code 132! [12/11/23 19:17:31] INFO Booting worker with pid: 489 [12/11/23 19:18:00] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:18:00] INFO Initialized request thread pool with 6 threads. [12/11/23 19:18:00] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:18:04] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:18:04] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:18:44] ERROR Worker (pid:489) was sent code 132! [12/11/23 19:18:44] INFO Booting worker with pid: 520 [12/11/23 19:19:16] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:19:16] INFO Initialized request thread pool with 6 threads. [12/11/23 19:19:16] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:19:18] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:19:27] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:19:27] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:21:17] CRITICAL WORKER TIMEOUT (pid:520) [12/11/23 19:21:29] ERROR Worker (pid:520) was sent code 132! [12/11/23 19:21:30] INFO Booting worker with pid: 551 [12/11/23 19:22:08] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:22:08] INFO Initialized request thread pool with 6 threads. [12/11/23 19:22:08] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:22:10] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:22:10] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:23:14] ERROR Worker (pid:551) was sent code 132! [12/11/23 19:23:14] INFO Booting worker with pid: 581 [12/11/23 19:23:55] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:23:55] INFO Initialized request thread pool with 6 threads. [12/11/23 19:23:55] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:23:56] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:24:03] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:24:04] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:25:56] CRITICAL WORKER TIMEOUT (pid:581) [12/11/23 19:26:05] ERROR Worker (pid:581) was sent code 132! [12/11/23 19:26:05] INFO Booting worker with pid: 612 [12/11/23 19:26:45] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:26:45] INFO Initialized request thread pool with 6 threads. [12/11/23 19:26:45] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:26:47] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:26:49] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:27:57] ERROR Worker (pid:612) was sent code 132! [12/11/23 19:27:57] INFO Booting worker with pid: 642 [12/11/23 19:28:31] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:28:31] INFO Initialized request thread pool with 6 threads. [12/11/23 19:28:31] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:28:34] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:28:34] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:29:36] ERROR Worker (pid:642) was sent code 132! [12/11/23 19:29:36] INFO Booting worker with pid: 672 [12/11/23 19:30:09] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:30:09] INFO Initialized request thread pool with 6 threads. [12/11/23 19:30:09] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:30:15] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:30:15] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:31:46] ERROR Worker (pid:672) was sent code 132! [12/11/23 19:31:46] INFO Booting worker with pid: 696 [12/11/23 19:32:17] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:32:17] INFO Initialized request thread pool with 6 threads. [12/11/23 19:32:17] INFO Loading facial recognition model 'buffalo_l' [12/11/23 19:32:18] INFO Loading clip model 'ViT-B-32__openai' [12/11/23 19:32:26] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:32:26] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. '(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: d794b736-f75b-4dbb-a242-08103bb6712e)')' thrown while requesting HEAD https://huggingface.co/microsoft/resnet-5 0/resolve/main/config.json [12/11/23 19:32:38] WARNING '(ReadTimeoutError("HTTPSConnectionPool(host='huggi ngface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: d794b736-f75b-4dbb-a242-08103bb6712e)')' thrown while requesting HEAD https://huggingface.co/microsoft/resnet-50/resolve/ main/config.json Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: [12/11/23 19:36:49] ERROR Worker (pid:696) was sent code 132! [12/11/23 19:36:49] INFO Booting worker with pid: 727 [12/11/23 19:37:29] INFO Created in-memory cache with unloading after 300s of inactivity. [12/11/23 19:37:29] INFO Initialized request thread pool with 6 threads. [12/11/23 19:37:29] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. [12/11/23 19:37:29] INFO ONNX model not found in cache directory for 'microsoft/resnet-50'.Exporting optimized model for future use. Framework not specified. Using pt to export to ONNX. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNext ImageProcessor instead. warnings.warn( Using the export variant default. Available variants are: - default: The default ONNX variant. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Using framework PyTorch: 2.1.0 /opt/venv/lib/python3.11/site-packages/transformers/models/resnet/modeling_resnet.py:95: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so th is value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if num_channels != self.num_channels: ``` Lots of error's with code 132 here, and even a critical error at the beginning, with the suggestion it may have run out of memory? This device has 3.63GB of ram and 2GB of swap. I haven't seen any signs of low memory issues when any other programs that are running, and right now it is quite happily sitting at 50% memory usage.
OVERLORD commented 2026-02-05 03:40:37 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 11, 2023):

Oh, so the core dumps aren't because of the idle shutdown, but because it's crashing repeatedly. It looks like you have object tagging jobs running. I think the model exporting process for this job is buggy on some ARM devices, so try disabling Image Tagging in the admin settings and restarting the stack. We should still look into avoiding core dumps for SIGKILLs either way.

@mertalev commented on GitHub (Dec 11, 2023): Oh, so the core dumps aren't because of the idle shutdown, but because it's crashing repeatedly. It looks like you have object tagging jobs running. I think the model exporting process for this job is buggy on some ARM devices, so try disabling Image Tagging in the admin settings and restarting the stack. We should still look into avoiding core dumps for SIGKILLs either way.
OVERLORD commented 2026-02-05 03:40:40 +03:00
Author
Owner
Copy Link

@BlueWolfProduction commented on GitHub (Dec 11, 2023):

Yes, disabling the tagging system and restarting it seems to indeed prevent it from filling up my storage.

What exactly is the difference between the 'Tag objects' and 'Encode clip' job? I am trying to figure out on what feature exactly I am missing out on. Both seems to be about classifying the images?

Oh, so the core dumps aren't because of the idle shutdown, but because it's crashing repeatedly. It looks like you have object tagging jobs running. I think the model exporting process for this job is buggy on some ARM devices, so try disabling Image Tagging in the admin settings and restarting the stack. We should still look into avoiding core dumps for SIGKILLs either way.

When I took that log, the tagging system was still running. Do you think it's worth it for me to keep it running until it's saying the job is finished, but it is still filling storage, to see if the logs are any different? Or do you expect the errors/logs to be the same? Probably won't be until tomorrow when I can do that as I need to be around making sure it doesn't completely fill the other drive.

By the way, thank you very much for all of these quick responses. I appreciate it.

@BlueWolfProduction commented on GitHub (Dec 11, 2023): Yes, disabling the tagging system and restarting it seems to indeed prevent it from filling up my storage. What exactly is the difference between the 'Tag objects' and 'Encode clip' job? I am trying to figure out on what feature exactly I am missing out on. Both seems to be about classifying the images? > Oh, so the core dumps aren't because of the idle shutdown, but because it's crashing repeatedly. It looks like you have object tagging jobs running. I think the model exporting process for this job is buggy on some ARM devices, so try disabling Image Tagging in the admin settings and restarting the stack. We should still look into avoiding core dumps for SIGKILLs either way. When I took that log, the tagging system was still running. Do you think it's worth it for me to keep it running until it's saying the job is finished, but it is still filling storage, to see if the logs are any different? Or do you expect the errors/logs to be the same? Probably won't be until tomorrow when I can do that as I need to be around making sure it doesn't completely fill the other drive. By the way, thank you very much for all of these quick responses. I appreciate it.
OVERLORD commented 2026-02-05 03:40:42 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 12, 2023):

The Tag Images job generates textual tags like "tabby cat" or "scuba diver" that can appear in the Explore page and used for metadata searches. The Encode CLIP job allows for more powerful searches that fetch images based on similarity to the query, not whether the image has a matching keyword. The Tag Images job predates the Encode CLIP job and is generally vestigial. It will probably be removed once we add smart albums.

@mertalev commented on GitHub (Dec 12, 2023): The Tag Images job generates textual tags like "tabby cat" or "scuba diver" that can appear in the Explore page and used for metadata searches. The Encode CLIP job allows for more powerful searches that fetch images based on similarity to the query, not whether the image has a matching keyword. The Tag Images job predates the Encode CLIP job and is generally vestigial. It will probably be removed once we add smart albums.
OVERLORD commented 2026-02-05 03:40:48 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 12, 2023):

The tagging job will never successfully complete in your case, so I don't recommend keeping it running.

@mertalev commented on GitHub (Dec 12, 2023): The tagging job will never successfully complete in your case, so I don't recommend keeping it running.
OVERLORD commented 2026-02-05 03:40:50 +03:00
Author
Owner
Copy Link

@aviv926 commented on GitHub (Dec 12, 2023):

Seems like a recurring problem

https://github.com/immich-app/immich/issues/5102

@aviv926 commented on GitHub (Dec 12, 2023): Seems like a recurring problem https://github.com/immich-app/immich/issues/5102
OVERLORD commented 2026-02-05 03:40:59 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 12, 2023):

Yeah, seems like the root cause was this and not the requests being spooled 😅

@mertalev commented on GitHub (Dec 12, 2023): Yeah, seems like the root cause was this and not the requests being spooled 😅
OVERLORD commented 2026-02-05 03:41:03 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 15, 2023):

@mertalev it's still possible for the ML service to generate core dumps, right? Should we keep this issue open until that has been disabled?

@bo0tzz commented on GitHub (Dec 15, 2023): @mertalev it's still possible for the ML service to generate core dumps, right? Should we keep this issue open until that has been disabled?
OVERLORD commented 2026-02-05 03:41:07 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Dec 15, 2023):

Yeah that makes sense. I think we need to make this change to disable core dumps https://stackoverflow.com/a/64007494

@mertalev commented on GitHub (Dec 15, 2023): Yeah that makes sense. I think we need to make this change to disable core dumps https://stackoverflow.com/a/64007494
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1769
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?