immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

[BUG] Gunicorn CPU & Memory utilization high even when no jobs running #769

New Issue
Closed
opened 2026-02-04 22:25:04 +03:00 by OVERLORD · 12 comments
OVERLORD commented 2026-02-04 22:25:04 +03:00
Owner
Copy Link

Originally created by @uninvitedguest on GitHub (Mar 24, 2023).

The bug

After updating to include the typesense container and running the "Encode Clip" job, CPU and memory utilization by Gunicorn is still incredibly high.

image

Restarting the containers results in more moderate usage.

image

Running on a Rock 5B SBC.

The OS that Immich Server is running on

Armbian 23.02 Jammy

Version of Immich Server

1.51.1

Version of Immich Mobile App

1.51.0 Build 74

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: [ "/bin/sh", "./start-server.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    networks:
      - immich-internal

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: [ "/bin/sh", "./start-microservices.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always
    networks:
      - immich-internal

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/ssd/immich/model-cache:/cache
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    restart: always
    networks:
      - immich-internal

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: [ "/bin/sh", "./entrypoint.sh" ]
    env_file:
      - stack.env
    restart: always
    networks:
      - immich-internal

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - /mnt/ssd/immich/tsdata:/data
    env_file:
      - stack.env
    networks:
      - immich-internal
      
  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always
    networks:
      - immich-internal

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - /mnt/ssd/immich/database:/var/lib/postgresql/data
    restart: always
    networks:
      - immich-internal
  
  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always
    networks:
      - proxy
      - immich-internal

volumes:
  pgdata:
  model-cache:
  tsdata:

networks:
  proxy:
    external: true
  immich-internal:
    external: true

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/ssd/immich/upload
TYPESENSE_API_KEY=api-key
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

1. Click "All" to queue the Encode Clip job
2. Wait 2 days (in my case).
3. When all jobs are complete, check CPU utilization in gotop

Additional information

No response

Originally created by @uninvitedguest on GitHub (Mar 24, 2023). ### The bug After updating to include the typesense container and running the "Encode Clip" job, CPU and memory utilization by Gunicorn is still incredibly high. ![image](https://user-images.githubusercontent.com/24210277/227546781-cfd060ed-7700-4085-ab58-08f074281dfe.png) Restarting the containers results in more moderate usage. ![image](https://user-images.githubusercontent.com/24210277/227553253-a5ae2115-a91b-4b17-9bee-1757f9ee7e7f.png) Running on a Rock 5B SBC. ### The OS that Immich Server is running on Armbian 23.02 Jammy ### Version of Immich Server 1.51.1 ### Version of Immich Mobile App 1.51.0 Build 74 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:release entrypoint: [ "/bin/sh", "./start-server.sh" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - stack.env environment: - NODE_ENV=production depends_on: - redis - database - typesense networks: - immich-internal immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:release entrypoint: [ "/bin/sh", "./start-microservices.sh" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - stack.env environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always networks: - immich-internal immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:release volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /mnt/ssd/immich/model-cache:/cache env_file: - stack.env environment: - NODE_ENV=production restart: always networks: - immich-internal immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:release entrypoint: [ "/bin/sh", "./entrypoint.sh" ] env_file: - stack.env restart: always networks: - immich-internal typesense: container_name: immich_typesense image: typesense/typesense:0.24.0 environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data logging: driver: none volumes: - /mnt/ssd/immich/tsdata:/data env_file: - stack.env networks: - immich-internal redis: container_name: immich_redis image: redis:6.2 restart: always networks: - immich-internal database: container_name: immich_postgres image: postgres:14 env_file: - stack.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - /mnt/ssd/immich/database:/var/lib/postgresql/data restart: always networks: - immich-internal immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:release environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 logging: driver: none depends_on: - immich-server restart: always networks: - proxy - immich-internal volumes: pgdata: model-cache: tsdata: networks: proxy: external: true immich-internal: external: true ``` ### Your .env content ```Shell DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_PASSWORD=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis UPLOAD_LOCATION=/mnt/ssd/immich/upload TYPESENSE_API_KEY=api-key PUBLIC_LOGIN_PAGE_MESSAGE= IMMICH_WEB_URL=http://immich-web:3000 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003 ``` ### Reproduction steps ```bash 1. Click "All" to queue the Encode Clip job 2. Wait 2 days (in my case). 3. When all jobs are complete, check CPU utilization in gotop ``` ### Additional information _No response_
OVERLORD added the 🗄️server label 2026-02-04 22:25:04 +03:00
OVERLORD commented 2026-02-04 22:25:11 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 24, 2023):

At the moment, the machine learning models are configured to be lazy loaded on an as-needs basis. However, once loaded, they stay in memory, which is what the memory usage remains high. Restarting the container is a temporary way to reduce the usage, yes. I have not found a way to programatically reduce the memory usage so far. Related, there may be an issue where multiple processes are loading the same model, resulting in even high usage. That last one requires further investigation and should be addressable.

@jrasm91 commented on GitHub (Mar 24, 2023): At the moment, the machine learning models are configured to be lazy loaded on an as-needs basis. However, once loaded, they stay in memory, which is what the memory usage remains high. Restarting the container is a temporary way to reduce the usage, yes. I have not found a way to programatically reduce the memory usage so far. Related, there may be an issue where multiple processes are loading the same model, resulting in even high usage. That last one requires further investigation and should be addressable.
OVERLORD commented 2026-02-04 22:25:15 +03:00
Author
Owner
Copy Link

@lukaszwawrzyk commented on GitHub (Apr 22, 2023):

This would explain the memory usage, but what about CPU? I seem to have similar problem. I am not using Immich on my mobile or web and still it takes 1 core of CPU for this gunicorn.

@lukaszwawrzyk commented on GitHub (Apr 22, 2023): This would explain the memory usage, but what about CPU? I seem to have similar problem. I am not using Immich on my mobile or web and still it takes 1 core of CPU for this gunicorn.
OVERLORD commented 2026-02-04 22:25:20 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Apr 22, 2023):

@lukaszwawrzyk was there anything running in the job panel at the time? What is your docker stats output look like?

@alextran1502 commented on GitHub (Apr 22, 2023): @lukaszwawrzyk was there anything running in the job panel at the time? What is your `docker stats` output look like?
OVERLORD commented 2026-02-04 22:25:25 +03:00
Author
Owner
Copy Link

@lukaszwawrzyk commented on GitHub (Apr 22, 2023):

It seems like there are no jobs running
image
Cpu usage ranges from 60-130%
image

@lukaszwawrzyk commented on GitHub (Apr 22, 2023): It seems like there are no jobs running ![image](https://user-images.githubusercontent.com/10235198/233773224-cb473feb-cc2f-4ff0-8e33-dad6c489bc5f.png) Cpu usage ranges from 60-130% ![image](https://user-images.githubusercontent.com/10235198/233773271-95621195-7f47-4a1b-b4fa-f2ffa4f6e1f4.png)
OVERLORD commented 2026-02-04 22:25:34 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Apr 22, 2023):

@lukaszwawrzyk ate you using a single docker image for Immich? The all-in-one one?

@alextran1502 commented on GitHub (Apr 22, 2023): @lukaszwawrzyk ate you using a single docker image for Immich? The all-in-one one?
OVERLORD commented 2026-02-04 22:25:40 +03:00
Author
Owner
Copy Link

@lukaszwawrzyk commented on GitHub (Apr 22, 2023):

Yes, I am using it through the Home Assistant addon: https://github.com/alexbelgium/hassio-addons/tree/master/immich

@lukaszwawrzyk commented on GitHub (Apr 22, 2023): Yes, I am using it through the Home Assistant addon: https://github.com/alexbelgium/hassio-addons/tree/master/immich
OVERLORD commented 2026-02-04 22:25:43 +03:00
Author
Owner
Copy Link

@domeq commented on GitHub (Apr 29, 2023):

I'm having the same problem. I just installed immich for the first time by following the recommended docker-compose setup.

I turned off all video transcoding to not overload the server, uploaded ~150 pictures and a few short 4k hevc videos from an Android mobile phone using the app. Everything went smoothly, but then gunicorn processes kept consuming CPU even though no jobs were running according to UI.

Had to turn everything off for now, it made my server overheat and become unstable.

edit: I restarted it and CPU load is no longer the problem. It seems like some jobs related to processing uploaded media didn't want to finish.

@domeq commented on GitHub (Apr 29, 2023): I'm having the same problem. I just installed immich for the first time by following the [recommended docker-compose setup](https://documentation.immich.app/docs/install/docker-compose). I turned off all video transcoding to not overload the server, uploaded ~150 pictures and a few short 4k hevc videos from an Android mobile phone using the app. Everything went smoothly, but then `gunicorn` processes kept consuming CPU even though no jobs were running according to UI. Had to turn everything off for now, it made my server overheat and become unstable. **edit**: I restarted it and CPU load is no longer the problem. It seems like some jobs related to processing uploaded media didn't want to finish.
OVERLORD commented 2026-02-04 22:25:49 +03:00
Author
Owner
Copy Link

@swarupbhc commented on GitHub (Jul 21, 2023):

At the moment, the machine learning models are configured to be lazy loaded on an as-needs basis. However, once loaded, they stay in memory, which is what the memory usage remains high. Restarting the container is a temporary way to reduce the usage, yes. I have not found a way to programatically reduce the memory usage so far. Related, there may be an issue where multiple processes are loading the same model, resulting in even high usage. That last one requires further investigation and should be addressable.

@jrasm91 have you found any way to reduce the ram usage programmatically? It's really causing problems, so I had to disable machine learning feature manually. but I still need those feature in the future. please look into it.

@swarupbhc commented on GitHub (Jul 21, 2023): > At the moment, the machine learning models are configured to be lazy loaded on an as-needs basis. However, once loaded, they stay in memory, which is what the memory usage remains high. Restarting the container is a temporary way to reduce the usage, yes. I have not found a way to programatically reduce the memory usage so far. Related, there may be an issue where multiple processes are loading the same model, resulting in even high usage. That last one requires further investigation and should be addressable. @jrasm91 have you found any way to reduce the ram usage programmatically? It's really causing problems, so I had to disable machine learning feature manually. but I still need those feature in the future. please look into it.
OVERLORD commented 2026-02-04 22:25:52 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jul 21, 2023):

@swarupbhc you can check the progress here https://github.com/immich-app/immich/pull/3340

@alextran1502 commented on GitHub (Jul 21, 2023): @swarupbhc you can check the progress here https://github.com/immich-app/immich/pull/3340
OVERLORD commented 2026-02-04 22:25:58 +03:00
Author
Owner
Copy Link

@AverTry commented on GitHub (Feb 20, 2024):

Kept crashing one pc, setup on another, and the heat is a joke. After external library is done with thumbnails, AI turned off, gunicorn won't let go of the CPU. This is a PC killer.
image

@AverTry commented on GitHub (Feb 20, 2024): Kept crashing one pc, setup on another, and the heat is a joke. After external library is done with thumbnails, AI turned off, gunicorn won't let go of the CPU. This is a PC killer. ![image](https://github.com/immich-app/immich/assets/72357571/602f5048-8591-49d2-b0f5-6670cc1492d7)
OVERLORD commented 2026-02-04 22:26:04 +03:00
Author
Owner
Copy Link

@rochiey commented on GitHub (Jan 31, 2025):

Sheesh, I thought I'm the only one.
Image

@rochiey commented on GitHub (Jan 31, 2025): Sheesh, I thought I'm the only one. ![Image](https://github.com/user-attachments/assets/6ed5af7c-4fe1-406b-b012-c534a4262951)
OVERLORD commented 2026-02-04 22:26:09 +03:00
Author
Owner
Copy Link

@GiorgioMendieta commented on GitHub (Feb 16, 2025):

Same here

Image

@GiorgioMendieta commented on GitHub (Feb 16, 2025): Same here ![Image](https://github.com/user-attachments/assets/3c26732d-037f-41bf-a498-7c5aeeb4daa7)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#769
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?