immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

[BUG] Resources Utilization Spike #785

New Issue
Closed
opened 2026-02-04 22:37:32 +03:00 by OVERLORD · 12 comments
OVERLORD commented 2026-02-04 22:37:32 +03:00
Owner
Copy Link

Originally created by @azurecrash on GitHub (Mar 31, 2023).

The bug

I’ve deployed Immich and am trying to backup photos from iOS but after like 15 images, the resources on the server (an older 4 core x86 Mac mini) get DESTROYED by the micro services container and what appears to be ffmpeg. I am therefore assuming (perhaps incorrectly) that the transcode is killing the backup. Has been like 8 hours since deploy and I have 100 of 30k assets backed up!

My setup:
Compute: 2011 Mac mini, 4 core i5 x86, 4gb memory running Debian
Storage: Synology NAS drive, SMB share, mounted to the Mac mini

Other details:
The containers are all deployed through Portainer. Mobile app seems to work fine. Web interface seems to work fine. In writing this post and fetching the logs, I've noticed that the micro services container has restarted 104 times since it was created....

Logs from the micro services container are attached. Last 1000 lines.

_immich_microservices_logs-5.txt

Here is a graph of CPU load pre-and-post one of these backup spurts:

Screenshot 2023-03-30 at 10 15 38 PM

The OS that Immich Server is running on

Debian

Version of Immich Server

1.52.1

Version of Immich Mobile App

1.52.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - stack.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:
  
  #

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/asp-photos/
TYPESENSE_API_KEY=some-random-text
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

1.Instantiate containers with docker compose above.
2.Create admin user
3.Download and install iOS app (latest)
4.Initiate a foreground or background or manual backup
5.Observe 6-10 assets get uploaded then pause
6.Observe Server resource usage in Grafana get pegged

Additional information

No response

Originally created by @azurecrash on GitHub (Mar 31, 2023). ### The bug I’ve deployed Immich and am trying to backup photos from iOS but after like 15 images, the resources on the server (an older 4 core x86 Mac mini) get DESTROYED by the micro services container and what appears to be ffmpeg. I am therefore assuming (perhaps incorrectly) that the transcode is killing the backup. Has been like 8 hours since deploy and I have 100 of 30k assets backed up! My setup: Compute: 2011 Mac mini, 4 core i5 x86, 4gb memory running Debian Storage: Synology NAS drive, SMB share, mounted to the Mac mini Other details: The containers are all deployed through Portainer. Mobile app seems to work fine. Web interface seems to work fine. In writing this post and fetching the logs, I've noticed that the micro services container has restarted 104 times since it was created.... Logs from the micro services container are attached. Last 1000 lines. [_immich_microservices_logs-5.txt](https://github.com/immich-app/immich/files/11117508/_immich_microservices_logs-5.txt) Here is a graph of CPU load pre-and-post one of these backup spurts: <img width="425" alt="Screenshot 2023-03-30 at 10 15 38 PM" src="https://user-images.githubusercontent.com/13999581/229005815-a6bbecf5-8e94-4c14-8cfa-b5a9e33f8246.png"> ### The OS that Immich Server is running on Debian ### Version of Immich Server 1.52.1 ### Version of Immich Mobile App 1.52.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-server.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - stack.env environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-microservices.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - stack.env environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:release volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - model-cache:/cache env_file: - stack.env environment: - NODE_ENV=production restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:release entrypoint: ["/bin/sh", "./entrypoint.sh"] env_file: - stack.env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.0 environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data logging: driver: none volumes: - tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2 restart: always database: container_name: immich_postgres image: postgres:14 env_file: - stack.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:release environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 logging: driver: none depends_on: - immich-server restart: always volumes: pgdata: model-cache: tsdata: # ``` ### Your .env content ```Shell DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_PASSWORD=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis UPLOAD_LOCATION=/mnt/asp-photos/ TYPESENSE_API_KEY=some-random-text PUBLIC_LOGIN_PAGE_MESSAGE= IMMICH_WEB_URL=http://immich-web:3000 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003 ``` ### Reproduction steps ```bash 1.Instantiate containers with docker compose above. 2.Create admin user 3.Download and install iOS app (latest) 4.Initiate a foreground or background or manual backup 5.Observe 6-10 assets get uploaded then pause 6.Observe Server resource usage in Grafana get pegged ``` ### Additional information _No response_
OVERLORD commented 2026-02-04 22:37:39 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

Try recreating your microservices container or mounting the reverse geocoding folder and delete the invalid file.

@jrasm91 commented on GitHub (Mar 31, 2023): Try recreating your microservices container or mounting the reverse geocoding folder and delete the invalid file.
OVERLORD commented 2026-02-04 22:37:46 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

See https://github.com/immich-app/immich/issues/1104 for more details.

@jrasm91 commented on GitHub (Mar 31, 2023): See https://github.com/immich-app/immich/issues/1104 for more details.
OVERLORD commented 2026-02-04 22:37:52 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

Recreating the container has no effect.

Don't follow re: mounting the reverse geocode folder; I don't see a bind mount in docker-compose. What's the invalid file? That is not clear from the logs.

@azurecrash commented on GitHub (Mar 31, 2023): Recreating the container has no effect. Don't follow re: mounting the reverse geocode folder; I don't see a bind mount in docker-compose. What's the invalid file? That is not clear from the logs.
OVERLORD commented 2026-02-04 22:37:56 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

Did you read through #1104?

@jrasm91 commented on GitHub (Mar 31, 2023): Did you read through #1104?
OVERLORD commented 2026-02-04 22:38:00 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

Yes, I did, and while the symptoms seem similar, the solution is unclear.

  1. The location of the reverse geocode file downloaded from geonames.org is unclear; however, I validated that there's no issue access that URL.
  2. I cannot test anything from inside the container; attempting to login to it (bash, ash, sh) via portainer fails every time with an error thrown that the container is restarting
  3. I have modified docker compose to mount all volumes to folders on the host, will see if that makes a difference....
@azurecrash commented on GitHub (Mar 31, 2023): Yes, I did, and while the symptoms seem similar, the solution is unclear. 1) The location of the reverse geocode file downloaded from geonames.org is unclear; however, I validated that there's no issue access that URL. 2) I cannot test anything from inside the container; attempting to login to it (bash, ash, sh) via portainer fails every time with an error thrown that the container is restarting 3) I have modified docker compose to mount all volumes to folders on the host, will see if that makes a difference....
OVERLORD commented 2026-02-04 22:38:04 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

There is a folder in the microservices container at this location: /usr/src/app/.reverse-geocoding-dump, which contains some invalid data. I would probably recommend adding a new volume mount for the microservices container that's something like this:

    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /path/to/some/folder:/usr/src/app/.reverse-geocoding-dump

Then, delete any files inside that folder, and restart the microservices container.

@jrasm91 commented on GitHub (Mar 31, 2023): There is a folder in the microservices container at this location: `/usr/src/app/.reverse-geocoding-dump`, which contains some invalid data. I would probably recommend adding a new volume mount for the microservices container that's something like this: ``` volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /path/to/some/folder:/usr/src/app/.reverse-geocoding-dump ``` Then, delete any files inside that folder, and restart the microservices container.
OVERLORD commented 2026-02-04 22:38:10 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

You got it; will give it a shot now!

@azurecrash commented on GitHub (Mar 31, 2023): You got it; will give it a shot now!
OVERLORD commented 2026-02-04 22:38:13 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

Some progress; mapping that folder to the host seems to have alleviated (or at least slowed down) the container restart issue. Resources are still absolutely jammed, in accordance with my graph I showed in the original post. If it's helpful, here's the latest log dump.
_immich_microservices_logs-6.txt

@azurecrash commented on GitHub (Mar 31, 2023): Some progress; mapping that folder to the host seems to have alleviated (or at least slowed down) the container restart issue. Resources are still absolutely jammed, in accordance with my graph I showed in the original post. If it's helpful, here's the latest log dump. [_immich_microservices_logs-6.txt](https://github.com/immich-app/immich/files/11123047/_immich_microservices_logs-6.txt)
OVERLORD commented 2026-02-04 22:38:17 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

FYI - If you hop into discord you can open a help-support-ticket to get faster responses.

Looking at the logs looks like there's a few things going on:

  1. Machine learning definitely uses a lot of resources, you may want to disable it during the initial upload. You can set MACHINE_LEARNING_URL=false to accomplish that.
  2. Transcoding is failing because it can't find the file. This seems to have been a bug introduced in the latest release. The file is getting moved to the "storage template path", but the video encoding job is trying to read it from the old path. Not a big deal as you can run the transcode video jobs for "missing" from the admin screen after the uploads finish.
@jrasm91 commented on GitHub (Mar 31, 2023): FYI - If you hop into discord you can open a help-support-ticket to get faster responses. Looking at the logs looks like there's a few things going on: 1. Machine learning definitely uses a lot of resources, you may want to disable it during the initial upload. You can set `MACHINE_LEARNING_URL=false` to accomplish that. 2. Transcoding is failing because it can't find the file. This seems to have been a bug introduced in the latest release. The file is getting moved to the "storage template path", but the video encoding job is trying to read it from the old path. Not a big deal as you can run the transcode video jobs for "missing" from the admin screen after the uploads finish.
OVERLORD commented 2026-02-04 22:38:19 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

Thanks so much! Let me give that env tweak a try. Will look into hopping onto Discord ASAP. Thanks again!

@azurecrash commented on GitHub (Mar 31, 2023): Thanks so much! Let me give that env tweak a try. Will look into hopping onto Discord ASAP. Thanks again!
OVERLORD commented 2026-02-04 22:38:25 +03:00
Author
Owner
Copy Link

@azurecrash commented on GitHub (Mar 31, 2023):

Turning off ML has definitely made the system more performant and stable, though CPU load is still getting as high as 2k% to 4k%. Is this "normal"? Thanks again!

@azurecrash commented on GitHub (Mar 31, 2023): Turning off ML has definitely made the system more performant and stable, though CPU load is still getting as high as 2k% to 4k%. Is this "normal"? Thanks again!
OVERLORD commented 2026-02-04 22:38:28 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 31, 2023):

Resolved in Discord by adding CPU limitation to the microservices container

@alextran1502 commented on GitHub (Mar 31, 2023): Resolved in Discord by adding CPU limitation to the microservices container
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#785
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?