immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 12 Packages Projects Releases 10 Wiki Activity

[BUG] CLI error when uploading a folder with 8k images #1664

New Issue
Closed
opened 2026-02-05 02:56:22 +03:00 by OVERLORD · 8 comments
OVERLORD commented 2026-02-05 02:56:22 +03:00
Owner
Copy Link

Originally created by @SitramSoft on GitHub (Nov 21, 2023).

The bug

When I tried to upload a folder with over 8k images and videos using the new CLI, I got the following error. I think the node process ran out of memory then was killed by the OS. My laptop runs Arch and has 16 GB of RAM memory. The process consumed according to the Resource Monitor, a bit over 12 GB RAM before I got the error.

image

Uploading a folder that contained around 1.3k images and videos using the new CLI worked.

Below is a snippet from the content of the folder and the file count:
image

The OS that Immich Server is running on

Ubuntu 22.04.2 LTS

Version of Immich Server

v1.88.1

Version of Immich Mobile App

v1.88.0 build.112

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.9"

services:
#Immich - self hosted photos and videos management application - https://immich.app/
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    depends_on:
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    depends_on:
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    cpus: 6
    restart: always


volumes:
  model-cache:
  tsdata:

Your .env content

COMPOSE_PROJECT_NAME=serenity_immich

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/home/sitram/data/photos_immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
# Typesense
TYPESENSE_ENABLED=true
TYPESENSE_API_KEY=xxx
DB_PASSWORD=xxx

# Timezone
TZ=Europe/Bucharest

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=192.168.0.101
DB_USERNAME=xxx
DB_DATABASE_NAME=xxx

REDIS_HOSTNAME=192.168.0.101

Reproduction steps

1. immich login-key http://192.168.0.101:2283/api xxx
2. immich upload --dry-run  --recursive --album '2020 - Sabrina' /home/sitram/mounts/data/photos/Sabrina/2020

Additional information

Redis and database container are hosted in a common docker container for my entire homlab, that's why they are not part of the docker-compose.yml content

Originally created by @SitramSoft on GitHub (Nov 21, 2023). ### The bug When I tried to upload a folder with over 8k images and videos using the new CLI, I got the following error. I think the node process ran out of memory then was killed by the OS. My laptop runs Arch and has 16 GB of RAM memory. The process consumed according to the Resource Monitor, a bit over 12 GB RAM before I got the error. ![image](https://github.com/immich-app/immich/assets/8177290/13f36fbd-780b-4739-91c9-3ef2436f2409) Uploading a folder that contained around 1.3k images and videos using the new CLI worked. Below is a snippet from the content of the folder and the file count: ![image](https://github.com/immich-app/immich/assets/8177290/53b1cf6f-a90f-40fe-9b78-e3eb4d73411b) ### The OS that Immich Server is running on Ubuntu 22.04.2 LTS ### Version of Immich Server v1.88.1 ### Version of Immich Mobile App v1.88.0 build.112 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.9" services: #Immich - self hosted photos and videos management application - https://immich.app/ immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 labels: - "com.centurylinklabs.watchtower.enable=false" depends_on: - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env labels: - "com.centurylinklabs.watchtower.enable=false" depends_on: - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env labels: - "com.centurylinklabs.watchtower.enable=false" restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - tsdata:/data labels: - "com.centurylinklabs.watchtower.enable=false" cpus: 6 restart: always volumes: model-cache: tsdata: ``` ### Your .env content ```Shell COMPOSE_PROJECT_NAME=serenity_immich # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/home/sitram/data/photos_immich # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords # Typesense TYPESENSE_ENABLED=true TYPESENSE_API_KEY=xxx DB_PASSWORD=xxx # Timezone TZ=Europe/Bucharest # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=192.168.0.101 DB_USERNAME=xxx DB_DATABASE_NAME=xxx REDIS_HOSTNAME=192.168.0.101 ``` ### Reproduction steps ```bash 1. immich login-key http://192.168.0.101:2283/api xxx 2. immich upload --dry-run --recursive --album '2020 - Sabrina' /home/sitram/mounts/data/photos/Sabrina/2020 ``` ### Additional information Redis and database container are hosted in a common docker container for my entire homlab, that's why they are not part of the docker-compose.yml content
OVERLORD commented 2026-02-05 02:56:32 +03:00
Author
Owner
Copy Link

@esmoyer commented on GitHub (Nov 21, 2023):

Similar boat, wanting to move from the AIO install to the new compose; but hitting this limit on resources; I also noticed that it doesn't give a progress bar like the old CLI tool did, not sure if that is a bug or by design, not a huge issue.

@esmoyer commented on GitHub (Nov 21, 2023): Similar boat, wanting to move from the AIO install to the new compose; but hitting this limit on resources; I also noticed that it doesn't give a progress bar like the old CLI tool did, not sure if that is a bug or by design, not a huge issue.
OVERLORD commented 2026-02-05 02:56:42 +03:00
Author
Owner
Copy Link

@commanderalpha commented on GitHub (Nov 21, 2023):

Same problem here.
First the CLI is consuming huge ammount of memory depending on the image count. In my case ~17.5 GB for aorund 11k images.
Second issue: After the copy process is started, it gets slower and slower. I Aborted the process after around 1GB uploaded. Then the speed was around one pictures every 5 second. The first pictures copies fast as usual.

I tried this with the CLI directly in the Immich Container and also on my windows machine where i installed the CLI through npm.

@commanderalpha commented on GitHub (Nov 21, 2023): Same problem here. First the CLI is consuming huge ammount of memory depending on the image count. In my case ~17.5 GB for aorund 11k images. Second issue: After the copy process is started, it gets slower and slower. I Aborted the process after around 1GB uploaded. Then the speed was around one pictures every 5 second. The first pictures copies fast as usual. I tried this with the CLI directly in the Immich Container and also on my windows machine where i installed the CLI through npm.
OVERLORD commented 2026-02-05 02:56:53 +03:00
Author
Owner
Copy Link

@esmoyer commented on GitHub (Nov 22, 2023):

Definitely something weird going on. Tried running on smaller individual folders and it will crash my entire Proxmox server with CPU stalls but am not able to see anything in any logs. Resources don't seem to spike, that I notice.

@esmoyer commented on GitHub (Nov 22, 2023): Definitely something weird going on. Tried running on smaller individual folders and it will crash my entire Proxmox server with CPU stalls but am not able to see anything in any logs. Resources don't seem to spike, that I notice.
OVERLORD commented 2026-02-05 02:57:00 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 22, 2023):

@esmoyer how many ram do you have on the machine you are trying to run the cli on?

@alextran1502 commented on GitHub (Nov 22, 2023): @esmoyer how many ram do you have on the machine you are trying to run the cli on?
OVERLORD commented 2026-02-05 02:57:05 +03:00
Author
Owner
Copy Link

@esmoyer commented on GitHub (Nov 22, 2023):

4 CPU; 16GB RAM; 8GB Swap; it could very well be something else, but when I used the old CLI tool months ago it seemed to go a lot smoother. This is my first time using the new CLI tool, so not sure what to expect.

@esmoyer how many ram do you have on the machine you are trying to run the cli on?

@esmoyer commented on GitHub (Nov 22, 2023): 4 CPU; 16GB RAM; 8GB Swap; it could very well be something else, but when I used the old CLI tool months ago it seemed to go a lot smoother. This is my first time using the new CLI tool, so not sure what to expect. > @esmoyer how many ram do you have on the machine you are trying to run the cli on?
OVERLORD commented 2026-02-05 02:57:08 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 22, 2023):

@esmoyer can you update the CLI tool to the latest version 2.0.4 to see if it still causes the issue?

You can do npm i -g @immich/cli to update it

@alextran1502 commented on GitHub (Nov 22, 2023): @esmoyer can you update the CLI tool to the latest version 2.0.4 to see if it still causes the issue? You can do `npm i -g @immich/cli` to update it
OVERLORD commented 2026-02-05 02:57:11 +03:00
Author
Owner
Copy Link

@SitramSoft commented on GitHub (Nov 22, 2023):

@alextran1502 I can confirm that the immich upload command in --dry-run mode is successfully executed on two folders with a large amount of images and videos in them. Issue can be closed.

Thanks for a quick fix!

@SitramSoft commented on GitHub (Nov 22, 2023): @alextran1502 I can confirm that the `immich upload` command in `--dry-run` mode is successfully executed on two folders with a large amount of images and videos in them. Issue can be closed. Thanks for a quick fix!
OVERLORD commented 2026-02-05 02:57:15 +03:00
Author
Owner
Copy Link

@esmoyer commented on GitHub (Nov 22, 2023):

@alextran1502 seems to be working well now, thanks for the quick fix!

@esmoyer commented on GitHub (Nov 22, 2023): @alextran1502 seems to be working well now, thanks for the quick fix!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1664
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?