immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 26 Packages Projects Releases 10 Wiki Activity

"Error: EMFILE: too many open files" when adding large external library #2775

New Issue
Closed
opened 2026-02-05 07:02:26 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 07:02:26 +03:00
Owner
Copy Link

Originally created by @ehcheng on GitHub (Apr 7, 2024).

The bug

When scanning for new files on large external libraries, I am receiving errors like this:

immich_microservices | [Nest] 7 - 04/07/2024, 4:57:29 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/cold/Lytro/2011/20110520_evt2_dongguan'
immich_microservices | [Nest] 7 - 04/07/2024, 4:57:29 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/cold/Lytro/2011/20110520_evt2_dongguan'
immich_microservices | [Nest] 7 - 04/07/2024, 7:11:04 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/life/2011/20110305 Livia's concert, dinner'
immich_microservices | [Nest] 7 - 04/07/2024, 7:11:04 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/life/2011/20110305 Livia's concert, dinner'
immich_microservices | [Nest] 7 - 04/07/2024, 7:11:38 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/travel/Japan, Ogasawara 200910'
immich_microservices | [Nest] 7 - 04/07/2024, 7:11:38 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/travel/Japan, Ogasawara 200910'

There is 1 library task that spins up, and after it throws the error the task goes away.

One external library with 65K assets imported fine, as did another with 28K. but folders with more seem to be failing. I don't know the cut off, but some of the folders I'm importing have 100K+ assets.

The OS that Immich Server is running on

OS X Sonoma 14.4.1

Version of Immich Server

v1.101.0

Version of Immich Mobile App

web

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /Volumes/media/Pictures/travel_wildlife:/mnt/media/travel:ro
      - /Volumes/water/pictures/Life:/mnt/media/life_current:ro
      - /Volumes/water/pictures/Travel:/mnt/media/travel_current:ro
      - /Volumes/ice/frozen003-family/videos:/mnt/media/frozen003-family-videos:ro
      - /Volumes/ice/frozen003-family/scans:/mnt/media/frozen003-family-scans:ro
      - /Volumes/ice/cold/Pictures:/mnt/media/cold:ro
      - /Volumes/ice/frozen009-pictures-2019/Pictures/Travel:/mnt/media/frozen009-pictures-2019/travel:ro
      - /Volumes/ice/frozen009-pictures-2019/Pictures/Life:/mnt/media/frozen009-pictures-2019/life:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /Volumes/media/Pictures/travel_wildlife:/mnt/media/travel:ro
      - /Volumes/water/pictures/Life:/mnt/media/life_current:ro
      - /Volumes/water/pictures/Travel:/mnt/media/travel_current:ro
      - /Volumes/ice/frozen003-family/videos:/mnt/media/frozen003-family-videos:ro
      - /Volumes/ice/frozen003-family/scans:/mnt/media/frozen003-family-scans:ro
      - /Volumes/ice/cold/Pictures:/mnt/media/cold:ro
      - /Volumes/ice/frozen009-pictures-2019/Pictures/Travel:/mnt/media/frozen009-pictures-2019/travel:ro
      - /Volumes/ice/frozen009-pictures-2019/Pictures/Life:/mnt/media/frozen009-pictures-2019/life:ro

    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. mounted large volumes over SMB on a Mac
2. added folders on those volumes to docker-compose.yml (e.g., "/Volumes/ice/folder")
3. added those paths to an external library
4. scanned new files on those library

Additional information

No response

Originally created by @ehcheng on GitHub (Apr 7, 2024). ### The bug When scanning for new files on large external libraries, I am receiving errors like this: immich_microservices | [Nest] 7 - 04/07/2024, 4:57:29 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/cold/Lytro/2011/20110520_evt2_dongguan' immich_microservices | [Nest] 7 - 04/07/2024, 4:57:29 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/cold/Lytro/2011/20110520_evt2_dongguan' immich_microservices | [Nest] 7 - 04/07/2024, 7:11:04 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/life/2011/20110305 Livia's concert, dinner' immich_microservices | [Nest] 7 - 04/07/2024, 7:11:04 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/life/2011/20110305 Livia's concert, dinner' immich_microservices | [Nest] 7 - 04/07/2024, 7:11:38 AM ERROR [JobService] Unable to run job handler (library/library-refresh): Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/travel/Japan, Ogasawara 200910' immich_microservices | [Nest] 7 - 04/07/2024, 7:11:38 AM ERROR [JobService] Error: EMFILE: too many open files, scandir '/mnt/media/frozen009-pictures-2019/travel/Japan, Ogasawara 200910' There is 1 library task that spins up, and after it throws the error the task goes away. One external library with 65K assets imported fine, as did another with 28K. but folders with more seem to be failing. I don't know the cut off, but some of the folders I'm importing have 100K+ assets. ### The OS that Immich Server is running on OS X Sonoma 14.4.1 ### Version of Immich Server v1.101.0 ### Version of Immich Mobile App web ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'immich'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - /Volumes/media/Pictures/travel_wildlife:/mnt/media/travel:ro - /Volumes/water/pictures/Life:/mnt/media/life_current:ro - /Volumes/water/pictures/Travel:/mnt/media/travel_current:ro - /Volumes/ice/frozen003-family/videos:/mnt/media/frozen003-family-videos:ro - /Volumes/ice/frozen003-family/scans:/mnt/media/frozen003-family-scans:ro - /Volumes/ice/cold/Pictures:/mnt/media/cold:ro - /Volumes/ice/frozen009-pictures-2019/Pictures/Travel:/mnt/media/frozen009-pictures-2019/travel:ro - /Volumes/ice/frozen009-pictures-2019/Pictures/Life:/mnt/media/frozen009-pictures-2019/life:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding command: ['start.sh', 'microservices'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - /Volumes/media/Pictures/travel_wildlife:/mnt/media/travel:ro - /Volumes/water/pictures/Life:/mnt/media/life_current:ro - /Volumes/water/pictures/Travel:/mnt/media/travel_current:ro - /Volumes/ice/frozen003-family/videos:/mnt/media/frozen003-family-videos:ro - /Volumes/ice/frozen003-family/scans:/mnt/media/frozen003-family-scans:ro - /Volumes/ice/cold/Pictures:/mnt/media/cold:ro - /Volumes/ice/frozen009-pictures-2019/Pictures/Travel:/mnt/media/frozen009-pictures-2019/travel:ro - /Volumes/ice/frozen009-pictures-2019/Pictures/Life:/mnt/media/frozen009-pictures-2019/life:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5 restart: always database: container_name: immich_postgres image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. mounted large volumes over SMB on a Mac 2. added folders on those volumes to docker-compose.yml (e.g., "/Volumes/ice/folder") 3. added those paths to an external library 4. scanned new files on those library ``` ### Additional information _No response_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2775
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?