immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 26 Packages Projects Releases 10 Wiki Activity

Removal of users externalPath restriction for external libraries causes immich to scan the entire server #2613

New Issue
Closed
opened 2026-02-05 06:18:58 +03:00 by OVERLORD · 17 comments
OVERLORD commented 2026-02-05 06:18:58 +03:00
Owner
Copy Link

Originally created by @dorianim on GitHub (Mar 17, 2024).

EDIT the issue was caused by Immich scanning the entire server for assets, because an external library pointing to root (/) existed and after the externalPath restriction was removed in v1.98.0, there was noting stopping the indexing of the whole server. See https://github.com/immich-app/immich/issues/8020#issuecomment-2010975608

The bug

After upgrading from v1.98.0 to v1.98.2, I am seeing a very high CPU utilization from immich related tasks and the webinterface is extremely slow and largely unusable:
image

Before the upgrade, everything was working fine. I didn't change any configuration nor triggered any jobs. Just did a docker compose pull && docker compose up -d.

Did I miss something obvious? What can I do to resolve this? I have left the server running for several days in a row and also tried restarting it, but that didn't change anything.
What can I do to troubleshoot the issue further?

The OS that Immich Server is running on

Ubuntu 20.04 LTS

Version of Immich Server

v1.98.2

Version of Immich Mobile App

v1.98.2

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always
    labels:
      traefik.enable: true
      traefik.http.routers.immich.rule: "Host(`photos.dorian.im`) || Host(`photos.itsblue.de`)"
      traefik.http.routers.immich.entrypoints: webs
    
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - ./model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - ./pgdata:/var/lib/postgresql/data
    restart: always

Your .env content

IMMICH_VERSION=release

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################
    
REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Config
###################################################################################
UPLOAD_LOCATION=/mnt/data/immich
    
###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=xxx

###################################################################################
# JWT SECRET
###################################################################################
JWT_SECRET=xxx

Reproduction steps

1. start immich

Additional information

I have started using immmich very early on and upgraded version by version ever since.

Originally created by @dorianim on GitHub (Mar 17, 2024). > **EDIT** the issue was caused by Immich scanning the entire server for assets, because an external library pointing to root (`/`) existed and after the `externalPath` restriction was removed in v1.98.0, there was noting stopping the indexing of the whole server. See https://github.com/immich-app/immich/issues/8020#issuecomment-2010975608 ### The bug After upgrading from v1.98.0 to v1.98.2, I am seeing a very high CPU utilization from immich related tasks and the webinterface is extremely slow and largely unusable: ![image](https://github.com/immich-app/immich/assets/30153207/6cbbd06c-7835-41b0-9751-444c840fc945) Before the upgrade, everything was working fine. I didn't change any configuration nor triggered any jobs. Just did a `docker compose pull && docker compose up -d`. Did I miss something obvious? What can I do to resolve this? I have left the server running for several days in a row and also tried restarting it, but that didn't change anything. What can I do to troubleshoot the issue further? ### The OS that Immich Server is running on Ubuntu 20.04 LTS ### Version of Immich Server v1.98.2 ### Version of Immich Mobile App v1.98.2 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'immich'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 environment: - NODE_ENV=production depends_on: - redis - database restart: always labels: traefik.enable: true traefik.http.routers.immich.rule: "Host(`photos.dorian.im`) || Host(`photos.itsblue.de`)" traefik.http.routers.immich.entrypoints: webs immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'microservices'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env environment: - NODE_ENV=production depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - ./model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5 restart: always database: container_name: immich_postgres image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - ./pgdata:/var/lib/postgresql/data restart: always ``` ### Your .env content ```Shell IMMICH_VERSION=release ################################################################################### # Database ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_PASSWORD=xxx DB_DATABASE_NAME=immich # Optional Database settings: # DB_PORT=5432 ################################################################################### # Redis ################################################################################### REDIS_HOSTNAME=immich_redis # Optional Redis settings: # REDIS_PORT=6379 # REDIS_DBINDEX=0 # REDIS_PASSWORD= # REDIS_SOCKET= ################################################################################### # Upload File Config ################################################################################### UPLOAD_LOCATION=/mnt/data/immich ################################################################################### # Typesense ################################################################################### TYPESENSE_API_KEY=xxx ################################################################################### # JWT SECRET ################################################################################### JWT_SECRET=xxx ``` ### Reproduction steps ```bash 1. start immich ``` ### Additional information I have started using immmich very early on and upgraded version by version ever since.
OVERLORD commented 2026-02-05 06:19:03 +03:00
Author
Owner
Copy Link

@virtualized-human commented on GitHub (Mar 17, 2024):

its basicly just doing some jobs, can you show the container logs?

@virtualized-human commented on GitHub (Mar 17, 2024): its basicly just doing some jobs, can you show the container logs?
OVERLORD commented 2026-02-05 06:19:05 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 17, 2024):

Hmm, after the latest restart, the high CPU usage is gone.

However, the server is still misbehaving. When I restart it, it briefly seems to work, the web interface loads and simple endpoints like /_app/version.json return valid responses.
The timeline however is completely empty. The reason seems to be that the /api/asset/time-bucket endpoint never sends a response:
image

And after some time, even the simple endpoints don't return any response anymore and the web interface just times out.

The Logs seem to show nothing out of the ordinary:
image

@dorianim commented on GitHub (Mar 17, 2024): Hmm, after the latest restart, the high CPU usage is gone. However, the server is still misbehaving. When I restart it, it briefly seems to work, the web interface loads and simple endpoints like `/_app/version.json` return valid responses. The timeline however is completely empty. The reason seems to be that the `/api/asset/time-bucket` endpoint never sends a response: ![image](https://github.com/immich-app/immich/assets/30153207/d29d6bcd-dac8-45b6-b4a0-5fada825eef7) And after some time, even the simple endpoints don't return any response anymore and the web interface just times out. The Logs seem to show nothing out of the ordinary: ![image](https://github.com/immich-app/immich/assets/30153207/82ed3141-3bfc-4c7b-8cac-4d48d55012a8)
OVERLORD commented 2026-02-05 06:19:10 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 17, 2024):

It looks like you have one months with more than 5000, 6000 assets in that month? Is it the correct observation

@alextran1502 commented on GitHub (Mar 17, 2024): It looks like you have one months with more than 5000, 6000 assets in that month? Is it the correct observation
OVERLORD commented 2026-02-05 06:19:14 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 17, 2024):

No, I don't think so. I hardly have a month with more than 1000 assets.

@dorianim commented on GitHub (Mar 17, 2024): No, I don't think so. I hardly have a month with more than 1000 assets.
OVERLORD commented 2026-02-05 06:19:20 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 17, 2024):

Can you try on different browser?

@alextran1502 commented on GitHub (Mar 17, 2024): Can you try on different browser?
OVERLORD commented 2026-02-05 06:19:23 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 17, 2024):

I don't think, it's a browser problem. The endpoints also don't respond when I try with curl.

@dorianim commented on GitHub (Mar 17, 2024): I don't think, it's a browser problem. The endpoints also don't respond when I try with curl.
OVERLORD commented 2026-02-05 06:19:26 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 17, 2024):

Can you try restart the server and try again?

On Sun, Mar 17, 2024 at 11:40 AM Dorian Zedler @.***>
wrote:

I don't think, it's a browser problem. The endpoints also don't respond
when I try with curl.

—
Reply to this email directly, view it on GitHub
https://github.com/immich-app/immich/issues/8020#issuecomment-2002533617,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGONL7X334EJN2PNVFEJTYDYYXBO3AVCNFSM6AAAAABE2FTXVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGUZTGNRRG4
.
You are receiving this because you commented.Message ID:
@.***>

@alextran1502 commented on GitHub (Mar 17, 2024): Can you try restart the server and try again? On Sun, Mar 17, 2024 at 11:40 AM Dorian Zedler ***@***.***> wrote: > I don't think, it's a browser problem. The endpoints also don't respond > when I try with curl. > > — > Reply to this email directly, view it on GitHub > <https://github.com/immich-app/immich/issues/8020#issuecomment-2002533617>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AGONL7X334EJN2PNVFEJTYDYYXBO3AVCNFSM6AAAAABE2FTXVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGUZTGNRRG4> > . > You are receiving this because you commented.Message ID: > ***@***.***> >
OVERLORD commented 2026-02-05 06:19:30 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 18, 2024):

Tried several times. It always results in this situation :(
Can I safely downgrade to a previous version to isolate the issue?

@dorianim commented on GitHub (Mar 18, 2024): Tried several times. It always results in this situation :( Can I safely downgrade to a previous version to isolate the issue?
OVERLORD commented 2026-02-05 06:19:32 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Mar 18, 2024):

Might be related to #7909

@mertalev commented on GitHub (Mar 18, 2024): Might be related to #7909
OVERLORD commented 2026-02-05 06:19:34 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 18, 2024):

Yes, I think, I had the rescan issue as well. That's probaply what caused the high cpu usage. But does it also explain why my instance bricks itself?

@dorianim commented on GitHub (Mar 18, 2024): Yes, I think, I had the rescan issue as well. That's probaply what caused the high cpu usage. But does it also explain why my instance bricks itself?
OVERLORD commented 2026-02-05 06:19:36 +03:00
Author
Owner
Copy Link

@wangweitung commented on GitHub (Mar 20, 2024):

I also have this issue

图片

@wangweitung commented on GitHub (Mar 20, 2024): I also have this issue ![图片](https://github.com/immich-app/immich/assets/8080125/3cefa2a5-b8fe-4c0b-97aa-560ad3345134)
OVERLORD commented 2026-02-05 06:19:40 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 21, 2024):

I did some more digging and I think I found the issue. For some reason there are hundreds of thousands of assets in my database all created on 13th just seconds apart...
image
image
image

This seems really strange to me, I have no idea where those assets are coming from... But it's for sure the reason for all the troubles.

Edit: The library does not contain a single asset from that day...
image

Edit 2:
Another curious thing I've found:
The Original path of the strange assets! originalPath of normal assets:
image

originalPath of strange assets:
image

So it looks like it somehow consumed thumbnails as uploads

@dorianim commented on GitHub (Mar 21, 2024): I did some more digging and I think I found the issue. For some reason there are hundreds of thousands of assets in my database all created on 13th just seconds apart... ![image](https://github.com/immich-app/immich/assets/30153207/5acca159-016d-4713-8027-46ad2c1b2b96) ![image](https://github.com/immich-app/immich/assets/30153207/e6dd06e1-c648-4240-8a5e-5cfaf10260c5) ![image](https://github.com/immich-app/immich/assets/30153207/32335e6c-7a40-4d37-b19b-decd77ad2287) This seems really strange to me, I have no idea where those assets are coming from... But it's for sure the reason for all the troubles. Edit: The library does not contain a single asset from that day... ![image](https://github.com/immich-app/immich/assets/30153207/de4b5f7f-87f2-4f85-ab47-6ea96b9dab6e) Edit 2: Another curious thing I've found: The Original path of the strange assets! originalPath of normal assets: ![image](https://github.com/immich-app/immich/assets/30153207/cc908525-691c-4301-b601-dbda96a0e32d) originalPath of strange assets: ![image](https://github.com/immich-app/immich/assets/30153207/c71fbfda-2233-4243-a059-f0a4c612e95f) So it looks like it somehow consumed thumbnails as uploads
OVERLORD commented 2026-02-05 06:19:42 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 21, 2024):

Ha! Found it :)
For some reason there is an external library which points to the root directory for my user. So it has basically scanned each asset on the entire server...
image

Looks like I created it a long time ago and forgot about it and in a recent release it started to import stuff:
image

@dorianim commented on GitHub (Mar 21, 2024): Ha! Found it :) For some reason there is an external library which points to the root directory for my user. So it has basically scanned each asset on the entire server... ![image](https://github.com/immich-app/immich/assets/30153207/96617687-8b7b-4b36-a84c-205c803360a0) Looks like I created it a long time ago and forgot about it and in a recent release it started to import stuff: ![image](https://github.com/immich-app/immich/assets/30153207/0b054431-72c5-438e-814c-55304149d2c6)
OVERLORD commented 2026-02-05 06:19:45 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 21, 2024):

I have now deleted the external library and the asset count is decreasing rapidly. I hope, it will be cleaned up by tomorrow :)
I'd suggest adding some kind of check to make sure the user is not trying to import the root directory of the server 😅

@dorianim commented on GitHub (Mar 21, 2024): I have now deleted the external library and the asset count is decreasing rapidly. I hope, it will be cleaned up by tomorrow :) I'd suggest adding some kind of check to make sure the user is not trying to import the root directory of the server :sweat_smile:
OVERLORD commented 2026-02-05 06:19:46 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Mar 21, 2024):

We changed it to ignore anything in the immich media folder, so it at least won't explode when that happens now haha

@mertalev commented on GitHub (Mar 21, 2024): We changed it to ignore anything in the immich media folder, so it at least won't explode when that happens now haha
OVERLORD commented 2026-02-05 06:19:48 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Mar 21, 2024):

Oh wait, I don't think the check we added would catch that. It checks if the import path is in the media folder, but doesn't check if the path contains the media folder.

@mertalev commented on GitHub (Mar 21, 2024): Oh wait, I don't think the check we added would catch that. It checks if the import path is in the media folder, but doesn't check if the path contains the media folder.
OVERLORD commented 2026-02-05 06:19:51 +03:00
Author
Owner
Copy Link

@dorianim commented on GitHub (Mar 21, 2024):

I think, the issue was caused by the removal of the external path from the user:
https://github.com/immich-app/immich/blob/v1.98.0/server/src/infra/migrations/1708425975121-RemoveExternalPath.ts

This was used to make sure something like this doesn't happen, if I recall correctly.

Everything is back to normal now btw :)

@dorianim commented on GitHub (Mar 21, 2024): I think, the issue was caused by the removal of the external path from the user: https://github.com/immich-app/immich/blob/v1.98.0/server/src/infra/migrations/1708425975121-RemoveExternalPath.ts This was used to make sure something like this doesn't happen, if I recall correctly. Everything is back to normal now btw :)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2613
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?