immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 26 Packages Projects Releases 10 Wiki Activity

[BUG] Server crashes after, opning ../admin/repair #2084

New Issue
Closed
opened 2026-02-05 05:02:42 +03:00 by OVERLORD · 5 comments
OVERLORD commented 2026-02-05 05:02:42 +03:00
Owner
Copy Link

Originally created by @Torstein-Eide on GitHub (Jan 30, 2024).

The bug

The server crashes after admins try to open ../admin/repair.

Docker log:

<--- Last few GCs --->
[7:0x5c862320000]    16399 ms: Mark-Compact 250.8 (258.6) -> 249.9 (258.9) MB, 104.03 / 0.00 ms  (average mu = 0.308, current mu = 0.182) allocation failure; scavenge might not succeed
[7:0x5c862320000]    16589 ms: Mark-Compact 251.0 (258.9) -> 250.1 (259.1) MB, 184.65 / 0.00 ms  (average mu = 0.156, current mu = 0.028) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
 1: 0xca5430 node::Abort() [immich_server]
 2: 0xb7807d  [immich_server]
 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
 5: 0x10dc0e5  [immich_server]
 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server]
 7: 0x10ca081 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]
 8: 0x10cb215 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]
 9: 0x10a8866 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_server]
10: 0x15035f6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_server]
11: 0x193cef6  [immich_server]
[Nest] 7  - 01/30/2024, 3:06:06 PM     LOG [SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 01/30/2024, 3:06:07 PM     LOG [AppService] Feature Flags: {
  "clipEncode": true,
  "facialRecognition": true,
  "map": true,
  "reverseGeocoding": true,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": false,
  "oauthAutoLaunch": false,
  "passwordLogin": true,
  "configFile": false
}

The OS that Immich Server is running on

Docker, Ubuntu 20.04

Version of Immich Server

1.9.3

Version of Immich Mobile App

1.9.3

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - $USER1_path:/user1:ro
      - $USER2_path:/user2:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        reservations:
          memory: 2G


  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.yml
      service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - $USER1_path:/user1:ro
      - $USER2_path:/user2:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        reservations:
          memory: 2G

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/volum1/@docker/immich/upload_files

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD="Some random password"

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. open ../admin/repair
2. get error popup `Request failed with status code 503 - 503 - 
undefined`
3. see server is restarting.
...

Additional information

No response

Originally created by @Torstein-Eide on GitHub (Jan 30, 2024). ### The bug The server crashes after admins try to open `../admin/repair`. ## Docker log: ````js <--- Last few GCs ---> [7:0x5c862320000] 16399 ms: Mark-Compact 250.8 (258.6) -> 249.9 (258.9) MB, 104.03 / 0.00 ms (average mu = 0.308, current mu = 0.182) allocation failure; scavenge might not succeed [7:0x5c862320000] 16589 ms: Mark-Compact 251.0 (258.9) -> 250.1 (259.1) MB, 184.65 / 0.00 ms (average mu = 0.156, current mu = 0.028) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace ----- 1: 0xca5430 node::Abort() [immich_server] 2: 0xb7807d [immich_server] 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server] 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server] 5: 0x10dc0e5 [immich_server] 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server] 7: 0x10ca081 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server] 8: 0x10cb215 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server] 9: 0x10a8866 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_server] 10: 0x15035f6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_server] 11: 0x193cef6 [immich_server] [Nest] 7 - 01/30/2024, 3:06:06 PM LOG [SystemConfigService] LogLevel=log (set via system config) [Nest] 7 - 01/30/2024, 3:06:07 PM LOG [AppService] Feature Flags: { "clipEncode": true, "facialRecognition": true, "map": true, "reverseGeocoding": true, "sidecar": true, "search": true, "trash": true, "oauth": false, "oauthAutoLaunch": false, "passwordLogin": true, "configFile": false } ```` ### The OS that Immich Server is running on Docker, Ubuntu 20.04 ### Version of Immich Server 1.9.3 ### Version of Immich Mobile App 1.9.3 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - $USER1_path:/user1:ro - $USER2_path:/user2:ro - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always deploy: resources: reservations: memory: 2G immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} extends: file: hwaccel.yml service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - $USER1_path:/user1:ro - $USER2_path:/user2:ro - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always deploy: resources: reservations: memory: 2G immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always deploy: resources: reservations: cpus: '0.001' memory: 50M redis: container_name: immich_redis image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a restart: always deploy: resources: reservations: cpus: '0.001' memory: 50M database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always deploy: resources: reservations: cpus: '0.001' memory: 50M volumes: pgdata: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/volum1/@docker/immich/upload_files # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD="Some random password" # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. open ../admin/repair 2. get error popup `Request failed with status code 503 - 503 - undefined` 3. see server is restarting. ... ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 05:02:53 +03:00
Author
Owner
Copy Link

@aviv926 commented on GitHub (Jan 31, 2024):

Have you tried allocating more RAM?
It seems that the system is unable to render the page as required because there is not enough RAM

JavaScript heap out of memory

@aviv926 commented on GitHub (Jan 31, 2024): Have you tried allocating more RAM? It seems that the system is unable to render the page as required because there is not enough RAM `JavaScript heap out of memory`
OVERLORD commented 2026-02-05 05:02:58 +03:00
Author
Owner
Copy Link

@Torstein-Eide commented on GitHub (Jan 31, 2024):

It was running unlimited.

currenly it looks like this, with 100MB in use:

$docker compose stats
CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O   PIDS
af665bd080b4   immich_microservices      2.10%     250.7MiB / 2GiB     12.24%    540MB / 2.57GB    0B / 0B     24
ecfa9c5d2b9f   immich_machine_learning   50.54%    200.8MiB / 2GiB     9.80%     8.43GB / 138MB    0B / 0B     7
da70fd4cb0b6   immich_server             0.17%     117.7MiB / 2GiB     5.75%     54.9MB / 240MB    0B / 0B     12
9725ad84f7cf   immich_postgres           0.01%     424MiB / 2GiB       20.71%    3.15GB / 2.05GB   0B / 0B     20
75f96f41e1ad   immich_redis              0.28%     45MiB / 2GiB        2.20%     584MB / 188MB     0B / 0B     5
@Torstein-Eide commented on GitHub (Jan 31, 2024): It was running unlimited. currenly it looks like this, with 100MB in use: ````java $docker compose stats CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS af665bd080b4 immich_microservices 2.10% 250.7MiB / 2GiB 12.24% 540MB / 2.57GB 0B / 0B 24 ecfa9c5d2b9f immich_machine_learning 50.54% 200.8MiB / 2GiB 9.80% 8.43GB / 138MB 0B / 0B 7 da70fd4cb0b6 immich_server 0.17% 117.7MiB / 2GiB 5.75% 54.9MB / 240MB 0B / 0B 12 9725ad84f7cf immich_postgres 0.01% 424MiB / 2GiB 20.71% 3.15GB / 2.05GB 0B / 0B 20 75f96f41e1ad immich_redis 0.28% 45MiB / 2GiB 2.20% 584MB / 188MB 0B / 0B 5 ````
OVERLORD commented 2026-02-05 05:03:06 +03:00
Author
Owner
Copy Link

@aviv926 commented on GitHub (Jan 31, 2024):

It was running unlimited.

currently it looks like this, with 100MB in use:

$docker compose stats
CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O   PIDS
af665bd080b4   immich_microservices      2.10%     250.7MiB / 2GiB     12.24%    540MB / 2.57GB    0B / 0B     24
ecfa9c5d2b9f   immich_machine_learning   50.54%    200.8MiB / 2GiB     9.80%     8.43GB / 138MB    0B / 0B     7
da70fd4cb0b6   immich_server             0.17%     117.7MiB / 2GiB     5.75%     54.9MB / 240MB    0B / 0B     12
9725ad84f7cf   immich_postgres           0.01%     424MiB / 2GiB       20.71%    3.15GB / 2.05GB   0B / 0B     20
75f96f41e1ad   immich_redis              0.28%     45MiB / 2GiB        2.20%     584MB / 188MB     0B / 0B     5

Try to check the compose stats when you try enter to the repair page

@aviv926 commented on GitHub (Jan 31, 2024): > It was running unlimited. > > currently it looks like this, with 100MB in use: > ````java > $docker compose stats > CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS > af665bd080b4 immich_microservices 2.10% 250.7MiB / 2GiB 12.24% 540MB / 2.57GB 0B / 0B 24 > ecfa9c5d2b9f immich_machine_learning 50.54% 200.8MiB / 2GiB 9.80% 8.43GB / 138MB 0B / 0B 7 > da70fd4cb0b6 immich_server 0.17% 117.7MiB / 2GiB 5.75% 54.9MB / 240MB 0B / 0B 12 > 9725ad84f7cf immich_postgres 0.01% 424MiB / 2GiB 20.71% 3.15GB / 2.05GB 0B / 0B 20 > 75f96f41e1ad immich_redis 0.28% 45MiB / 2GiB 2.20% 584MB / 188MB 0B / 0B 5 > > ```` Try to check the compose stats when you try enter to the repair page
OVERLORD commented 2026-02-05 05:03:12 +03:00
Author
Owner
Copy Link

@diyoyo commented on GitHub (Feb 11, 2024):

I have a Javascript Heap OoM as well, but for a different problem: immich won't update the new Library files as the file count is already very big.

Currently, I found this Sof issue: https://stackoverflow.com/questions/38558989/node-js-heap-out-of-memory
where they suggest the use of the following:
NODE_OPTIONS="--max-old-space-size=4096"

I'm not sure I can just paste it right away in the docker-compose file, but I'll try. If you have better suggestion to use that env variable, let me know.
Of course, in your case, having 2G as your upper limit may be the problem...

Also, I'm wondering if we could play with 'shm_size'. It was in my docker-compose.yml file when I had tried photoview, but I'm not sure whether it's related to the tech stack choices or if this applies to any docker container that needs more memory...

@diyoyo commented on GitHub (Feb 11, 2024): I have a Javascript Heap OoM as well, but for a different problem: immich won't update the new Library files as the file count is already very big. Currently, I found this Sof issue: https://stackoverflow.com/questions/38558989/node-js-heap-out-of-memory where they suggest the use of the following: ```NODE_OPTIONS="--max-old-space-size=4096"``` I'm not sure I can just paste it right away in the `docker-compose` file, but I'll try. If you have better suggestion to use that env variable, let me know. Of course, in your case, having 2G as your upper limit may be the problem... Also, I'm wondering if we could play with 'shm_size'. It was in my docker-compose.yml file when I had tried `photoview`, but I'm not sure whether it's related to the tech stack choices or if this applies to any docker container that needs more memory...
OVERLORD commented 2026-02-05 05:03:20 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Apr 27, 2024):

#4510

@alextran1502 commented on GitHub (Apr 27, 2024): #4510
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2084
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?