[BUG] JavaScript heap out of memory #1474

New Issue

OVERLORD · 2026-02-05T01:59:57+03:00

OVERLORD commented

2026-02-05 01:59:57 +03:00

Originally created by @dtirer on GitHub (Oct 18, 2023).

The bug

Attaching to immich_proxy, immich_server, immich_microservices, immich_postgres, immich_redis, immich_machine_learning, immich_web, immich_typesense
immich_machine_learning    | [10/18/23 16:16:32] INFO     Starting gunicorn 21.2.0
immich_machine_learning    | [10/18/23 16:16:32] INFO     Listening at: http://0.0.0.0:3003 (9)
immich_machine_learning    | [10/18/23 16:16:32] INFO     Using worker: uvicorn.workers.UvicornWorker
immich_machine_learning    | [10/18/23 16:16:32] INFO     Booting worker with pid: 10
immich_postgres            |
immich_postgres            | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres            |
immich_postgres            | 2023-10-18 16:16:31.585 UTC [1] LOG:  starting PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
immich_postgres            | 2023-10-18 16:16:31.585 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
immich_postgres            | 2023-10-18 16:16:31.585 UTC [1] LOG:  listening on IPv6 address "::", port 5432
immich_postgres            | 2023-10-18 16:16:31.609 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
immich_postgres            | 2023-10-18 16:16:31.651 UTC [27] LOG:  database system was shut down at 2023-10-18 16:15:39 UTC
immich_postgres            | 2023-10-18 16:16:31.695 UTC [1] LOG:  database system is ready to accept connections
immich_web                 | Listening on 0.0.0.0:3000
redis_1                    | 1:C 18 Oct 2023 16:16:31.577 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1                    | 1:C 18 Oct 2023 16:16:31.577 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1                    | 1:C 18 Oct 2023 16:16:31.577 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1                    | 1:M 18 Oct 2023 16:16:31.577 * monotonic clock: POSIX clock_gettime
redis_1                    | 1:M 18 Oct 2023 16:16:31.577 * Running mode=standalone, port=6379.
redis_1                    | 1:M 18 Oct 2023 16:16:31.577 # Server initialized
redis_1                    | 1:M 18 Oct 2023 16:16:31.577 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1                    | 1:M 18 Oct 2023 16:16:31.577 * Ready to accept connections
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +131ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] DomainModule dependencies initialized +7ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [NestFactory] Starting Nest application...
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +39ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +5ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +123ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] AppModule dependencies initialized +4ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [InstanceLoader] DomainModule dependencies initialized +5ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AssetController {/asset}: +85ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/map-marker, GET} route +2ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/memory-lane, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/random, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/download/info, POST} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/download/archive, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/download/:id, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/statistics, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/time-buckets, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/time-bucket, GET} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/jobs, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/restore, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/trash/empty, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/trash/restore, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/:id, PUT} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AssetController {/asset}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/upload, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/import, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/file/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/thumbnail/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/curated-objects, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/curated-locations, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/search-terms, GET} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/search, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/:deviceId, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/assetById/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/check, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/exist, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/asset/bulk-upload-check, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AppController {/}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/refresh-config, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AlbumController {/album}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/count, GET} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id, PATCH} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] APIKeyController {/api-key}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/api-key, POST} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/api-key, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/api-key/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/api-key/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/api-key/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AuditController {/audit}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/audit/deletes, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/audit/file-report, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/audit/file-report/checksum, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/audit/file-report/fix, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] AuthController {/auth}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/login, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/admin-sign-up, POST} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/devices, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/devices, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/devices/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/validateToken, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/change-password, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/auth/logout, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] JobController {/jobs}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/jobs, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/jobs/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] LibraryController {/library}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library, POST} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id/statistics, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id/scan, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/library/:id/removeOffline, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] OAuthController {/oauth}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/mobile-redirect, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/config, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/authorize, POST} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/callback, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/link, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/oauth/unlink, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] PartnerController {/partner}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/partner, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/partner/:id, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/partner/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] SearchController {/search}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/search/person, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/search, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/search/explore, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] ServerInfoController {/server-info}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info, GET} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/ping, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/version, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/features, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/config, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/stats, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/server-info/media-types, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] SharedLinkController {/shared-link}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/me, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, PATCH} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/:id, DELETE} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/:id/assets, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/shared-link/:id/assets, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] SystemConfigController {/system-config}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/system-config, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/system-config/defaults, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/system-config, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/system-config/storage-template-options, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] TagController {/tag}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id, PATCH} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id/assets, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id/assets, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/tag/:id/assets, DELETE} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] UserController {/user}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/info/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/me, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/count, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/:id, DELETE} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/:id/restore, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/profile-image, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/user/profile-image/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RoutesResolver] PersonController {/person}: +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person/:id, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person/:id, PUT} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person/:id/thumbnail, GET} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person/:id/assets, GET} route +1ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [RouterExplorer] Mapped {/person/:id/merge, POST} route +0ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:33 PM     LOG [SearchService] Running bootstrap
immich_server              | Request #1697645793895: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server              | Request #1697645793895: Sleeping for 4s and then retrying request...
immich_server              | Request #1697645793895: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server              | Request #1697645793895: Sleeping for 4s and then retrying request...
immich_machine_learning    | [10/18/23 16:16:40] INFO     Created in-memory cache with unloading disabled.
immich_machine_learning    | [10/18/23 16:16:40] INFO     Initialized request thread pool with 8 threads.
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [AppService] Feature Flags: {
immich_server              |   "clipEncode": true,
immich_server              |   "facialRecognition": true,
immich_server              |   "tagImage": true,
immich_server              |   "map": true,
immich_server              |   "reverseGeocoding": true,
immich_server              |   "sidecar": true,
immich_server              |   "search": true,
immich_server              |   "trash": true,
immich_server              |   "oauth": false,
immich_server              |   "oauthAutoLaunch": false,
immich_server              |   "passwordLogin": true,
immich_server              |   "configFile": false
immich_server              | }
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [NestApplication] Nest application successfully started +8ms
immich_server              | [Nest] 7  - 10/18/2023, 4:16:41 PM     LOG [ImmichServer] Immich Server is listening on http://[::1]:3001 [v1.82.0] [PRODUCTION]
immich_server              | [Nest] 7  - 10/18/2023, 4:16:47 PM     LOG [CommunicationRepository] New websocket connection: e-0AA1FduEeER_pXAAAB
immich_microservices       |
immich_microservices       | <--- Last few GCs --->
immich_microservices       |
immich_microservices       | [7:0x251ac310000]   197308 ms: Mark-Compact 4038.9 (4131.0) -> 4021.1 (4130.5) MB, 4766.49 / 0.00 ms  (average mu = 0.118, current mu = 0.054) allocation failure; scavenge might not succeed
immich_microservices       | [7:0x251ac310000]   202309 ms: Mark-Compact 4038.9 (4131.0) -> 4023.2 (4132.2) MB, 4913.29 / 0.00 ms  (average mu = 0.067, current mu = 0.018) allocation failure; scavenge might not succeed
immich_microservices       |
immich_microservices       |
immich_microservices       | <--- JS stacktrace --->
immich_microservices       |
immich_microservices       | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices       |  1: 0xc99960 node::Abort() [immich_microservices]
immich_microservices       |  2: 0xb6ffcb  [immich_microservices]
immich_microservices       |  3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
immich_microservices       |  4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
immich_microservices       |  5: 0x10d06a5  [immich_microservices]
immich_microservices       |  6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
immich_microservices       |  7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
immich_microservices       |  8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
immich_microservices       |  9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
immich_microservices       | 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
immich_microservices       | 13: 0x7f8c02399ef6
immich_microservices       | Aborted (core dumped)
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +25ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +121ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] DomainModule dependencies initialized +7ms
immich_microservices       | [Nest] 7  - 10/18/2023, 4:20:05 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms

The OS that Immich Server is running on

Debian Bookworm

Version of Immich Server

v1.82.0

Version of Immich Mobile App

v1.82.0

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /home/me/photos/test_photos:/mnt/media/test_photos:ro
    env_file:
      - .env
    networks:
      - inside
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    networks:
      - inside
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /home/me/photos/test_photos:/mnt/media/test_photos:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    networks:
      - inside
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ./volumes/model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    networks:
      - inside
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    networks:
      - inside
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    networks:
      - inside
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - ./volumes/pgdata:/var/lib/postgresql/data
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - ./volumes/tsdata:/data
    restart: always
    networks:
      - inside

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    networks:
      - outside
      - inside
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

networks:
  outside:
    name: immich_outside
  inside:
    internal: true



### Your .env content

```Shell
###################################################################################
# Database
###################################################################################

DB_HOSTNAME=redcated
DB_USERNAME=redcated
DB_PASSWORD=redcated
DB_DATABASE_NAME=redcated

# Optional Database settings:
# DB_PORT=redcated

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=redcated
# REDIS_DBINDEX=0
# REDIS_PASSWORD=redcated
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/home/me/immich_photos

###################################################################################
# JWT SECRET
#
# This JWT_SECRET is used to sign the authentication keys for user login
# You should set it to a long randomly generated value
# You can use this command to generate one: openssl rand -base64 128
###################################################################################

JWT_SECRET=redcated
###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

DISABLE_REVERSE_GEOCODING=true
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

TYPESENSE_API_KEY=redcated

Reproduction steps

1. `docker-compose up -d` to start the service
2. `docker-compose logs -f` to follow the logs

`immich-microservices` begin executing processes that hog most of my CPU

Eventually, the posted memory issue occurs and the server application seems to restart

Additional information

No response

Originally created by @dtirer on GitHub (Oct 18, 2023). ### The bug ``` Attaching to immich_proxy, immich_server, immich_microservices, immich_postgres, immich_redis, immich_machine_learning, immich_web, immich_typesense immich_machine_learning | [10/18/23 16:16:32] INFO Starting gunicorn 21.2.0 immich_machine_learning | [10/18/23 16:16:32] INFO Listening at: http://0.0.0.0:3003 (9) immich_machine_learning | [10/18/23 16:16:32] INFO Using worker: uvicorn.workers.UvicornWorker immich_machine_learning | [10/18/23 16:16:32] INFO Booting worker with pid: 10 immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2023-10-18 16:16:31.585 UTC [1] LOG: starting PostgreSQL 14.9 (Debian 14.9-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit immich_postgres | 2023-10-18 16:16:31.585 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2023-10-18 16:16:31.585 UTC [1] LOG: listening on IPv6 address "::", port 5432 immich_postgres | 2023-10-18 16:16:31.609 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2023-10-18 16:16:31.651 UTC [27] LOG: database system was shut down at 2023-10-18 16:15:39 UTC immich_postgres | 2023-10-18 16:16:31.695 UTC [1] LOG: database system is ready to accept connections immich_web | Listening on 0.0.0.0:3000 redis_1 | 1:C 18 Oct 2023 16:16:31.577 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo redis_1 | 1:C 18 Oct 2023 16:16:31.577 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started redis_1 | 1:C 18 Oct 2023 16:16:31.577 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf redis_1 | 1:M 18 Oct 2023 16:16:31.577 * monotonic clock: POSIX clock_gettime redis_1 | 1:M 18 Oct 2023 16:16:31.577 * Running mode=standalone, port=6379. redis_1 | 1:M 18 Oct 2023 16:16:31.577 # Server initialized redis_1 | 1:M 18 Oct 2023 16:16:31.577 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. redis_1 | 1:M 18 Oct 2023 16:16:31.577 * Ready to accept connections immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +131ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] DomainModule dependencies initialized +7ms immich_microservices | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [NestFactory] Starting Nest application... immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +39ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] ConfigModule dependencies initialized +5ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +123ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] AppModule dependencies initialized +4ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [InstanceLoader] DomainModule dependencies initialized +5ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AssetController {/asset}: +85ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/map-marker, GET} route +2ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/memory-lane, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/random, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/download/info, POST} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/download/archive, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/download/:id, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/statistics, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/time-buckets, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/time-bucket, GET} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/jobs, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/restore, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/trash/empty, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/trash/restore, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/:id, PUT} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AssetController {/asset}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/upload, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/import, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/file/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/thumbnail/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/curated-objects, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/curated-locations, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/search-terms, GET} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/search, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/:deviceId, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/assetById/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/check, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/exist, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/asset/bulk-upload-check, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AppController {/}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/refresh-config, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AlbumController {/album}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/count, GET} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id, PATCH} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] APIKeyController {/api-key}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/api-key, POST} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/api-key, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/api-key/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/api-key/:id, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/api-key/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AuditController {/audit}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/audit/deletes, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/audit/file-report, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/audit/file-report/checksum, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/audit/file-report/fix, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] AuthController {/auth}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/login, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/admin-sign-up, POST} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/devices, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/devices, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/devices/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/validateToken, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/change-password, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/auth/logout, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] JobController {/jobs}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/jobs, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/jobs/:id, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] LibraryController {/library}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library, POST} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id/statistics, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id/scan, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/library/:id/removeOffline, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] OAuthController {/oauth}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/mobile-redirect, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/config, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/authorize, POST} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/callback, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/link, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/oauth/unlink, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] PartnerController {/partner}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/partner, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/partner/:id, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/partner/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] SearchController {/search}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/search/person, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/search, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/search/explore, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] ServerInfoController {/server-info}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info, GET} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/ping, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/version, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/features, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/config, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/stats, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/server-info/media-types, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] SharedLinkController {/shared-link}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/me, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/:id, PATCH} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/:id, DELETE} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/:id/assets, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/shared-link/:id/assets, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] SystemConfigController {/system-config}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/system-config, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/system-config/defaults, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/system-config, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/system-config/storage-template-options, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] TagController {/tag}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id, PATCH} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, DELETE} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] UserController {/user}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/info/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/me, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/count, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/:id, DELETE} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/:id/restore, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/profile-image, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/user/profile-image/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RoutesResolver] PersonController {/person}: +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person/:id, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person/:id, PUT} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person/:id/thumbnail, GET} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person/:id/assets, GET} route +1ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [RouterExplorer] Mapped {/person/:id/merge, POST} route +0ms immich_server | [Nest] 7 - 10/18/2023, 4:16:33 PM LOG [SearchService] Running bootstrap immich_server | Request #1697645793895: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" immich_server | Request #1697645793895: Sleeping for 4s and then retrying request... immich_server | Request #1697645793895: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" immich_server | Request #1697645793895: Sleeping for 4s and then retrying request... immich_machine_learning | [10/18/23 16:16:40] INFO Created in-memory cache with unloading disabled. immich_machine_learning | [10/18/23 16:16:40] INFO Initialized request thread pool with 8 threads. immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [AppService] Feature Flags: { immich_server | "clipEncode": true, immich_server | "facialRecognition": true, immich_server | "tagImage": true, immich_server | "map": true, immich_server | "reverseGeocoding": true, immich_server | "sidecar": true, immich_server | "search": true, immich_server | "trash": true, immich_server | "oauth": false, immich_server | "oauthAutoLaunch": false, immich_server | "passwordLogin": true, immich_server | "configFile": false immich_server | } immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [NestApplication] Nest application successfully started +8ms immich_server | [Nest] 7 - 10/18/2023, 4:16:41 PM LOG [ImmichServer] Immich Server is listening on http://[::1]:3001 [v1.82.0] [PRODUCTION] immich_server | [Nest] 7 - 10/18/2023, 4:16:47 PM LOG [CommunicationRepository] New websocket connection: e-0AA1FduEeER_pXAAAB immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | [7:0x251ac310000] 197308 ms: Mark-Compact 4038.9 (4131.0) -> 4021.1 (4130.5) MB, 4766.49 / 0.00 ms (average mu = 0.118, current mu = 0.054) allocation failure; scavenge might not succeed immich_microservices | [7:0x251ac310000] 202309 ms: Mark-Compact 4038.9 (4131.0) -> 4023.2 (4132.2) MB, 4913.29 / 0.00 ms (average mu = 0.067, current mu = 0.018) allocation failure; scavenge might not succeed immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory immich_microservices | 1: 0xc99960 node::Abort() [immich_microservices] immich_microservices | 2: 0xb6ffcb [immich_microservices] immich_microservices | 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] immich_microservices | 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] immich_microservices | 5: 0x10d06a5 [immich_microservices] immich_microservices | 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] immich_microservices | 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] immich_microservices | 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] immich_microservices | 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] immich_microservices | 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] immich_microservices | 13: 0x7f8c02399ef6 immich_microservices | Aborted (core dumped) immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +25ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +121ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] DomainModule dependencies initialized +7ms immich_microservices | [Nest] 7 - 10/18/2023, 4:20:05 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms ``` ### The OS that Immich Server is running on Debian Bookworm ### Version of Immich Server v1.82.0 ### Version of Immich Mobile App v1.82.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-server.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /home/me/photos/test_photos:/mnt/media/test_photos:ro env_file: - .env networks: - inside environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-microservices.sh"] networks: - inside volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /home/me/photos/test_photos:/mnt/media/test_photos:ro env_file: - .env depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:release networks: - inside volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - ./volumes/model-cache:/cache env_file: - .env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:release entrypoint: ["/bin/sh", "./entrypoint.sh"] networks: - inside env_file: - .env restart: always redis: container_name: immich_redis networks: - inside image: redis:6.2 restart: always database: container_name: immich_postgres image: postgres:14 networks: - inside env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - ./volumes/pgdata:/var/lib/postgresql/data restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.0 environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data logging: driver: none volumes: - ./volumes/tsdata:/data restart: always networks: - inside immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:release networks: - outside - inside environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 logging: driver: none depends_on: - immich-server restart: always networks: outside: name: immich_outside inside: internal: true ``` ``` ### Your .env content ```Shell ################################################################################### # Database ################################################################################### DB_HOSTNAME=redcated DB_USERNAME=redcated DB_PASSWORD=redcated DB_DATABASE_NAME=redcated # Optional Database settings: # DB_PORT=redcated ################################################################################### # Redis ################################################################################### REDIS_HOSTNAME=immich_redis # Optional Redis settings: # REDIS_PORT=redcated # REDIS_DBINDEX=0 # REDIS_PASSWORD=redcated # REDIS_SOCKET= ################################################################################### # Upload File Location # # This is the location where uploaded files are stored. ################################################################################### UPLOAD_LOCATION=/home/me/immich_photos ################################################################################### # JWT SECRET # # This JWT_SECRET is used to sign the authentication keys for user login # You should set it to a long randomly generated value # You can use this command to generate one: openssl rand -base64 128 ################################################################################### JWT_SECRET=redcated ################################################################################### # Reverse Geocoding # # Reverse geocoding is done locally which has a small impact on memory usage # This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable # This ranges from 0-3 with 3 being the most precise # 3 - Cities > 500 population: ~200MB RAM # 2 - Cities > 1000 population: ~150MB RAM # 1 - Cities > 5000 population: ~80MB RAM # 0 - Cities > 15000 population: ~40MB RAM #################################################################################### DISABLE_REVERSE_GEOCODING=true # REVERSE_GEOCODING_PRECISION=3 #################################################################################### # WEB - Optional # # Custom message on the login page, should be written in HTML form. # For example: # PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich. Email: demo@demo.de Password: demo" #################################################################################### PUBLIC_LOGIN_PAGE_MESSAGE= #################################################################################### # Alternative Service Addresses - Optional # # This is an advanced feature for users who may be running their immich services on different hosts. # It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers. # Note: immich-microservices is bound to 3002, but no references are made #################################################################################### IMMICH_WEB_URL=http://immich-web:3000 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003 TYPESENSE_API_KEY=redcated ``` ### Reproduction steps ```bash 1. `docker-compose up -d` to start the service 2. `docker-compose logs -f` to follow the logs `immich-microservices` begin executing processes that hog most of my CPU Eventually, the posted memory issue occurs and the server application seems to restart ``` ### Additional information _No response_

OVERLORD closed this issue

2026-02-05 01:59:59 +03:00

OVERLORD commented

2026-02-05 02:00:01 +03:00

@alextran1502 commented on GitHub (Oct 18, 2023):

Which actions were going on when you encounter this issue?

@alextran1502 commented on GitHub (Oct 18, 2023): Which actions were going on when you encounter this issue?

OVERLORD commented

2026-02-05 02:00:03 +03:00

@dtirer commented on GitHub (Oct 18, 2023):

Which actions were going on when you encounter this issue?

I updated the log output so there's more context. It shows what happens from the moment the container starts up, to the point where the error ocurrs.

As far as I can tell, no particular jobs are running. When I view the list of jobs in the admin panel, nothing is running and it shows that the "Extract Metadata" job is paused

If I go to the "Repair" section, there do seem to be a lot of errors with files

@dtirer commented on GitHub (Oct 18, 2023): > Which actions were going on when you encounter this issue? I updated the log output so there's more context. It shows what happens from the moment the container starts up, to the point where the error ocurrs. As far as I can tell, no particular jobs are running. When I view the list of jobs in the admin panel, nothing is running and it shows that the "Extract Metadata" job is paused If I go to the "Repair" section, there do seem to be a lot of errors with files ![Screenshot 2023-10-18 at 12 22 38 PM](https://github.com/immich-app/immich/assets/639812/1a861db1-2e1c-479a-8338-dd0958e514ee)

OVERLORD commented

2026-02-05 02:00:07 +03:00

@dtirer commented on GitHub (Oct 18, 2023):

Okay this seems to be a result of some docker networking changes I made.

I previously had all the immich containers on a proxy network. My nginx proxy manager was always apart of this network.

Then I tried adjusting the docker-compose file for immich so that only the immiich-proxy container was apart of the proxy network, and the rest of the immich containers were apart of their own internal network. This seems to be causing the above issue.

Is there a particular reason why this would be happening?

@dtirer commented on GitHub (Oct 18, 2023): Okay this seems to be a result of some docker networking changes I made. I previously had all the immich containers on a `proxy` network. My nginx proxy manager was always apart of this network. Then I tried adjusting the docker-compose file for immich so that only the immiich-proxy container was apart of the `proxy` network, and the rest of the immich containers were apart of their own internal network. This seems to be causing the above issue. Is there a particular reason why this would be happening?

OVERLORD commented

2026-02-05 02:00:10 +03:00

@alextran1502 commented on GitHub (Oct 18, 2023):

I am not sure tbh. I ran into this issue once when running the Medatada extraction job again for all assets. But after restarting the server it seems to work fine so we haven't been able to identify the cause of heap out of memory

@alextran1502 commented on GitHub (Oct 18, 2023): I am not sure tbh. I ran into this issue once when running the Medatada extraction job again for all assets. But after restarting the server it seems to work fine so we haven't been able to identify the cause of heap out of memory

OVERLORD commented

2026-02-05 02:00:13 +03:00

@kfwebdev commented on GitHub (Oct 19, 2023):

I'm also seeing this issue every 5 minutes for immich-microservices only while trying to build thumbnails.

My instance is running Immich v1.82.0 on a Qnap TS-453E with Celeron J6412 4-core, 8gb ram.
Takes up about 50% of cpu and memory resources while running.
Tried reducing job concurrencies all down to 1 and disabling prefer wide gamut but no change.

<--- Last few GCs --->

[7:0x2a39c310000]   189769 ms: Scavenge 2031.4 (2075.5) -> 2028.4 (2076.0) MB, 8.44 / 0.00 ms  (average mu = 0.300, current mu = 0.265) allocation failure; 
[7:0x2a39c310000]   189817 ms: Scavenge 2032.4 (2076.5) -> 2029.4 (2077.0) MB, 8.37 / 0.00 ms  (average mu = 0.300, current mu = 0.265) allocation failure; 
[7:0x2a39c310000]   189871 ms: Scavenge 2033.3 (2077.5) -> 2030.4 (2086.0) MB, 8.08 / 0.00 ms  (average mu = 0.300, current mu = 0.265) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices]
10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
12: 0x7f3e9ab59ef6 
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +61ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +8ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +215ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] InfraModule dependencies initialized +4ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] DomainModule dependencies initialized +16ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:30:38 AM     LOG [JobService] Updating queue concurrency settings

<--- Last few GCs --->

[7:0x4cd7a310000]   178904 ms: Scavenge 2027.4 (2072.0) -> 2024.4 (2072.5) MB, 8.21 / 0.00 ms  (average mu = 0.283, current mu = 0.242) allocation failure; 
[7:0x4cd7a310000]   178948 ms: Scavenge 2028.4 (2073.0) -> 2025.4 (2073.5) MB, 7.87 / 0.00 ms  (average mu = 0.283, current mu = 0.242) allocation failure; 
[7:0x4cd7a310000]   179009 ms: Scavenge 2029.1 (2073.7) -> 2026.1 (2082.2) MB, 9.07 / 0.00 ms  (average mu = 0.283, current mu = 0.242) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices]
10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
12: 0x7f3772d59ef6 
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +51ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +7ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] DomainModule dependencies initialized +13ms
[Nest] 7  - 10/19/2023, 6:33:41 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms

@kfwebdev commented on GitHub (Oct 19, 2023): I'm also seeing this issue every 5 minutes for immich-microservices only while trying to build thumbnails. My instance is running Immich v1.82.0 on a Qnap TS-453E with Celeron J6412 4-core, 8gb ram. Takes up about 50% of cpu and memory resources while running. Tried reducing job concurrencies all down to 1 and disabling prefer wide gamut but no change. ``` <--- Last few GCs ---> [7:0x2a39c310000] 189769 ms: Scavenge 2031.4 (2075.5) -> 2028.4 (2076.0) MB, 8.44 / 0.00 ms (average mu = 0.300, current mu = 0.265) allocation failure; [7:0x2a39c310000] 189817 ms: Scavenge 2032.4 (2076.5) -> 2029.4 (2077.0) MB, 8.37 / 0.00 ms (average mu = 0.300, current mu = 0.265) allocation failure; [7:0x2a39c310000] 189871 ms: Scavenge 2033.3 (2077.5) -> 2030.4 (2086.0) MB, 8.08 / 0.00 ms (average mu = 0.300, current mu = 0.265) allocation failure; <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices] 10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 12: 0x7f3e9ab59ef6 [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [NestFactory] Starting Nest application... [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +61ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] ConfigModule dependencies initialized +8ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +215ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] InfraModule dependencies initialized +4ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] DomainModule dependencies initialized +16ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:30:38 AM LOG [JobService] Updating queue concurrency settings <--- Last few GCs ---> [7:0x4cd7a310000] 178904 ms: Scavenge 2027.4 (2072.0) -> 2024.4 (2072.5) MB, 8.21 / 0.00 ms (average mu = 0.283, current mu = 0.242) allocation failure; [7:0x4cd7a310000] 178948 ms: Scavenge 2028.4 (2073.0) -> 2025.4 (2073.5) MB, 7.87 / 0.00 ms (average mu = 0.283, current mu = 0.242) allocation failure; [7:0x4cd7a310000] 179009 ms: Scavenge 2029.1 (2073.7) -> 2026.1 (2082.2) MB, 9.07 / 0.00 ms (average mu = 0.283, current mu = 0.242) allocation failure; <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices] 10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 12: 0x7f3772d59ef6 [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [NestFactory] Starting Nest application... [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +51ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] ConfigModule dependencies initialized +7ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] InfraModule dependencies initialized +3ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] DomainModule dependencies initialized +13ms [Nest] 7 - 10/19/2023, 6:33:41 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms ```

OVERLORD commented

2026-02-05 02:00:16 +03:00

@jrasm91 commented on GitHub (Oct 19, 2023):

This almost looks like an issue with generating a thumb for s specific file. Any way to figure out which one?

@jrasm91 commented on GitHub (Oct 19, 2023): This almost looks like an issue with generating a thumb for s specific file. Any way to figure out which one?

OVERLORD commented

2026-02-05 02:00:18 +03:00

@kfwebdev commented on GitHub (Oct 19, 2023):

It looks like I was able to stop the crashing for my particular setup by recreating the immich-microservices container with environment variable:
NODE_OPTIONS=--max-old-space-size=5096

I tried NODE_OPTIONS=--max-old-space-size=4096 and it would still crash after hitting 4gb ram although it took longer than 5 minutes to hit the limit. At 5096, the thumbnails are getting created. I also reduced my concurrency to 3 thumbnails instead of the default 5.

I noticed that Video Transcoding had started to process as well. I paused that and the memory usage went down considerably. Maybe the combination of thumbnail generation and video transcoding was causing Node to continue eating up memory.
Now sitting at about 35% of 8gb memory while processing 3 thumbnails in concurrency. Cpu usage maintaining about 90%.

Update:
I disabled Video Transcoding in the settings and recreated immich-microservices again with NODE_OPTIONS=--max-old-space-size=4096. It's now happily creating thumbnails without crashing at 1 concurrency and about 30-40% cpu threshold. 2 concurrency thumbnails pushes it up about 70%. Same 35% of 8gb memory threshold.

In conclusion, it seems that the Node JavaScript heap out of memory error was due to multiple jobs eating up memory until it hit the default max-old-space-size causing a crash.

For reference my Immich instance has 6827 photos and 1570 videos.
Qnap TS-453e running container station.

@kfwebdev commented on GitHub (Oct 19, 2023): It looks like I was able to stop the crashing for my particular setup by recreating the `immich-microservices` container with environment variable: `NODE_OPTIONS=--max-old-space-size=5096` I tried `NODE_OPTIONS=--max-old-space-size=4096` and it would still crash after hitting 4gb ram although it took longer than 5 minutes to hit the limit. At 5096, the thumbnails are getting created. I also reduced my concurrency to 3 thumbnails instead of the default 5. I noticed that Video Transcoding had started to process as well. I paused that and the memory usage went down considerably. Maybe the combination of thumbnail generation and video transcoding was causing Node to continue eating up memory. Now sitting at about 35% of 8gb memory while processing 3 thumbnails in concurrency. Cpu usage maintaining about 90%. **Update:** I disabled Video Transcoding in the settings and recreated `immich-microservices` again with `NODE_OPTIONS=--max-old-space-size=4096`. It's now happily creating thumbnails without crashing at 1 concurrency and about 30-40% cpu threshold. 2 concurrency thumbnails pushes it up about 70%. Same 35% of 8gb memory threshold. In conclusion, it seems that the Node JavaScript heap out of memory error was due to multiple jobs eating up memory until it hit the default `max-old-space-size` causing a crash. For reference my Immich instance has 6827 photos and 1570 videos. Qnap TS-453e running container station.

OVERLORD commented

2026-02-05 02:00:20 +03:00

@jrasm91 commented on GitHub (Oct 19, 2023):

Thanks for the update, what you have reported makes sense. We will try to investigate the issue more.

@jrasm91 commented on GitHub (Oct 19, 2023): Thanks for the update, what you have reported makes sense. We will try to investigate the issue more.

OVERLORD commented

2026-02-05 02:00:22 +03:00

@piciuok commented on GitHub (Oct 20, 2023):

Same here, on QNAP TS-453D.

My library:
PHOTOS: 23970
VIDEOS: 5922
STORAGE: 506 GB

Now, i increase size to 10GB - maybe now it process something successfully

@piciuok commented on GitHub (Oct 20, 2023): Same here, on QNAP TS-453D. My library: PHOTOS: 23970 VIDEOS: 5922 STORAGE: 506 GB Now, i increase size to 10GB - maybe now it process something successfully

OVERLORD commented

2026-02-05 02:00:24 +03:00

@piciuok commented on GitHub (Oct 20, 2023):

Unfortunately, still:

[6:0x30174310000]  1650770 ms: Mark-Compact 10075.2 (10272.5) -> 10057.3 (10273.0) MB, 30121.98 / 0.00 ms  (average mu = 0.111, current mu = 0.035) allocation failure; scavenge might not succeed
[6:0x30174310000]  1676958 ms: Mark-Compact 10075.0 (10273.0) -> 10059.3 (10274.5) MB, 25888.95 / 0.00 ms  (average mu = 0.066, current mu = 0.011) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7f5a8ba99ef6 ```

How can i locate problematic file?

@piciuok commented on GitHub (Oct 20, 2023): Unfortunately, still: ```<--- Last few GCs ---> [6:0x30174310000] 1650770 ms: Mark-Compact 10075.2 (10272.5) -> 10057.3 (10273.0) MB, 30121.98 / 0.00 ms (average mu = 0.111, current mu = 0.035) allocation failure; scavenge might not succeed [6:0x30174310000] 1676958 ms: Mark-Compact 10075.0 (10273.0) -> 10059.3 (10274.5) MB, 25888.95 / 0.00 ms (average mu = 0.066, current mu = 0.011) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7f5a8ba99ef6 ``` How can i locate problematic file?

OVERLORD commented

2026-02-05 02:00:29 +03:00

@jrasm91 commented on GitHub (Oct 24, 2023):

I believe this may be related to reverse geocoding. Can we try deleting the contents of /usr/src/app/.reverse-geocoding-dump (or equivalent) and then restarting the microservices container?

@jrasm91 commented on GitHub (Oct 24, 2023): I believe this may be related to reverse geocoding. Can we try deleting the contents of `/usr/src/app/.reverse-geocoding-dump` (or equivalent) and then restarting the microservices container?

OVERLORD commented

2026-02-05 02:00:31 +03:00

@fatwhy commented on GitHub (Oct 27, 2023):

same issues with microservices, Javascript heap out of memory and core dump when generating thumbnails. On version 1.82.1 running on sysnology NAS.

<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99960 node::Abort() [immich_microservices]
2: 0xb6ffcb [immich_microservices]
3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
5: 0x10d06a5 [immich_microservices]
6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices]
10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
12: 0x7f3c49f19ef6
Aborted (core dumped)

@fatwhy commented on GitHub (Oct 27, 2023): same issues with microservices, Javascript heap out of memory and core dump when generating thumbnails. On version 1.82.1 running on sysnology NAS. <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10ea49a v8::internal::Heap::HandleGCRequest() [immich_microservices] 10: 0x1055907 v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 11: 0x14f7322 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 12: 0x7f3c49f19ef6 Aborted (core dumped)

OVERLORD commented

2026-02-05 02:00:34 +03:00

@jrasm91 commented on GitHub (Oct 27, 2023):

Does it help if the jobs are all paused originally and then microservices is started? Many of these errors are right when microservices are starting and the first thing it does is initialize the reverse geocoder. Also can these out of memory issues be reproduced while reverse geocoding is disabled?

@jrasm91 commented on GitHub (Oct 27, 2023): Does it help if the jobs are all paused originally and then microservices is started? Many of these errors are right when microservices are starting and the first thing it does is initialize the reverse geocoder. Also can these out of memory issues be reproduced while reverse geocoding is disabled?

OVERLORD commented

2026-02-05 02:00:37 +03:00

@fatwhy commented on GitHub (Oct 27, 2023):

All jobs were paused after core dumped, then restarting microservices would result in core dump again shortly.

@fatwhy commented on GitHub (Oct 27, 2023): All jobs were paused after core dumped, then restarting microservices would result in core dump again shortly.

OVERLORD commented

2026-02-05 02:00:39 +03:00

@knakalm commented on GitHub (Nov 4, 2023):

I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage.

immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always
    dns:
      - 192.168.0.100
      - 8.8.8.8

@knakalm commented on GitHub (Nov 4, 2023): I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage. ``` immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - stack.env depends_on: - redis - database - typesense restart: always dns: - 192.168.0.100 - 8.8.8.8 ```

OVERLORD commented

2026-02-05 02:00:41 +03:00

@EricKemmeren commented on GitHub (Nov 4, 2023):

I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage.
immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always
    dns:
      - 192.168.0.100
      - 8.8.8.8

I first had to disable reverse geocoding, but with the added DNS option I could enable it again without eating up my CPU and RAM, thank you!

@EricKemmeren commented on GitHub (Nov 4, 2023): > I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage. > > ``` > immich-microservices: > container_name: immich_microservices > image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} > # extends: > # file: hwaccel.yml > # service: hwaccel > command: ["start.sh", "microservices"] > volumes: > - ${UPLOAD_LOCATION}:/usr/src/app/upload > - /etc/localtime:/etc/localtime:ro > env_file: > - stack.env > depends_on: > - redis > - database > - typesense > restart: always > dns: > - 192.168.0.100 > - 8.8.8.8 > ``` I first had to disable reverse geocoding, but with the added DNS option I could enable it again without eating up my CPU and RAM, thank you!

OVERLORD commented

2026-02-05 02:00:44 +03:00

@fatwhy commented on GitHub (Nov 5, 2023):

I have updated the DNS settings and so far so good. It has been running for over a hour without crashing. Keeping fingers crossed.

@fatwhy commented on GitHub (Nov 5, 2023): I have updated the DNS settings and so far so good. It has been running for over a hour without crashing. Keeping fingers crossed.

OVERLORD commented

2026-02-05 02:00:47 +03:00

@fatwhy commented on GitHub (Nov 5, 2023):

still running after 7 hours. Thanks for the DNS info!

@fatwhy commented on GitHub (Nov 5, 2023): still running after 7 hours. Thanks for the DNS info!

OVERLORD commented

2026-02-05 02:00:50 +03:00

@atlas-shrugged08 commented on GitHub (Nov 5, 2023):

I was facing the same issue, I added the DNS settings (changed the IP to match the IP of my server(?)). Now the frequency has reduced a lot (thank you @knakalm )and the resources are not being used when idle but the oom still gets triggered if I do an action like "merging 2 faces" - tried that a couple of times with different faces and was consistently reproduced. logs below.

2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [NestFactory] Starting Nest application...
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +25ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +149ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] DomainModule dependencies initialized +12ms
2023-11-04 19:46:05 [Nest] 6  - 11/05/2023, 12:46:05 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [SearchService] Running bootstrap
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [NestApplication] Nest application successfully started +28ms
2023-11-04 19:46:20 [Nest] 6  - 11/05/2023, 12:46:20 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 
2023-11-04 19:49:02 
2023-11-04 19:49:02 <--- Last few GCs --->
2023-11-04 19:49:02 
2023-11-04 19:49:02 [6:0x465c02f0000]   177298 ms: Mark-Compact (reduce) 2044.8 (2081.9) -> 2044.2 (2081.7) MB, 1351.85 / 0.01 ms  (+ 98.3 ms in 37 steps since start of marking, biggest step 5.4 ms, walltime since start of marking 1489 ms) (average mu = 0.319, current mu = 0[6:0x465c02f0000]   177310 ms: Scavenge 2045.3 (2081.7) -> 2044.4 (2084.7) MB, 5.04 / 0.00 ms  (average mu = 0.319, current mu = 0.322) allocation failure; 
2023-11-04 19:49:02 
2023-11-04 19:49:02 
2023-11-04 19:49:02 <--- JS stacktrace --->
2023-11-04 19:49:02 
2023-11-04 19:49:02 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-11-04 19:49:03  1: 0xc7b9ec node::Abort() [immich_microservices]
2023-11-04 19:49:03  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-04 19:49:03  3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-04 19:49:03  4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-04 19:49:03  5: 0x1081d4c  [immich_microservices]
2023-11-04 19:49:03  6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-11-04 19:49:03  7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-11-04 19:49:03  8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-04 19:49:03  9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices]
2023-11-04 19:49:03 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
2023-11-04 19:49:03 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-04 19:49:03 12: 0x1862a84  [immich_microservices]
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [NestFactory] Starting Nest application...
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +41ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +10ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +195ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] DomainModule dependencies initialized +14ms
2023-11-04 19:49:07 [Nest] 7  - 11/05/2023, 12:49:07 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [SearchService] Running bootstrap
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [NestApplication] Nest application successfully started +47ms
2023-11-04 19:49:14 [Nest] 7  - 11/05/2023, 12:49:14 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 
2023-11-04 19:50:18 [Nest] 7  - 11/05/2023, 12:50:18 AM     LOG [JobService] Updating queue concurrency settings
2023-11-04 19:51:33 
2023-11-04 19:51:33 <--- Last few GCs --->
2023-11-04 19:51:33 
2023-11-04 19:51:33 [7:0x254422f0000]   145482 ms: Scavenge 2045.0 (2083.2) -> 2043.4 (2083.4) MB, 6.09 / 0.04 ms  (average mu = 0.323, current mu = 0.307) allocation failure; 
2023-11-04 19:51:33 [7:0x254422f0000]   146611 ms: Scavenge (reduce) 2045.1 (2083.4) -> 2043.5 (2080.7) MB, 27.28 / 0.04 ms  (average mu = 0.323, current mu = 0.307) allocation failure; 
2023-11-04 19:51:33 [7:0x254422f0000]   146656 ms: Scavenge (reduce) 2044.3 (2080.7) -> 2043.5 (2080.7) MB, 8.94 / 0.04 ms  (average mu = 0.323, current mu = 0.307) allocation failure; 
2023-11-04 19:51:33 
2023-11-04 19:51:33 
2023-11-04 19:51:33 <--- JS stacktrace --->
2023-11-04 19:51:33 
2023-11-04 19:51:33 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-11-04 19:51:33  1: 0xc7b9ec node::Abort() [immich_microservices]
2023-11-04 19:51:33  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-04 19:51:33  3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-04 19:51:33  4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-04 19:51:33  5: 0x1081d4c  [immich_microservices]
2023-11-04 19:51:33  6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-11-04 19:51:33  7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-11-04 19:51:33  8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-04 19:51:33  9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices]
2023-11-04 19:51:33 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
2023-11-04 19:51:33 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-04 19:51:33 12: 0x1862a84  [immich_microservices]
2023-11-04 19:51:36 [Nest] 7  - 11/05/2023, 12:51:36 AM     LOG [NestFactory] Starting Nest application...
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +44ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +10ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +158ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] DomainModule dependencies initialized +16ms
2023-11-04 19:51:37 [Nest] 7  - 11/05/2023, 12:51:37 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [SearchService] Running bootstrap
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [NestApplication] Nest application successfully started +37ms
2023-11-04 19:51:44 [Nest] 7  - 11/05/2023, 12:51:44 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 
2023-11-05 04:14:12 
2023-11-05 04:14:12 <--- Last few GCs --->
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [NestFactory] Starting Nest application...
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +49ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +6ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] DomainModule dependencies initialized +20ms
2023-11-05 04:14:17 [Nest] 7  - 11/05/2023, 10:14:17 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [SearchService] Running bootstrap
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [NestApplication] Nest application successfully started +35ms
2023-11-05 04:14:25 [Nest] 7  - 11/05/2023, 10:14:25 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 
2023-11-05 04:14:12 
2023-11-05 04:14:12 [7:0x2ea362f0000] 33755937 ms: Scavenge 2041.4 (2083.4) -> 2039.7 (2083.7) MB, 4.69 / 0.00 ms  (average mu = 0.286, current mu = 0.284) allocation failure; 
2023-11-05 04:14:12 [7:0x2ea362f0000] 33755944 ms: Scavenge 2041.5 (2083.7) -> 2039.9 (2083.7) MB, 4.31 / 0.00 ms  (average mu = 0.286, current mu = 0.284) allocation failure; 
2023-11-05 04:14:12 [7:0x2ea362f0000] 33755964 ms: Scavenge 2041.7 (2083.7) -> 2040.0 (2087.9) MB, 11.60 / 0.00 ms  (average mu = 0.286, current mu = 0.284) allocation failure; 
2023-11-05 04:14:12 
2023-11-05 04:14:12 
2023-11-05 04:14:12 <--- JS stacktrace --->
2023-11-05 04:14:12 
2023-11-05 04:14:12 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-11-05 04:14:12  1: 0xc7b9ec node::Abort() [immich_microservices]
2023-11-05 04:14:12  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-05 04:14:12  3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 04:14:12  4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 04:14:12  5: 0x1081d4c  [immich_microservices]
2023-11-05 04:14:12  6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-11-05 04:14:12  7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-11-05 04:14:12  8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-05 04:14:12  9: 0x1070f8c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
2023-11-05 04:14:12 10: 0x1071d50 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
2023-11-05 04:14:12 11: 0x1051a6c v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
2023-11-05 04:14:12 12: 0x1462844 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-05 04:14:12 13: 0x1862a84  [immich_microservices]
2023-11-05 04:17:11 
2023-11-05 04:17:11 <--- Last few GCs --->
2023-11-05 04:17:11 
2023-11-05 04:17:11 [7:0x4200c2f0000]   176180 ms: Scavenge 2045.2 (2082.7) -> 2044.4 (2082.7) MB, 4.01 / 0.00 ms  (average mu = 0.313, current mu = 0.327) allocation failure; 
2023-11-05 04:17:11 [7:0x4200c2f0000]   176189 ms: Scavenge 2045.3 (2082.7) -> 2044.5 (2082.9) MB, 5.66 / 0.00 ms  (average mu = 0.313, current mu = 0.327) allocation failure; 
2023-11-05 04:17:11 [7:0x4200c2f0000]   176205 ms: Scavenge 2045.4 (2082.9) -> 2044.5 (2084.9) MB, 5.72 / 0.00 ms  (average mu = 0.313, current mu = 0.327) allocation failure; 
2023-11-05 04:17:11 
2023-11-05 04:17:11 
2023-11-05 04:17:11 <--- JS stacktrace --->
2023-11-05 04:17:11 
2023-11-05 04:17:11 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-11-05 04:17:11  1: 0xc7b9ec node::Abort() [immich_microservices]
2023-11-05 04:17:11  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-05 04:17:11  3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 04:17:11  4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 04:17:11  5: 0x1081d4c  [immich_microservices]
2023-11-05 04:17:11  6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-11-05 04:17:11  7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-11-05 04:17:11  8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-05 04:17:11  9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices]
2023-11-05 04:17:11 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
2023-11-05 04:17:11 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-05 04:17:11 12: 0x1862a84  [immich_microservices]
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [NestFactory] Starting Nest application...
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +36ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +152ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] DomainModule dependencies initialized +14ms
2023-11-05 04:17:15 [Nest] 7  - 11/05/2023, 10:17:15 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [SearchService] Running bootstrap
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [NestApplication] Nest application successfully started +22ms
2023-11-05 04:17:22 [Nest] 7  - 11/05/2023, 10:17:22 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION]

Edit: Adding the latest one, if it helps:

023-11-05 12:23:37 [Nest] 7  - 11/05/2023, 6:23:37 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-05 12:23:37 [Nest] 7  - 11/05/2023, 6:23:37 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
2023-11-05 12:23:37 [Nest] 7  - 11/05/2023, 6:23:37 PM     LOG [InstanceLoader] DomainModule dependencies initialized +18ms
2023-11-05 12:23:37 [Nest] 7  - 11/05/2023, 6:23:37 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [MetadataService] Initialized local reverse geocoder with cities500
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [SearchService] Running bootstrap
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-05 12:23:33 
2023-11-05 12:23:33 <--- Last few GCs --->
2023-11-05 12:23:33 .3[7:0x51b6c2f0000]   508084 ms: Mark-Compact (reduce) 2047.2 (2083.6) -> 2046.4 (2083.8) MB, 1009.26 / 0.04 ms  (+ 37.5 ms in 14 steps since start of marking, biggest step 5.6 ms, walltime since start of marking 1113 ms) (average mu = 0.413, current mu = 0[7:0x51b6c2f0000]   510054 ms: Mark-Compact (reduce) 2047.4 (2083.8) -> 2046.8 (2084.3) MB, 1249.55 / 0.04 ms  (+ 44.5 ms in 15 steps since start of marking, biggest step 15.3 ms, walltime since start of marking 1330 ms) (average mu = 0.379, current mu = 
2023-11-05 12:23:33 
2023-11-05 12:23:33 <--- JS stacktrace --->
2023-11-05 12:23:33 
2023-11-05 12:23:33 FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2023-11-05 12:23:33  1: 0xc7b9ec node::Abort() [immich_microservices]
2023-11-05 12:23:33  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-05 12:23:33  3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 12:23:33  4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-05 12:23:33  5: 0x1081d4c  [immich_microservices]
2023-11-05 12:23:33  6: 0x1098224 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-05 12:23:33  7: 0x1070f8c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
2023-11-05 12:23:33  8: 0x1071d50 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
2023-11-05 12:23:33  9: 0x1051a6c v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
2023-11-05 12:23:33 10: 0x1462844 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-05 12:23:33 11: 0x1862a84  [immich_microservices]
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [NestApplication] Nest application successfully started +27ms
2023-11-05 12:23:44 [Nest] 7  - 11/05/2023, 6:23:44 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION]

@atlas-shrugged08 commented on GitHub (Nov 5, 2023): I was facing the same issue, I added the DNS settings (changed the IP to match the IP of my server(?)). Now the frequency has reduced a lot (thank you @knakalm )and the resources are not being used when idle but the oom still gets triggered if I do an action like "merging 2 faces" - tried that a couple of times with different faces and was consistently reproduced. logs below. ``` 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [NestFactory] Starting Nest application... 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +25ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +149ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] InfraModule dependencies initialized +2ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] DomainModule dependencies initialized +12ms 2023-11-04 19:46:05 [Nest] 6 - 11/05/2023, 12:46:05 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [SearchService] Running bootstrap 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [NestApplication] Nest application successfully started +28ms 2023-11-04 19:46:20 [Nest] 6 - 11/05/2023, 12:46:20 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 2023-11-04 19:49:02 2023-11-04 19:49:02 <--- Last few GCs ---> 2023-11-04 19:49:02 2023-11-04 19:49:02 [6:0x465c02f0000] 177298 ms: Mark-Compact (reduce) 2044.8 (2081.9) -> 2044.2 (2081.7) MB, 1351.85 / 0.01 ms (+ 98.3 ms in 37 steps since start of marking, biggest step 5.4 ms, walltime since start of marking 1489 ms) (average mu = 0.319, current mu = 0[6:0x465c02f0000] 177310 ms: Scavenge 2045.3 (2081.7) -> 2044.4 (2084.7) MB, 5.04 / 0.00 ms (average mu = 0.319, current mu = 0.322) allocation failure; 2023-11-04 19:49:02 2023-11-04 19:49:02 2023-11-04 19:49:02 <--- JS stacktrace ---> 2023-11-04 19:49:02 2023-11-04 19:49:02 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 2023-11-04 19:49:03 1: 0xc7b9ec node::Abort() [immich_microservices] 2023-11-04 19:49:03 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-04 19:49:03 3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-04 19:49:03 4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-04 19:49:03 5: 0x1081d4c [immich_microservices] 2023-11-04 19:49:03 6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 2023-11-04 19:49:03 7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 2023-11-04 19:49:03 8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-04 19:49:03 9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices] 2023-11-04 19:49:03 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 2023-11-04 19:49:03 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-04 19:49:03 12: 0x1862a84 [immich_microservices] 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [NestFactory] Starting Nest application... 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +41ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] ConfigModule dependencies initialized +10ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +195ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] InfraModule dependencies initialized +3ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] DomainModule dependencies initialized +14ms 2023-11-04 19:49:07 [Nest] 7 - 11/05/2023, 12:49:07 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [SearchService] Running bootstrap 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [NestApplication] Nest application successfully started +47ms 2023-11-04 19:49:14 [Nest] 7 - 11/05/2023, 12:49:14 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 2023-11-04 19:50:18 [Nest] 7 - 11/05/2023, 12:50:18 AM LOG [JobService] Updating queue concurrency settings 2023-11-04 19:51:33 2023-11-04 19:51:33 <--- Last few GCs ---> 2023-11-04 19:51:33 2023-11-04 19:51:33 [7:0x254422f0000] 145482 ms: Scavenge 2045.0 (2083.2) -> 2043.4 (2083.4) MB, 6.09 / 0.04 ms (average mu = 0.323, current mu = 0.307) allocation failure; 2023-11-04 19:51:33 [7:0x254422f0000] 146611 ms: Scavenge (reduce) 2045.1 (2083.4) -> 2043.5 (2080.7) MB, 27.28 / 0.04 ms (average mu = 0.323, current mu = 0.307) allocation failure; 2023-11-04 19:51:33 [7:0x254422f0000] 146656 ms: Scavenge (reduce) 2044.3 (2080.7) -> 2043.5 (2080.7) MB, 8.94 / 0.04 ms (average mu = 0.323, current mu = 0.307) allocation failure; 2023-11-04 19:51:33 2023-11-04 19:51:33 2023-11-04 19:51:33 <--- JS stacktrace ---> 2023-11-04 19:51:33 2023-11-04 19:51:33 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 2023-11-04 19:51:33 1: 0xc7b9ec node::Abort() [immich_microservices] 2023-11-04 19:51:33 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-04 19:51:33 3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-04 19:51:33 4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-04 19:51:33 5: 0x1081d4c [immich_microservices] 2023-11-04 19:51:33 6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 2023-11-04 19:51:33 7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 2023-11-04 19:51:33 8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-04 19:51:33 9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices] 2023-11-04 19:51:33 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 2023-11-04 19:51:33 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-04 19:51:33 12: 0x1862a84 [immich_microservices] 2023-11-04 19:51:36 [Nest] 7 - 11/05/2023, 12:51:36 AM LOG [NestFactory] Starting Nest application... 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +44ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] ConfigModule dependencies initialized +10ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +158ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] InfraModule dependencies initialized +3ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] DomainModule dependencies initialized +16ms 2023-11-04 19:51:37 [Nest] 7 - 11/05/2023, 12:51:37 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [SearchService] Running bootstrap 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [NestApplication] Nest application successfully started +37ms 2023-11-04 19:51:44 [Nest] 7 - 11/05/2023, 12:51:44 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 2023-11-05 04:14:12 2023-11-05 04:14:12 <--- Last few GCs ---> 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [NestFactory] Starting Nest application... 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +49ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] ConfigModule dependencies initialized +6ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] InfraModule dependencies initialized +2ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] DomainModule dependencies initialized +20ms 2023-11-05 04:14:17 [Nest] 7 - 11/05/2023, 10:14:17 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [SearchService] Running bootstrap 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [NestApplication] Nest application successfully started +35ms 2023-11-05 04:14:25 [Nest] 7 - 11/05/2023, 10:14:25 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] 2023-11-05 04:14:12 2023-11-05 04:14:12 [7:0x2ea362f0000] 33755937 ms: Scavenge 2041.4 (2083.4) -> 2039.7 (2083.7) MB, 4.69 / 0.00 ms (average mu = 0.286, current mu = 0.284) allocation failure; 2023-11-05 04:14:12 [7:0x2ea362f0000] 33755944 ms: Scavenge 2041.5 (2083.7) -> 2039.9 (2083.7) MB, 4.31 / 0.00 ms (average mu = 0.286, current mu = 0.284) allocation failure; 2023-11-05 04:14:12 [7:0x2ea362f0000] 33755964 ms: Scavenge 2041.7 (2083.7) -> 2040.0 (2087.9) MB, 11.60 / 0.00 ms (average mu = 0.286, current mu = 0.284) allocation failure; 2023-11-05 04:14:12 2023-11-05 04:14:12 2023-11-05 04:14:12 <--- JS stacktrace ---> 2023-11-05 04:14:12 2023-11-05 04:14:12 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 2023-11-05 04:14:12 1: 0xc7b9ec node::Abort() [immich_microservices] 2023-11-05 04:14:12 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-05 04:14:12 3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 04:14:12 4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 04:14:12 5: 0x1081d4c [immich_microservices] 2023-11-05 04:14:12 6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 2023-11-05 04:14:12 7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 2023-11-05 04:14:12 8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-05 04:14:12 9: 0x1070f8c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 2023-11-05 04:14:12 10: 0x1071d50 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 2023-11-05 04:14:12 11: 0x1051a6c v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 2023-11-05 04:14:12 12: 0x1462844 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-05 04:14:12 13: 0x1862a84 [immich_microservices] 2023-11-05 04:17:11 2023-11-05 04:17:11 <--- Last few GCs ---> 2023-11-05 04:17:11 2023-11-05 04:17:11 [7:0x4200c2f0000] 176180 ms: Scavenge 2045.2 (2082.7) -> 2044.4 (2082.7) MB, 4.01 / 0.00 ms (average mu = 0.313, current mu = 0.327) allocation failure; 2023-11-05 04:17:11 [7:0x4200c2f0000] 176189 ms: Scavenge 2045.3 (2082.7) -> 2044.5 (2082.9) MB, 5.66 / 0.00 ms (average mu = 0.313, current mu = 0.327) allocation failure; 2023-11-05 04:17:11 [7:0x4200c2f0000] 176205 ms: Scavenge 2045.4 (2082.9) -> 2044.5 (2084.9) MB, 5.72 / 0.00 ms (average mu = 0.313, current mu = 0.327) allocation failure; 2023-11-05 04:17:11 2023-11-05 04:17:11 2023-11-05 04:17:11 <--- JS stacktrace ---> 2023-11-05 04:17:11 2023-11-05 04:17:11 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 2023-11-05 04:17:11 1: 0xc7b9ec node::Abort() [immich_microservices] 2023-11-05 04:17:11 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-05 04:17:11 3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 04:17:11 4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 04:17:11 5: 0x1081d4c [immich_microservices] 2023-11-05 04:17:11 6: 0x1082268 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 2023-11-05 04:17:11 7: 0x1097a50 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 2023-11-05 04:17:11 8: 0x109809c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-05 04:17:11 9: 0x1099f90 v8::internal::Heap::HandleGCRequest() [immich_microservices] 2023-11-05 04:17:11 10: 0x1009f3c v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 2023-11-05 04:17:11 11: 0x1461ed8 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-05 04:17:11 12: 0x1862a84 [immich_microservices] 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [NestFactory] Starting Nest application... 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +36ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] ConfigModule dependencies initialized +3ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +152ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] InfraModule dependencies initialized +2ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] DomainModule dependencies initialized +14ms 2023-11-05 04:17:15 [Nest] 7 - 11/05/2023, 10:17:15 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [SearchService] Running bootstrap 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [NestApplication] Nest application successfully started +22ms 2023-11-05 04:17:22 [Nest] 7 - 11/05/2023, 10:17:22 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] ``` Edit: Adding the latest one, if it helps: ``` 023-11-05 12:23:37 [Nest] 7 - 11/05/2023, 6:23:37 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-05 12:23:37 [Nest] 7 - 11/05/2023, 6:23:37 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms 2023-11-05 12:23:37 [Nest] 7 - 11/05/2023, 6:23:37 PM LOG [InstanceLoader] DomainModule dependencies initialized +18ms 2023-11-05 12:23:37 [Nest] 7 - 11/05/2023, 6:23:37 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [MetadataService] Initialized local reverse geocoder with cities500 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [SearchService] Running bootstrap 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-05 12:23:33 2023-11-05 12:23:33 <--- Last few GCs ---> 2023-11-05 12:23:33 .3[7:0x51b6c2f0000] 508084 ms: Mark-Compact (reduce) 2047.2 (2083.6) -> 2046.4 (2083.8) MB, 1009.26 / 0.04 ms (+ 37.5 ms in 14 steps since start of marking, biggest step 5.6 ms, walltime since start of marking 1113 ms) (average mu = 0.413, current mu = 0[7:0x51b6c2f0000] 510054 ms: Mark-Compact (reduce) 2047.4 (2083.8) -> 2046.8 (2084.3) MB, 1249.55 / 0.04 ms (+ 44.5 ms in 15 steps since start of marking, biggest step 15.3 ms, walltime since start of marking 1330 ms) (average mu = 0.379, current mu = 2023-11-05 12:23:33 2023-11-05 12:23:33 <--- JS stacktrace ---> 2023-11-05 12:23:33 2023-11-05 12:23:33 FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 2023-11-05 12:23:33 1: 0xc7b9ec node::Abort() [immich_microservices] 2023-11-05 12:23:33 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-05 12:23:33 3: 0xe7a1ac v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 12:23:33 4: 0xe7a37c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-05 12:23:33 5: 0x1081d4c [immich_microservices] 2023-11-05 12:23:33 6: 0x1098224 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-05 12:23:33 7: 0x1070f8c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 2023-11-05 12:23:33 8: 0x1071d50 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 2023-11-05 12:23:33 9: 0x1051a6c v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 2023-11-05 12:23:33 10: 0x1462844 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-05 12:23:33 11: 0x1862a84 [immich_microservices] 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [NestApplication] Nest application successfully started +27ms 2023-11-05 12:23:44 [Nest] 7 - 11/05/2023, 6:23:44 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.84.0] [PRODUCTION] ```

OVERLORD commented

2026-02-05 02:00:53 +03:00

@fatwhy commented on GitHub (Nov 5, 2023):

It crashed again after running for a day. I will changed the thumbnail job settings from 5 to 3 and see if helps.

@fatwhy commented on GitHub (Nov 5, 2023): It crashed again after running for a day. I will changed the thumbnail job settings from 5 to 3 and see if helps.

OVERLORD commented

2026-02-05 02:00:57 +03:00

@nodis commented on GitHub (Nov 9, 2023):

I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage.
immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always
    dns:
      - 192.168.0.100
      - 8.8.8.8

@knakalm What IP address is' 192.168.0.100 'for you? thanks

@nodis commented on GitHub (Nov 9, 2023): > I've had the same problem. It turns out that if the "microservices" container has misconfigured DNS and cannot access public domains, it will eat up the entire --max-old-space-size=xxxx, accompanied by high CPU usage, and crash. As soon as I added DNS configuration to the container, it ran exactly as expected with minimal RAM usage. > > ``` > immich-microservices: > container_name: immich_microservices > image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} > # extends: > # file: hwaccel.yml > # service: hwaccel > command: ["start.sh", "microservices"] > volumes: > - ${UPLOAD_LOCATION}:/usr/src/app/upload > - /etc/localtime:/etc/localtime:ro > env_file: > - stack.env > depends_on: > - redis > - database > - typesense > restart: always > dns: > - 192.168.0.100 > - 8.8.8.8 > ``` @knakalm What IP address is' 192.168.0.100 'for you? thanks

OVERLORD commented

2026-02-05 02:00:59 +03:00

@knakalm commented on GitHub (Nov 9, 2023):

@knakalm What IP address is' 192.168.0.100 'for you? thanks

It's an address of local DNS server

@knakalm commented on GitHub (Nov 9, 2023): > @knakalm What IP address is' 192.168.0.100 'for you? thanks It's an address of local DNS server

OVERLORD commented

2026-02-05 02:01:02 +03:00

@MrPixel1 commented on GitHub (Nov 12, 2023):

I observed a similar behavior today after trying to get Immich up and running in a compose stack which does not have internet access.

The microservices service keeps eating RAM and crashes when it reaches its limit (or renders my raspberry pi unresponsive).
~~Tried disabling the reverse geocoding (DISABLE_REVERSE_GEOCODING=true in the .env file) but that doesn't seem to make a difference.~~
EDIT 1: Disabling geocoding solved the issue (I had to disable it using an immich.config file).

EDIT 2: The microservices service keeps crashing because of high RAM usage as soon as it tries generating thumbnails. I disabled ML, geocoding, video transcoding and I configured 1 concurrent job max per job type. I see two log messages like this:
LOG [MediaService] Successfully generated JPEG image thumbnail for asset ... and then it crashes/restarts.

@MrPixel1 commented on GitHub (Nov 12, 2023): I observed a similar behavior today after trying to get Immich up and running in a compose stack which does *not* have internet access. The microservices service keeps eating RAM and crashes when it reaches its limit (or renders my raspberry pi unresponsive). ~~Tried disabling the reverse geocoding (`DISABLE_REVERSE_GEOCODING=true` in the .env file) but that doesn't seem to make a difference.~~ EDIT 1: Disabling geocoding solved the issue (I had to disable it using an immich.config file). EDIT 2: The microservices service keeps crashing because of high RAM usage as soon as it tries generating thumbnails. I disabled ML, geocoding, video transcoding and I configured 1 concurrent job max per job type. I see two log messages like this: `LOG [MediaService] Successfully generated JPEG image thumbnail for asset ...` and then it crashes/restarts.

OVERLORD commented

2026-02-05 02:01:04 +03:00

@zackpollard commented on GitHub (Nov 26, 2023):

In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!

@zackpollard commented on GitHub (Nov 26, 2023): In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!

OVERLORD commented

2026-02-05 02:01:07 +03:00

@atlas-shrugged08 commented on GitHub (Nov 30, 2023):

@zackpollard fwiw, my originally reported problem still exists on 1.89.0

2023-11-30 10:24:51 [Nest] 6  - 11/30/2023, 4:24:51 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION] 
2023-11-30 10:24:47 Request #1701361463080: Sleeping for 4s and then retrying request...
2023-11-30 10:38:41 
2023-11-30 10:38:41 <--- Last few GCs --->
2023-11-30 10:38:41 
2023-11-30 10:38:41 [6:0x5c5262f0000]   859771 ms: Scavenge 2044.7 (2081.7) -> 2043.9 (2082.0) MB, 4.44 / 0.00 ms  (average mu = 0.326, current mu = 0.375) allocation failure; 
2023-11-30 10:38:41 [6:0x5c5262f0000]   859777 ms: Scavenge 2044.8 (2082.0) -> 2043.9 (2082.0) MB, 4.02 / 0.00 ms  (average mu = 0.326, current mu = 0.375) allocation failure; 
2023-11-30 10:38:41 [6:0x5c5262f0000]   859784 ms: Scavenge 2044.8 (2082.0) -> 2044.0 (2084.0) MB, 4.23 / 0.00 ms  (average mu = 0.326, current mu = 0.375) allocation failure; 
2023-11-30 10:38:41 
2023-11-30 10:38:41 
2023-11-30 10:38:41 <--- JS stacktrace --->
2023-11-30 10:38:41 
2023-11-30 10:38:41 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-11-30 10:38:41  1: 0xc7b9fc node::Abort() [immich_microservices]
2023-11-30 10:38:41  2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices]
2023-11-30 10:38:41  3: 0xe7a10c v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-30 10:38:41  4: 0xe7a2dc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-11-30 10:38:41  5: 0x1081cac  [immich_microservices]
2023-11-30 10:38:41  6: 0x10821c8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-11-30 10:38:41  7: 0x10979b0 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-11-30 10:38:41  8: 0x1097ffc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-11-30 10:38:41  9: 0x1099ef0 v8::internal::Heap::HandleGCRequest() [immich_microservices]
2023-11-30 10:38:41 10: 0x1009e9c v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
2023-11-30 10:38:41 11: 0x1461e38 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
2023-11-30 10:38:41 12: 0x1862a84  [immich_microservices]
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [NestFactory] Starting Nest application...
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +138ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] DomainModule dependencies initialized +11ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [MetadataRepository] Initializing metadata repository
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [MetadataService] Initialized local reverse geocoder
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [SearchService] Running bootstrap
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [NestApplication] Nest application successfully started +12ms
2023-11-30 10:38:44 [Nest] 7  - 11/30/2023, 4:38:44 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION]

@atlas-shrugged08 commented on GitHub (Nov 30, 2023): @zackpollard fwiw, my originally reported [problem](https://github.com/immich-app/immich/issues/4530#issuecomment-1793763586) still exists on 1.89.0 ``` 2023-11-30 10:24:51 [Nest] 6 - 11/30/2023, 4:24:51 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION] 2023-11-30 10:24:47 Request #1701361463080: Sleeping for 4s and then retrying request... 2023-11-30 10:38:41 2023-11-30 10:38:41 <--- Last few GCs ---> 2023-11-30 10:38:41 2023-11-30 10:38:41 [6:0x5c5262f0000] 859771 ms: Scavenge 2044.7 (2081.7) -> 2043.9 (2082.0) MB, 4.44 / 0.00 ms (average mu = 0.326, current mu = 0.375) allocation failure; 2023-11-30 10:38:41 [6:0x5c5262f0000] 859777 ms: Scavenge 2044.8 (2082.0) -> 2043.9 (2082.0) MB, 4.02 / 0.00 ms (average mu = 0.326, current mu = 0.375) allocation failure; 2023-11-30 10:38:41 [6:0x5c5262f0000] 859784 ms: Scavenge 2044.8 (2082.0) -> 2044.0 (2084.0) MB, 4.23 / 0.00 ms (average mu = 0.326, current mu = 0.375) allocation failure; 2023-11-30 10:38:41 2023-11-30 10:38:41 2023-11-30 10:38:41 <--- JS stacktrace ---> 2023-11-30 10:38:41 2023-11-30 10:38:41 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 2023-11-30 10:38:41 1: 0xc7b9fc node::Abort() [immich_microservices] 2023-11-30 10:38:41 2: 0xb6e418 void node::FPrintF<>(_IO_FILE*, char const*) [immich_microservices] 2023-11-30 10:38:41 3: 0xe7a10c v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-30 10:38:41 4: 0xe7a2dc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-11-30 10:38:41 5: 0x1081cac [immich_microservices] 2023-11-30 10:38:41 6: 0x10821c8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 2023-11-30 10:38:41 7: 0x10979b0 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 2023-11-30 10:38:41 8: 0x1097ffc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 2023-11-30 10:38:41 9: 0x1099ef0 v8::internal::Heap::HandleGCRequest() [immich_microservices] 2023-11-30 10:38:41 10: 0x1009e9c v8::internal::StackGuard::HandleInterrupts() [immich_microservices] 2023-11-30 10:38:41 11: 0x1461e38 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 2023-11-30 10:38:41 12: 0x1862a84 [immich_microservices] 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [NestFactory] Starting Nest application... 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +138ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] DomainModule dependencies initialized +11ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [MetadataRepository] Initializing metadata repository 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [MetadataService] Initialized local reverse geocoder 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [SearchService] Running bootstrap 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [NestApplication] Nest application successfully started +12ms 2023-11-30 10:38:44 [Nest] 7 - 11/30/2023, 4:38:44 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION] ```

OVERLORD commented

2026-02-05 02:01:11 +03:00

@phiii commented on GitHub (Dec 21, 2023):

This is happening for me too on v1.91.4 on Ubuntu Server even with DNS updates to the docker.yml. Extreme high CPU usage. I had to disable reverse geocode settings for the time being.

@phiii commented on GitHub (Dec 21, 2023): This is happening for me too on v1.91.4 on Ubuntu Server even with DNS updates to the docker.yml. Extreme high CPU usage. I had to disable reverse geocode settings for the time being.

OVERLORD commented

2026-02-05 02:01:14 +03:00

@macross5 commented on GitHub (Dec 21, 2023):

I had the save problem with oom on v1.91.4.
I could fix it with the following environment variable on the microcontroller:

NODE_OPTIONS=--max-old-space-size=8192

My synology has enough ram to handle it. I have to say aswell, that my external library is large (multiple TB).

@macross5 commented on GitHub (Dec 21, 2023): I had the save problem with oom on v1.91.4. I could fix it with the following environment variable on the microcontroller: `NODE_OPTIONS=--max-old-space-size=8192` My synology has enough ram to handle it. I have to say aswell, that my external library is large (multiple TB).

OVERLORD commented

2026-02-05 02:01:19 +03:00

@vb0 commented on GitHub (Jan 29, 2024):

There should be some kind of prominent runtime warning if memory tweaks are required for larger libraries. I've had immich_microservices crashing each time I would scan the library for months now, at first I haven't even noticed (except that nothing new got added). Then I've seen the "java running out of memory" and thought it's a regular bug and will be fixed, but for many releases nothing changed and now I tried this "max-old-space-size=8192" and it just works.

Just having too many objects for your configuration shouldn't be looking like a regular bug.

@vb0 commented on GitHub (Jan 29, 2024): There should be some kind of prominent runtime warning if memory tweaks are required for larger libraries. I've had immich_microservices crashing each time I would scan the library for months now, at first I haven't even noticed (except that nothing new got added). Then I've seen the "java running out of memory" and thought it's a regular bug and will be fixed, but for many releases nothing changed and now I tried this "max-old-space-size=8192" and it just works. Just having too many objects for your configuration shouldn't be looking like a regular bug.

OVERLORD commented

2026-02-05 02:01:21 +03:00

@ste7enm commented on GitHub (Apr 8, 2024):

Just to add, I am running v1.101.0 on Unraid and I was having a problem where the Immich docker would idle around >30% CPU usage. I added my DNS to the extra parameters field in advanced view (--dns 192.XXX.XX.XX for future Googlers) per the recommendation above and now I'm getting an average of 0.02% CPU at idle.
Wild.

Edit: This wasn't my pickle, I think. It did help keep it way down at idle yesterday, but after immich_microservices spooled up again in earnest, it does't ever go back down. Now idling (jobs paused) at 40% CPU again. Drats.

@ste7enm commented on GitHub (Apr 8, 2024): Just to add, I am running v1.101.0 on Unraid and I was having a problem where the Immich docker would idle around >30% CPU usage. I added my DNS to the extra parameters field in advanced view (`--dns 192.XXX.XX.XX` for future Googlers) per the recommendation above and now I'm getting an average of 0.02% CPU at idle. **Wild.** Edit: This wasn't my pickle, I think. It did help keep it way down at idle yesterday, but after immich_microservices spooled up again in earnest, it does't ever go back down. Now idling (jobs paused) at 40% CPU again. Drats.

Sign in to join this conversation.

Branches Tags

main

fix/web-people-hidden-state

fix-filename-search-label

chore/yank-cloud-id

chore/oauth-labels

renovate/machine-learning

uhthomas/mobile-fix-app-bar-fade

feat/debug-schema

renovate/typescript-projects

fix/25803

feat/asset-file-apis

chore/translations

fix/web-switch-label-clickable

release/next

fix/timezones

fix/time-zone-upserts

midzelis/wip

push-zpwsovysllvn

push-nwxlpmyzkyrl

push-nvnkszuqwppm

renovate/github-actions

push-smstsuupsowp

refactor/adaptive_image

push-olwpzvrxnomt

push-lmxsupnmxspl

feat/web-chromecast-video-looping

feat/use-native-clients

renovate/flutter

fix/create-face-edited

fix/mobile-ios-mtls

docs/contributing

docs/mise-mobile

renovate/grafana-monorepo

feature/bottom-buttons-order

feat/immich-mobile-ui-showcase

refactor/consolidate-image-requests

renovate/connectivity_plus-7.x

renovate/major-vitest-monorepo

renovate/pypi-python-multipart-vulnerability

fix/mobile-people-query

sqlite_thumbs

feat/html-text

chore/no-macro-validation

refactor/purchase-store

uhthomas/mobile-fix-asset-jump

feat/pano-ocr

feat/shared-link-login

fix/database-backup-db-names

fix-keep-correct-ios-shared-album-asset

fix-memory-generation-and-display

feat/verify-permissions

refactor/album-service-small-tests

fix/ml-rocm-build

fix/flipped-dimensions-mobile

push-vpxwmwwxwnvw

fix-migration-width-height

refactor/more-queries

revert/prettier-translations

refactor/asset-service-queries

fix/locale-settings-desc

chore/add-debug-log

feat/edit-filters

shared-deep-link-handler

feat/mobile-editing

feat/thumbnail-native-clients

feat/platform-clients

feat/integrity-checks-izzy

fix/foreground-cloud-sync

feat/dynamic-layout

filter-by-person

feat/csp

refactor/sidebar

fix/disable-editing

fix/view-timeline-deeplink

image-zoom-on-slow-connection

fix-consider-dar-for-video-dimension

fix/merged-edited-assets

perf/optimize-album-sort

open-api-fix

feat/create-job-with-dto

use-toast-primary

feat/vitest-4

feat/ios-fastlane-match

match-signing

fix-update-time-update-timeline

chore/translation-keys

feat/modal-routes

feat/panorama-tiles

feature/mobile-view-asset-owner

feat/system-settings

feature/show-activity-count

better-info-in-asset-viewer

fix/all-people-count

feat/location-favorites

feature/rearrange-buttons-2

fix/download-storage-template

feat/kb-shortcuts-mobile

fix/people-count

push-qolzzzzxrvvn

chore/originals-in-asset-files

feat/asset-size-columns

ben/tree-a11y

new-search-filter-ui

refactor/expectSelectedReadonly

refactor/mobile-grdb

push-qvuktpxmkknu

feat/mobile-native-local-sync

refactor/timeline_ops

fix/scrubber_end

feat/version.txt

feat/context-menus

feat/server-chunked-uploads

refactor/virtualsegment

refactor/rename_daymonth_groups

fix/restrict-android-bg-worker

feat/android-periodic-worker

fix-remote-sync-clean-up

refactor/timeline_move_ops

renovate/mapbox-mapbox-gl-rtl-text-0.x

fix/timeline_split_selectable

feat/keyboard_actions_help_modal

feat/static_frontend

feat/notification-warnign-android

feat/plugins2

feat/plugins

test/create-workflow-token-action

fix/docs-force

debug/search-result-similarity

debug/cf-chunked-uploads

feat/eslint_rule

feat/search-filter-album/web

refactor/timeline_photostream

refactor/timelineasset_asset

feat/session-permissions

feat/timeline_photostream_assetnav

feat/timeline_minor_optimize

feat/timeline_perf_nocomp

feat/timeline_search_results_actions

feat/timeline_search_results_page

fix/timeline_padding

fix/timeline_search_reactivity_warnings

feat/timeline_scrollbar

feat/timeline_stream_withviewer

fix/timeline_back_forth_nav

refactor/timeline_photostream_component

fix/generated-files-checks

fix/locate-button-local

chore/base-image-mimalloc

refactor/timeline_assetlayout

refactor/timeline_selectable

refactor/timeline_aware_actions

refactor/timeline_monthsegment

feat/remove-old-pages

chore/deps-gradle

tmp_photostream

tmp/lcms

feat/mobile-dynamic-thumbnails

fix/mobile-finer-thumbnail-concurrency

refactor/timeline1

refactor/extract_photostream

refactor/rename_load_api

refactor/timeline2

refactor/timeline3

feat/multi-select-asset-viewer

feat-no-thumbhash-cache

refactor/asset_grid

feat/faster-access-checks

fix/18991

fix/19543

chore/temp-remove

fix/21419

feat/mobile-hdr-images

chore/update-mise-lockfile

feat/mise-server-checks

feat/mise-ci

feat/windows-2025

feat/dev_cli

refactor/mobile-migrate-clients

fix/map-theme

fix/require-checkbox

chore/use_swc

feat/efficient-thumbnail-decoding

refactor/mobile-thumbhash

refactor/mobile-thumbhash-new

fix/mobile-uncached-zoom

feat/beta-background-upload

fix/beta-timeline-memories-setting

fix/failed-uploads-not-removed

feat/mobile-shared-album

feat/groups

drift-map-page

drift-auth-user-sync

fix/disable-memory

feat/add-to-album-action

edit-date-time-action

drift-people-page

sqlite-remove-isIn

feat/inline-storage-columns

chore/required-reviewers

refact/asset-manager

fix/folder-sort

pnpm

feat/widget-multiple-server-urls

chore/medium-tests-dbname

fix/web-no-iterator-find

fix/map-pan-interruption

track-livephotos

timeline_events

chore/oxlint-migration

feat/maintenance-worker

feat/dav

chore/demo-snapshot

refactor/server-side-dedupe

feat/integrity-checks

dev/recognition-eval

lighter_buckets_test

perf/postgres-queue

postgres-queue

focus_rings

refactor/web-stores-1

refactor/add-to-taken

feat/sort-places

feat/sidecar-asset-file

vet

tmp/demo-snapshot-preview

fix/server-migration-file-extension

refactor/mobile-v2

fix/asset-update-race-condition

rknn-toolkit-lite2

refactor/mobile-split-up-search-page

feature/Add-rocm-support-for-machine-learning

feat/rocm

chore/async-hash-file

feat/shared-link-view-count

feat/rotation

feat/graphql

feat/job-ids

feat/ignore-library-permission-error

feat/docker-compose-builder

feat/kysely-typeorm

mobile/onboarding

no-video-player

fix/server-qsv-output-format

chore/server-geodata-tweaks

mobile/native-video-player-no-hero

feat/xxhash

fix/docs-concurrency

feat/preload-ml-textual-model

feat/local-tileserver

refactor/exif-orientation

original-path-infix

refactor/mobile/login-form-1

feat/server-editor-endpoints

fix/server-qsv-vbr

fix-mobile-db-problems

feat/ml-armnn-conversion

feat/mobile/backup-with-album-info

feat/fast-initial-sync-1

chore/handle-output_dims

feat/server-more-robust-generation

feat/unassign-faces

feat/shortcuts-on-asset-grid

feat/background-upload

feat/capacitor-mobile-app-poc

feat/server-nvenc-hw-decoding

release/v1.105

fix/mobile-fetch-non-archive

feat/fine-grained-access-controls

web/automation-ui

feat/mobile-server-endpoint-save-dropdown

feat/blurhash-thumbnail

object-storage

feat/memories-animations

dev/metrics

ml/tflite

feat/ml-export-cli

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: immich-app/immich#1474