node inside microservices sometimes throws segfault #2600

Closed
opened 2026-02-05 06:16:19 +03:00 by OVERLORD · 8 comments
Owner

Originally created by @marcin-github on GitHub (Mar 15, 2024).

The bug

I noticed that from time to time I got coredumps from node working on microservices. As for now I don't know how to trigger it. In docker logs from this container I'm seeing only this (time of coredump):

immich_microservices       | [Nest] 7  - 03/15/2024, 8:35:53 AM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset aa1f9555-dc75-4a56-8231-c454d222e934
immich_microservices       | [Nest] 7  - 03/15/2024, 8:35:53 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset af620089-5a9d-42ff-9997-850054bd292e
immich_microservices       | [Nest] 7  - 03/15/2024, 8:36:11 AM     LOG [CommunicationRepository] Initialized websocket server
immich_microservices       | [Nest] 7  - 03/15/2024, 8:36:11 AM     LOG [SystemConfigService] LogLevel=log (set via system config)
immich_microservices       | [Nest] 7  - 03/15/2024, 8:36:11 AM     LOG [LibraryService] Starting to watch library c95a9339-590b-47b2-83a4-1f9de674744c with import path(s) /mnt/media/reszta
immich_microservices       | [Nest] 7  - 03/15/2024, 8:36:12 AM     LOG [LibraryService] Starting to watch library c7a97acb-5f03-45fb-aaf6-2b5bcc586bab with import path(s) /mnt/media/xxx
immich_microservices       | [Nest] 7  - 03/15/2024, 8:36:21 AM     LOG [LibraryService] Starting to watch library 1cc889e2-9374-4242-b698-cb57449a54de with import path(s) /mnt/media/yyy
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [MetadataRepository] Initializing metadata repository
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [MetadataService] Initialized local reverse geocoder
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [NestFactory] Starting Nest application...
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] BullModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] ConfigModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] BullModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] BullModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] InfraModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] DomainModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [NestApplication] Nest application successfully started
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.98.2] [PRODUCTION]
immich_microservices       | [Nest] 7  - 03/15/2024, 8:37:12 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 5711548b-5b37-4a7a-8991-f69b21eda04e

This is backtrace from gdb

# echo bt | gdb -q /usr/local/bin/node core-1710488153-0-11-\!usr\!local\!bin\!node-7
Reading symbols from /usr/local/bin/node...

warning: Can't open file /root/orcexec.JANkwe (deleted) during file-backed mapping note processing
[New LWP 18]
[New LWP 9]
[New LWP 10]
[New LWP 7]
[New LWP 14]
[New LWP 11]
[New LWP 16]
[New LWP 13]
[New LWP 3743]
[New LWP 3718]
[New LWP 12]
[New LWP 3747]
[New LWP 3744]
[New LWP 3737]
[New LWP 3746]
[New LWP 3749]
[New LWP 15]
[New LWP 3750]
[New LWP 17]
[New LWP 3748]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `immich_microservices                     '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f1c0fa30b9d in mi_malloc () from /usr/lib/x86_64-linux-gnu/libmimalloc.so.2
[Current thread is 1 (Thread 0x7f1be4ad36c0 (LWP 18))]
(gdb) #0  0x00007f1c0fa30b9d in mi_malloc () at /usr/lib/x86_64-linux-gnu/libmimalloc.so.2
#1  0x00007f1bec5f5679 in g_malloc () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f1bec8d995e in vips.argument_get_instance () at /usr/local/lib/libvips.so.42
#3  0x00007f1bec8da673 in vips_object_set_property () at /usr/local/lib/libvips.so.42
#4  0x00007f1bec55753d in  () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#5  0x00007f1bec55a5ac in g_object_set_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#6  0x00007f1bec55b11c in g_object_set () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#7  0x00007f1bec8df74a in vips_image_new () at /usr/local/lib/libvips.so.42
#8  0x00007f1bec850ef9 in vips_conversion_build () at /usr/local/lib/libvips.so.42
#9  0x00007f1bec862926 in vips_copy_build () at /usr/local/lib/libvips.so.42
#10 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#11 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#12 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42
#13 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42
#14 0x00007f1bec862c79 in vips_copy () at /usr/local/lib/libvips.so.42
#15 0x00007f1bec8e081d in vips_image_decode () at /usr/local/lib/libvips.so.42
#16 0x00007f1bec804c70 in vips_reduceh_build(_VipsObject*) () at /usr/local/lib/libvips.so.42
#17 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#18 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#19 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42
#20 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42
#21 0x00007f1bec805027 in vips_reduceh () at /usr/local/lib/libvips.so.42
#22 0x00007f1bec7fd171 in vips_resize_build () at /usr/local/lib/libvips.so.42
#23 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#24 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#25 0x00007f1becbef813 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) () at /usr/local/lib/libvips-cpp.so.42
#26 0x00007f1becc02f00 in vips::VImage::resize(double, vips::VOption*) const () at /usr/local/lib/libvips-cpp.so.42
#27 0x00007f1becc4e5fb in PipelineWorker::Execute() () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node
#28 0x00007f1becc3bbd1 in Napi::AsyncWorker::OnExecute(Napi::Env) () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node
#29 0x0000000000c67dc9 in node::ThreadPoolWork::ScheduleWork()::{lambda(uv_work_s*)#1}::_FUN(uv_work_s*) ()
#30 0x0000000001886e10 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122
#31 0x00007f1c0f59c134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#32 0x00007f1c0f61ba40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

and bt full

# echo "bt full" | gdb -q /usr/local/bin/node core-1710488153-0-11-\!usr\!local\!bin\!node-7 ode-7
Reading symbols from /usr/local/bin/node...

warning: Can't open file /root/orcexec.JANkwe (deleted) during file-backed mapping note processing
[New LWP 18]
[New LWP 9]
[New LWP 10]
[New LWP 7]
[New LWP 14]
[New LWP 11]
[New LWP 16]
[New LWP 13]
[New LWP 3743]
[New LWP 3718]
[New LWP 12]
[New LWP 3747]
[New LWP 3744]
[New LWP 3737]
[New LWP 3746]
[New LWP 3749]
[New LWP 15]
[New LWP 3750]
[New LWP 17]
[New LWP 3748]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `immich_microservices                     '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f1c0fa30b9d in mi_malloc () from /usr/lib/x86_64-linux-gnu/libmimalloc.so.2
[Current thread is 1 (Thread 0x7f1be4ad36c0 (LWP 18))]
(gdb) #0  0x00007f1c0fa30b9d in mi_malloc () at /usr/lib/x86_64-linux-gnu/libmimalloc.so.2
#1  0x00007f1bec5f5679 in g_malloc () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f1bec8d995e in vips.argument_get_instance () at /usr/local/lib/libvips.so.42
#3  0x00007f1bec8da673 in vips_object_set_property () at /usr/local/lib/libvips.so.42
#4  0x00007f1bec55753d in  () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#5  0x00007f1bec55a5ac in g_object_set_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#6  0x00007f1bec55b11c in g_object_set () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#7  0x00007f1bec8df74a in vips_image_new () at /usr/local/lib/libvips.so.42
#8  0x00007f1bec850ef9 in vips_conversion_build () at /usr/local/lib/libvips.so.42
#9  0x00007f1bec862926 in vips_copy_build () at /usr/local/lib/libvips.so.42
#10 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#11 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#12 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42
#13 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42
#14 0x00007f1bec862c79 in vips_copy () at /usr/local/lib/libvips.so.42
#15 0x00007f1bec8e081d in vips_image_decode () at /usr/local/lib/libvips.so.42
#16 0x00007f1bec804c70 in vips_reduceh_build(_VipsObject*) () at /usr/local/lib/libvips.so.42
#17 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#18 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#19 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42
#20 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42
#21 0x00007f1bec805027 in vips_reduceh () at /usr/local/lib/libvips.so.42
#22 0x00007f1bec7fd171 in vips_resize_build () at /usr/local/lib/libvips.so.42
#23 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42
#24 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42
#25 0x00007f1becbef813 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) () at /usr/local/lib/libvips-cpp.so.42
#26 0x00007f1becc02f00 in vips::VImage::resize(double, vips::VOption*) const () at /usr/local/lib/libvips-cpp.so.42
#27 0x00007f1becc4e5fb in PipelineWorker::Execute() () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node
#28 0x00007f1becc3bbd1 in Napi::AsyncWorker::OnExecute(Napi::Env) () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node
#29 0x0000000000c67dc9 in node::ThreadPoolWork::ScheduleWork()::{lambda(uv_work_s*)#1}::_FUN(uv_work_s*) ()
#30 0x0000000001886e10 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122
        w = 0x352021a2f90
        q = 0x352021a2fa8
        is_slow_work = 0
#31 0x00007f1c0f59c134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139757777401536, -8324149622998704423, -368, 2, 140726968429760, 139757769011200, 8236352952251834073, 8233337364946607833},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#32 0x00007f1c0f61ba40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

The OS that Immich Server is running on

docker container

Version of Immich Server

1.98.2

Version of Immich Mobile App


Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

---

version: '3.8'

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /home/logi/yyy:/mnt/media/yyy:ro
      - /home/logi/xxx:/mnt/media/xxx:ro
      - /home/bb/Pictures:/mnt/media/reszta:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /home/logi/yyy:/mnt/media/yyy:ro
      - /home/logi/xxx:/mnt/media/xxx:ro
      - /home/bb/Pictures:/mnt/media/reszta:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /home/logi/immich/model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    volumes:
      - /home/logi/immich/redis:/data
    command: --save 601 1 --set-proc-title yes --rdbcompression yes --activedefrag yes --rdbcompression yes --list-compress-depth 2
    environment:
      TZ: "Europe/Warsaw"
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.1
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      TZ: "Europe/Warsaw"
    volumes:
      - /home/logi/immich/pgdata:/var/lib/postgresql/data
    restart: always

Your .env content

UPLOAD_LOCATION=/home/logi/immich/immich
IMMICH_VERSION=release

DB_PASSWORD=zzzz

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

TZ=Europe/Warsaw

Reproduction steps

1. I don't now yet how to trigger coredump

Additional information

No response

Originally created by @marcin-github on GitHub (Mar 15, 2024). ### The bug I noticed that from time to time I got coredumps from node working on microservices. As for now I don't know how to trigger it. In docker logs from this container I'm seeing only this (time of coredump): ``` immich_microservices | [Nest] 7 - 03/15/2024, 8:35:53 AM LOG [MediaService] Successfully generated JPEG image thumbnail for asset aa1f9555-dc75-4a56-8231-c454d222e934 immich_microservices | [Nest] 7 - 03/15/2024, 8:35:53 AM LOG [MediaService] Successfully generated WEBP image thumbnail for asset af620089-5a9d-42ff-9997-850054bd292e immich_microservices | [Nest] 7 - 03/15/2024, 8:36:11 AM LOG [CommunicationRepository] Initialized websocket server immich_microservices | [Nest] 7 - 03/15/2024, 8:36:11 AM LOG [SystemConfigService] LogLevel=log (set via system config) immich_microservices | [Nest] 7 - 03/15/2024, 8:36:11 AM LOG [LibraryService] Starting to watch library c95a9339-590b-47b2-83a4-1f9de674744c with import path(s) /mnt/media/reszta immich_microservices | [Nest] 7 - 03/15/2024, 8:36:12 AM LOG [LibraryService] Starting to watch library c7a97acb-5f03-45fb-aaf6-2b5bcc586bab with import path(s) /mnt/media/xxx immich_microservices | [Nest] 7 - 03/15/2024, 8:36:21 AM LOG [LibraryService] Starting to watch library 1cc889e2-9374-4242-b698-cb57449a54de with import path(s) /mnt/media/yyy immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [MetadataRepository] Initializing metadata repository immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [MetadataService] Initialized local reverse geocoder immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] BullModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] ScheduleModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] ConfigModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] BullModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] BullModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] InfraModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] DomainModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [NestApplication] Nest application successfully started immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.98.2] [PRODUCTION] immich_microservices | [Nest] 7 - 03/15/2024, 8:37:12 AM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 5711548b-5b37-4a7a-8991-f69b21eda04e ``` This is backtrace from gdb ```c++ # echo bt | gdb -q /usr/local/bin/node core-1710488153-0-11-\!usr\!local\!bin\!node-7 Reading symbols from /usr/local/bin/node... warning: Can't open file /root/orcexec.JANkwe (deleted) during file-backed mapping note processing [New LWP 18] [New LWP 9] [New LWP 10] [New LWP 7] [New LWP 14] [New LWP 11] [New LWP 16] [New LWP 13] [New LWP 3743] [New LWP 3718] [New LWP 12] [New LWP 3747] [New LWP 3744] [New LWP 3737] [New LWP 3746] [New LWP 3749] [New LWP 15] [New LWP 3750] [New LWP 17] [New LWP 3748] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `immich_microservices '. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1c0fa30b9d in mi_malloc () from /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 [Current thread is 1 (Thread 0x7f1be4ad36c0 (LWP 18))] (gdb) #0 0x00007f1c0fa30b9d in mi_malloc () at /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 #1 0x00007f1bec5f5679 in g_malloc () at /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f1bec8d995e in vips.argument_get_instance () at /usr/local/lib/libvips.so.42 #3 0x00007f1bec8da673 in vips_object_set_property () at /usr/local/lib/libvips.so.42 #4 0x00007f1bec55753d in () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #5 0x00007f1bec55a5ac in g_object_set_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #6 0x00007f1bec55b11c in g_object_set () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #7 0x00007f1bec8df74a in vips_image_new () at /usr/local/lib/libvips.so.42 #8 0x00007f1bec850ef9 in vips_conversion_build () at /usr/local/lib/libvips.so.42 #9 0x00007f1bec862926 in vips_copy_build () at /usr/local/lib/libvips.so.42 #10 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #11 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #12 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42 #13 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42 #14 0x00007f1bec862c79 in vips_copy () at /usr/local/lib/libvips.so.42 #15 0x00007f1bec8e081d in vips_image_decode () at /usr/local/lib/libvips.so.42 #16 0x00007f1bec804c70 in vips_reduceh_build(_VipsObject*) () at /usr/local/lib/libvips.so.42 #17 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #18 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #19 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42 #20 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42 #21 0x00007f1bec805027 in vips_reduceh () at /usr/local/lib/libvips.so.42 #22 0x00007f1bec7fd171 in vips_resize_build () at /usr/local/lib/libvips.so.42 #23 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #24 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #25 0x00007f1becbef813 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) () at /usr/local/lib/libvips-cpp.so.42 #26 0x00007f1becc02f00 in vips::VImage::resize(double, vips::VOption*) const () at /usr/local/lib/libvips-cpp.so.42 #27 0x00007f1becc4e5fb in PipelineWorker::Execute() () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node #28 0x00007f1becc3bbd1 in Napi::AsyncWorker::OnExecute(Napi::Env) () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node #29 0x0000000000c67dc9 in node::ThreadPoolWork::ScheduleWork()::{lambda(uv_work_s*)#1}::_FUN(uv_work_s*) () #30 0x0000000001886e10 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122 #31 0x00007f1c0f59c134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #32 0x00007f1c0f61ba40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 ``` and bt full ```c++ # echo "bt full" | gdb -q /usr/local/bin/node core-1710488153-0-11-\!usr\!local\!bin\!node-7 ode-7 Reading symbols from /usr/local/bin/node... warning: Can't open file /root/orcexec.JANkwe (deleted) during file-backed mapping note processing [New LWP 18] [New LWP 9] [New LWP 10] [New LWP 7] [New LWP 14] [New LWP 11] [New LWP 16] [New LWP 13] [New LWP 3743] [New LWP 3718] [New LWP 12] [New LWP 3747] [New LWP 3744] [New LWP 3737] [New LWP 3746] [New LWP 3749] [New LWP 15] [New LWP 3750] [New LWP 17] [New LWP 3748] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `immich_microservices '. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1c0fa30b9d in mi_malloc () from /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 [Current thread is 1 (Thread 0x7f1be4ad36c0 (LWP 18))] (gdb) #0 0x00007f1c0fa30b9d in mi_malloc () at /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 #1 0x00007f1bec5f5679 in g_malloc () at /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f1bec8d995e in vips.argument_get_instance () at /usr/local/lib/libvips.so.42 #3 0x00007f1bec8da673 in vips_object_set_property () at /usr/local/lib/libvips.so.42 #4 0x00007f1bec55753d in () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #5 0x00007f1bec55a5ac in g_object_set_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #6 0x00007f1bec55b11c in g_object_set () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0 #7 0x00007f1bec8df74a in vips_image_new () at /usr/local/lib/libvips.so.42 #8 0x00007f1bec850ef9 in vips_conversion_build () at /usr/local/lib/libvips.so.42 #9 0x00007f1bec862926 in vips_copy_build () at /usr/local/lib/libvips.so.42 #10 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #11 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #12 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42 #13 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42 #14 0x00007f1bec862c79 in vips_copy () at /usr/local/lib/libvips.so.42 #15 0x00007f1bec8e081d in vips_image_decode () at /usr/local/lib/libvips.so.42 #16 0x00007f1bec804c70 in vips_reduceh_build(_VipsObject*) () at /usr/local/lib/libvips.so.42 #17 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #18 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #19 0x00007f1bec8ec349 in vips_call_required_optional () at /usr/local/lib/libvips.so.42 #20 0x00007f1bec8ecdef in vips_call_split () at /usr/local/lib/libvips.so.42 #21 0x00007f1bec805027 in vips_reduceh () at /usr/local/lib/libvips.so.42 #22 0x00007f1bec7fd171 in vips_resize_build () at /usr/local/lib/libvips.so.42 #23 0x00007f1bec8d9b89 in vips_object_build () at /usr/local/lib/libvips.so.42 #24 0x00007f1bec8e5442 in vips_cache_operation_buildp () at /usr/local/lib/libvips.so.42 #25 0x00007f1becbef813 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) () at /usr/local/lib/libvips-cpp.so.42 #26 0x00007f1becc02f00 in vips::VImage::resize(double, vips::VOption*) const () at /usr/local/lib/libvips-cpp.so.42 #27 0x00007f1becc4e5fb in PipelineWorker::Execute() () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node #28 0x00007f1becc3bbd1 in Napi::AsyncWorker::OnExecute(Napi::Env) () at /usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64.node #29 0x0000000000c67dc9 in node::ThreadPoolWork::ScheduleWork()::{lambda(uv_work_s*)#1}::_FUN(uv_work_s*) () #30 0x0000000001886e10 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122 w = 0x352021a2f90 q = 0x352021a2fa8 is_slow_work = 0 #31 0x00007f1c0f59c134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139757777401536, -8324149622998704423, -368, 2, 140726968429760, 139757769011200, 8236352952251834073, 8233337364946607833}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #32 0x00007f1c0f61ba40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 ``` ### The OS that Immich Server is running on docker container ### Version of Immich Server 1.98.2 ### Version of Immich Mobile App --- ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML --- version: '3.8' services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'immich'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /home/logi/yyy:/mnt/media/yyy:ro - /home/logi/xxx:/mnt/media/xxx:ro - /home/bb/Pictures:/mnt/media/reszta:ro - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'microservices'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /home/logi/yyy:/mnt/media/yyy:ro - /home/logi/xxx:/mnt/media/xxx:ro - /home/bb/Pictures:/mnt/media/reszta:ro - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - /home/logi/immich/model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5 volumes: - /home/logi/immich/redis:/data command: --save 601 1 --set-proc-title yes --rdbcompression yes --activedefrag yes --rdbcompression yes --list-compress-depth 2 environment: TZ: "Europe/Warsaw" restart: always database: container_name: immich_postgres image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.1 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} TZ: "Europe/Warsaw" volumes: - /home/logi/immich/pgdata:/var/lib/postgresql/data restart: always ``` ### Your .env content ```Shell UPLOAD_LOCATION=/home/logi/immich/immich IMMICH_VERSION=release DB_PASSWORD=zzzz # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis TZ=Europe/Warsaw ``` ### Reproduction steps ```bash 1. I don't now yet how to trigger coredump ``` ### Additional information _No response_
Author
Owner

@alextran1502 commented on GitHub (Mar 15, 2024):

Hello, what machine are you running Immich on?

@alextran1502 commented on GitHub (Mar 15, 2024): Hello, what machine are you running Immich on?
Author
Owner

@marcin-github commented on GitHub (Mar 16, 2024):

This is vm on kvm with CPU AMD Ryzen 5 3600 6-Core Processor

@marcin-github commented on GitHub (Mar 16, 2024): This is vm on kvm with CPU `AMD Ryzen 5 3600 6-Core Processor`
Author
Owner

@mertalev commented on GitHub (Mar 16, 2024):

libvips uses JIT compilation for SIMD processing. It seems like there's a bug somewhere in there. We can probably fix this on our end by using Highway instead of ORC, but we should report this to them as well so they can fix it.

@mertalev commented on GitHub (Mar 16, 2024): libvips uses JIT compilation for SIMD processing. It seems like there's a bug somewhere in there. We can probably fix this on our end by using Highway instead of ORC, but we should report this to them as well so they can fix it.
Author
Owner

@mertalev commented on GitHub (Mar 16, 2024):

Can you narrow down a particular image or kind of image that causes this? Also, does this still happen if you set the job concurrency to 1?

@mertalev commented on GitHub (Mar 16, 2024): Can you narrow down a particular image or kind of image that causes this? Also, does this still happen if you set the job concurrency to 1?
Author
Owner

@marcin-github commented on GitHub (Mar 16, 2024):

I'll try to narrow but:
a) I'm newbie in immich, i'm learning how it works and how reads its logs
b) my library is big and everything generates huge logs and takes time
c) it looks that even verbose level isn't helpfull. Meseems worker put into log information what it did but don't put what it will be doing. So when segfault happens it's not easy to correlate with image.

In last hour I got three crashes and only once I have something new in log:

immich_microservices       | ESC[32m[Nest] 7  - ESC[39m03/16/2024, 11:09:22 AM ESC[32m    LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset c780b6bd-2783-4244-a503-6d5e752da77eESC[39m
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-WARNING **: 11:09:22.893: instance of invalid non-instantiatable type 'glong'
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.894: g_signal_handler_is_connected: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_set: assertion 'G_IS_OBJECT (object)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-WARNING **: 11:09:22.895: instance with invalid (NULL) class pointer
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_signal_emit_valist: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-WARNING **: 11:09:22.895: instance with invalid (NULL) class pointer
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_signal_emit_valist: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
immich_microservices       | ESC[32m[Nest] 6  - ESC[39m03/16/2024, 11:09:54 AM ESC[32m    LOGESC[39m ESC[38;5;3m[CommunicationRepository] ESC[39mESC[32mInitialized websocket serverESC[39m
immich_microservices       | ESC[32m[Nest] 6  - ESC[39m03/16/2024, 11:09:54 AM ESC[32m    LOGESC[39m ESC[38;5;3m[SystemConfigService] ESC[39mESC[32mLogLevel=verbose (set via system config)ESC[39m
immich_microservices       | ESC[32m[Nest] 6  - ESC[39m03/16/2024, 11:09:54 AM ESC[32m    LOGESC[39m ESC[38;5;3m[LibraryService] ESC[39mESC[32mStarting to watch library 731008ff-f9c3-48a1-86e6-7ca61dd6b23b with import path(s) /mnt/media/reszta

so I'm not sure if this messages strictly corelated with segfault.

@marcin-github commented on GitHub (Mar 16, 2024): I'll try to narrow but: a) I'm newbie in immich, i'm learning how it works and how reads its logs b) my library is big and everything generates huge logs and takes time c) it looks that even verbose level isn't helpfull. Meseems worker put into log information what it did but don't put what it will be doing. So when segfault happens it's not easy to correlate with image. In last hour I got three crashes and only once I have something new in log: ``` immich_microservices | ESC[32m[Nest] 7 - ESC[39m03/16/2024, 11:09:22 AM ESC[32m LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset c780b6bd-2783-4244-a503-6d5e752da77eESC[39m immich_microservices | immich_microservices | (process:7): GLib-GObject-WARNING **: 11:09:22.893: instance of invalid non-instantiatable type 'glong' immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.894: g_signal_handler_is_connected: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_unref: assertion 'G_IS_OBJECT (object)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_set: assertion 'G_IS_OBJECT (object)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_unref: assertion 'G_IS_OBJECT (object)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-WARNING **: 11:09:22.895: instance with invalid (NULL) class pointer immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_signal_emit_valist: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-WARNING **: 11:09:22.895: instance with invalid (NULL) class pointer immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_signal_emit_valist: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 11:09:22.895: g_object_ref: assertion 'G_IS_OBJECT (object)' failed immich_microservices | ESC[32m[Nest] 6 - ESC[39m03/16/2024, 11:09:54 AM ESC[32m LOGESC[39m ESC[38;5;3m[CommunicationRepository] ESC[39mESC[32mInitialized websocket serverESC[39m immich_microservices | ESC[32m[Nest] 6 - ESC[39m03/16/2024, 11:09:54 AM ESC[32m LOGESC[39m ESC[38;5;3m[SystemConfigService] ESC[39mESC[32mLogLevel=verbose (set via system config)ESC[39m immich_microservices | ESC[32m[Nest] 6 - ESC[39m03/16/2024, 11:09:54 AM ESC[32m LOGESC[39m ESC[38;5;3m[LibraryService] ESC[39mESC[32mStarting to watch library 731008ff-f9c3-48a1-86e6-7ca61dd6b23b with import path(s) /mnt/media/reszta ``` so I'm not sure if this messages strictly corelated with segfault.
Author
Owner

@marcin-github commented on GitHub (Mar 16, 2024):

There is something like this GENERATE THUMBNAILS 6 failed but 6 failed isn't clickable, it would be nice to have possibility to see messages related to reported failed jobs?

edit:
crash from a couple minutes ago:

immich_microservices       | ESC[32m[Nest] 7  - ESC[39m03/16/2024, 12:30:15 PM ESC[32m    LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset f61c65f5-f671-4754-87c3-21e06ae353b1ESC[39m
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-WARNING **: 12:30:15.101: invalid (NULL) pointer instance
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-CRITICAL **: 12:30:15.101: g_signal_handler_is_connected: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
immich_microservices       | ESC[32m[Nest] 7  - ESC[39m03/16/2024, 12:30:15 PM ESC[32m    LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset b63b594c-dcea-4c5e-bd5b-8ec178588384ESC[39m
immich_microservices       |
immich_microservices       | (process:7): GLib-GObject-WARNING **: 12:30:15.102: g_object_set_is_valid_property: object class 'VipsImage' has no property named 'bands'

Another next crash (which happens after above) is without any message. Looks a little randomly (but I don't expect problems in hardware ;) this vm is heavy used and I didn't spot any suspected behaviour in other services)

@marcin-github commented on GitHub (Mar 16, 2024): There is something like this `GENERATE THUMBNAILS 6 failed ` but `6 failed` isn't clickable, it would be nice to have possibility to see messages related to reported failed jobs? edit: crash from a couple minutes ago: ``` immich_microservices | ESC[32m[Nest] 7 - ESC[39m03/16/2024, 12:30:15 PM ESC[32m LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset f61c65f5-f671-4754-87c3-21e06ae353b1ESC[39m immich_microservices | immich_microservices | (process:7): GLib-GObject-WARNING **: 12:30:15.101: invalid (NULL) pointer instance immich_microservices | immich_microservices | (process:7): GLib-GObject-CRITICAL **: 12:30:15.101: g_signal_handler_is_connected: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed immich_microservices | ESC[32m[Nest] 7 - ESC[39m03/16/2024, 12:30:15 PM ESC[32m LOGESC[39m ESC[38;5;3m[MediaService] ESC[39mESC[32mSuccessfully generated WEBP image thumbnail for asset b63b594c-dcea-4c5e-bd5b-8ec178588384ESC[39m immich_microservices | immich_microservices | (process:7): GLib-GObject-WARNING **: 12:30:15.102: g_object_set_is_valid_property: object class 'VipsImage' has no property named 'bands' ``` Another next crash (which happens after above) is without any message. Looks a little randomly (but I don't expect problems in hardware ;) this vm is heavy used and I didn't spot any suspected behaviour in other services)
Author
Owner

@marcin-github commented on GitHub (Mar 18, 2024):

Another log I catched:

immich_server              | [Nest] 7  - 03/18/2024, 11:57:56 AM   DEBUG [JobService] Handling command: queue=faceDetection,force=false







immich_microservices       |
immich_microservices       | <--- Last few GCs --->
immich_microservices       |
immich_microservices       | [7:0x5a0d6320000]   807352 ms: Mark-Compact 3992.6 (4135.1) -> 3976.7 (4135.4) MB, 2267.28 / 0.00 ms  (average mu = 0.130, current mu = 0.024) alloc
ation failure; scavenge might not succeed
immich_microservices       | [7:0x5a0d6320000]   809549 ms: Mark-Compact 3994.2 (4137.1) -> 3978.1 (4137.4) MB, 2122.46 / 0.00 ms  (average mu = 0.086, current mu = 0.034) alloc
ation failure; scavenge might not succeed
immich_microservices       |
immich_microservices       |
immich_microservices       | <--- JS stacktrace --->
immich_microservices       |
immich_microservices       | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices       | ----- Native stack trace -----
immich_microservices       |

immich_microservices       |
immich_microservices       |  1: 0xca5580 node::Abort() [immich_microservices]
immich_microservices       |  2: 0xb781f9  [immich_microservices]
immich_microservices       |  3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
immich_microservices       |  4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
immich_microservices       |  5: 0x10dc505  [immich_microservices]
immich_microservices       |  6: 0x10dca94 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
immich_microservices       |  7: 0x10f3984 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immi
ch_microservices]
immich_microservices       |  8: 0x10f419c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_
microservices]
immich_microservices       |  9: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::
internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 10: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8:
:internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 11: 0x10a7d56 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 12: 0x1099984 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8:
:internal::AllocationAlignment) [immich_microservices]
immich_microservices       | 13: 0x109c166 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [immich_microservices]
immich_microservices       | 14: 0x13ec3a7 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType
) [immich_microservices]
immich_microservices       | 15: 0xed95cd v8::String::Utf8Length(v8::Isolate*) const [immich_microservices]
immich_microservices       | 16: 0xc729e7  [immich_microservices]
immich_microservices       | 17: 0x7f4c96c8d546

@marcin-github commented on GitHub (Mar 18, 2024): Another log I catched: ``` immich_server | [Nest] 7 - 03/18/2024, 11:57:56 AM DEBUG [JobService] Handling command: queue=faceDetection,force=false immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | [7:0x5a0d6320000] 807352 ms: Mark-Compact 3992.6 (4135.1) -> 3976.7 (4135.4) MB, 2267.28 / 0.00 ms (average mu = 0.130, current mu = 0.024) alloc ation failure; scavenge might not succeed immich_microservices | [7:0x5a0d6320000] 809549 ms: Mark-Compact 3994.2 (4137.1) -> 3978.1 (4137.4) MB, 2122.46 / 0.00 ms (average mu = 0.086, current mu = 0.034) alloc ation failure; scavenge might not succeed immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory immich_microservices | ----- Native stack trace ----- immich_microservices | immich_microservices | immich_microservices | 1: 0xca5580 node::Abort() [immich_microservices] immich_microservices | 2: 0xb781f9 [immich_microservices] immich_microservices | 3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] immich_microservices | 4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] immich_microservices | 5: 0x10dc505 [immich_microservices] immich_microservices | 6: 0x10dca94 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] immich_microservices | 7: 0x10f3984 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immi ch_microservices] immich_microservices | 8: 0x10f419c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_ microservices] immich_microservices | 9: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8:: internal::AllocationAlignment) [immich_microservices] immich_microservices | 10: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8: :internal::AllocationAlignment) [immich_microservices] immich_microservices | 11: 0x10a7d56 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 12: 0x1099984 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8: :internal::AllocationAlignment) [immich_microservices] immich_microservices | 13: 0x109c166 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [immich_microservices] immich_microservices | 14: 0x13ec3a7 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType ) [immich_microservices] immich_microservices | 15: 0xed95cd v8::String::Utf8Length(v8::Isolate*) const [immich_microservices] immich_microservices | 16: 0xc729e7 [immich_microservices] immich_microservices | 17: 0x7f4c96c8d546 ```
Author
Owner

@marcin-github commented on GitHub (Apr 9, 2024):

segfault was due to broken images. I didn't track which one triggers bug.

@marcin-github commented on GitHub (Apr 9, 2024): segfault was due to broken images. I didn't track which one triggers bug.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2600