A couple integrity questions #2102

Closed
opened 2026-02-05 05:07:08 +03:00 by OVERLORD · 17 comments
Owner

Originally created by @andrewdunndev on GitHub (Feb 1, 2024).

The bug

feel free to convert this to a discussion, if actual bugs emerge from the discussion it would be better to track them individually

In discussion with @jrasm91 wanted to enumerate a couple integrity related log errors on my instance, these are likely instance specific "holes" that can be fallen into (especially if you were an early adopter). Tooling for examining instance integrity would be fantastic but realizing that it's potentially a lower priority than, say, search at this time.

On this instance all jobs have been re-run fully on the entire dataset.

I have never manipulated the underlying files or database, these errors in state were achieved through likely having a long running instance and not getting on top of these errors when they initially emerged.

Most likely these are all missing files that have "ghosted" over time. Some excerpt errors, will happily dig into each of these in more detail. There appears to be many (~300 within a ~200k) instances of these two types of errors which looks like missing assets or potentially originating file formats that immich isn't prepared for (e.g. VHS conversions of home movies where I may have ffmpeg'ed poorly):

Thumbnail Generation

0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "09b82c91-36c5-461f-8305-e98e68c9a341"
0695508d4f2c }
0695508d4f2c 
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Error: Input file contains unsupported image format
0695508d4f2c     at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
0695508d4f2c     at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:41:14)
0695508d4f2c     at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:121:44)
0695508d4f2c     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
0695508d4f2c     at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:108:28)
0695508d4f2c     at async /usr/src/app/dist/domain/job/job.service.js:116:37
0695508d4f2c     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
0695508d4f2c     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)

0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "e7cbe6b7-8647-4401-84f0-2a299a29a24b"
0695508d4f2c }
0695508d4f2c 
0695508d4f2c [Nest] 21  - 02/01/2024, 8:12:40 AM   ERROR [MediaRepository] ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
0695508d4f2c   built with gcc 12 (Debian 12.2.0-14)
0695508d4f2c   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
0695508d4f2c   libavutil      58.  2.100 / 58.  2.100
0695508d4f2c   libavcodec     60.  3.100 / 60.  3.100
0695508d4f2c   libavformat    60.  3.100 / 60.  3.100
0695508d4f2c   libavdevice    60.  1.100 / 60.  1.100
0695508d4f2c   libavfilter     9.  3.100 /  9.  3.100
0695508d4f2c   libswscale      7.  1.100 /  7.  1.100
0695508d4f2c   libswresample   4. 10.100 /  4. 10.100
0695508d4f2c   libpostproc    57.  1.100 / 57.  1.100
0695508d4f2c [matroska,webm @ 0x29018120180] max_analyze_duration 5000000 reached at 5024000 microseconds st:1
0695508d4f2c [matroska,webm @ 0x29018120180] Could not find codec parameters for stream 0 (Video: mpeg4, 1 reference frame, none(left), 1920x1080): unspecified pixel format
0695508d4f2c Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
0695508d4f2c upload/library/andrew/2017/2017-01-02/00011.mkv: could not seek to position 0.000
0695508d4f2c Input #0, matroska,webm, from 'upload/library/andrew/2017/2017-01-02/00011.mkv':
0695508d4f2c   Duration: 00:00:00.03, start: 0.000000, bitrate: 226225 kb/s
0695508d4f2c   Stream #0:0(eng): Video: mpeg4, 1 reference frame, none(left), 1920x1080, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn (default)
0695508d4f2c   Stream #0:1: Audio: ac3, 48000 Hz, stereo, fltp, 256 kb/s (default)
0695508d4f2c Stream mapping:
0695508d4f2c   Stream #0:0 -> #0:0 (mpeg4 (native) -> mjpeg (native))
0695508d4f2c Press [q] to stop, [?] for help
0695508d4f2c EOF in input file 0
0695508d4f2c Terminating demuxer thread 0
0695508d4f2c Cannot determine format of input stream 0:0 after EOF
0695508d4f2c Error marking filters as finished
0695508d4f2c [AVIOContext @ 0x29018240180] Statistics: 1168387 bytes read, 5 seeks
0695508d4f2c Conversion failed!
0695508d4f2c 

Video Transcoding

Likely ~400 of these on my instance. We've always had pixel devices (or "real" cameras) so I think this may be a different bug than the recent #6337.

0695508d4f2c [Nest] 21  - 02/01/2024, 8:27:10 AM   ERROR [JobService] Error: ffprobe exited with code 1
0695508d4f2c ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
0695508d4f2c   built with gcc 12 (Debian 12.2.0-14)
0695508d4f2c   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
0695508d4f2c   libavutil      58.  2.100 / 58.  2.100
0695508d4f2c   libavcodec     60.  3.100 / 60.  3.100
0695508d4f2c   libavformat    60.  3.100 / 60.  3.100
0695508d4f2c   libavdevice    60.  1.100 / 60.  1.100
0695508d4f2c   libavfilter     9.  3.100 /  9.  3.100
0695508d4f2c   libswscale      7.  1.100 /  7.  1.100
0695508d4f2c   libswresample   4. 10.100 /  4. 10.100
0695508d4f2c   libpostproc    57.  1.100 / 57.  1.100
0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] moov atom not found
0695508d4f2c upload/encoded-video/2350292b-85e4-4e11-9576-1eebd343ccd0/a1/23/a12304f3-e4c2-42c0-8f56-4c1ccd23f0e2-MP.mp4: Invalid data found when processing input
0695508d4f2c 
0695508d4f2c     at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
0695508d4f2c     at ChildProcess.emit (node:events:518:28)
0695508d4f2c     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
0695508d4f2c [Nest] 21  - 02/01/2024, 8:27:10 AM   ERROR [JobService] Object:
0695508d4f2c {
0695508d4f2c   "id": "ca4a1c29-9844-4ad1-aabf-e873aee207c5"
0695508d4f2c }

Other Errors

3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:22 PM   ERROR [Error: write EPIPE
3c05918f63e6     at afterWriteDispatched (node:internal/stream_base_commons:160:15)
3c05918f63e6     at writeGeneric (node:internal/stream_base_commons:151:3)
3c05918f63e6     at Socket._writeGeneric (node:net:952:11)
3c05918f63e6     at Socket._write (node:net:964:8)
3c05918f63e6     at doWrite (node:internal/streams/writable:590:12)
3c05918f63e6     at clearBuffer (node:internal/streams/writable:773:7)
3c05918f63e6     at Writable.uncork (node:internal/streams/writable:523:7)
3c05918f63e6     at connectionCorkNT (node:_http_outgoing:955:8)
3c05918f63e6     at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:22 PM   ERROR [Error: write EPIPE
3c05918f63e6     at afterWriteDispatched (node:internal/stream_base_commons:160:15)
3c05918f63e6     at writeGeneric (node:internal/stream_base_commons:151:3)
3c05918f63e6     at Socket._writeGeneric (node:net:952:11)
3c05918f63e6     at Socket._write (node:net:964:8)
3c05918f63e6     at doWrite (node:internal/streams/writable:590:12)
3c05918f63e6     at clearBuffer (node:internal/streams/writable:773:7)
3c05918f63e6     at Writable.uncork (node:internal/streams/writable:523:7)
3c05918f63e6     at connectionCorkNT (node:_http_outgoing:955:8)
3c05918f63e6     at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:23 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:23 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:26 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:26 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:32 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file
3c05918f63e6 [Nest] 3  - 02/01/2024, 12:00:32 PM   ERROR [Error: write EPIPE
3c05918f63e6     at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE

While browsing I see this, maybe nginx headers not configured properly?

3c05918f63e6 [Nest] 3  - 02/01/2024, 12:03:48 PM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

The OS that Immich Server is running on

Centos 9 Stream, Podman, systemd

Version of Immich Server

v1.94.1

Version of Immich Mobile App

v1.94.1.build.121

Platform with the issue

  • Server
  • Web
  • Mobile
Originally created by @andrewdunndev on GitHub (Feb 1, 2024). ### The bug > feel free to convert this to a discussion, if actual bugs emerge from the discussion it would be better to track them individually In [discussion with @jrasm91](https://discord.com/channels/979116623879368755/1200092940614254724/1202431694511145010) wanted to enumerate a couple integrity related log errors on my instance, these are likely instance specific "holes" that can be fallen into (especially if you were an early adopter). Tooling for examining instance integrity would be fantastic but realizing that it's potentially a lower priority than, say, search at this time. On this instance all jobs have been re-run fully on the entire dataset. __I have never manipulated the underlying files or database, these errors in state were achieved through likely having a long running instance and not getting on top of these errors when they initially emerged__. Most likely these are all missing files that have "ghosted" over time. Some excerpt errors, will happily dig into each of these in more detail. There appears to be many (~300 within a ~200k) instances of these two types of errors which looks like missing assets or potentially originating file formats that immich isn't prepared for (e.g. VHS conversions of home movies where I may have ffmpeg'ed poorly): ## Thumbnail Generation ``` 0695508d4f2c [Nest] 21 - 02/01/2024, 8:12:40 AM ERROR [JobService] Object: 0695508d4f2c { 0695508d4f2c "id": "09b82c91-36c5-461f-8305-e98e68c9a341" 0695508d4f2c } 0695508d4f2c 0695508d4f2c [Nest] 21 - 02/01/2024, 8:12:40 AM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format 0695508d4f2c [Nest] 21 - 02/01/2024, 8:12:40 AM ERROR [JobService] Error: Input file contains unsupported image format 0695508d4f2c at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19) 0695508d4f2c at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:41:14) 0695508d4f2c at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:121:44) 0695508d4f2c at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 0695508d4f2c at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:108:28) 0695508d4f2c at async /usr/src/app/dist/domain/job/job.service.js:116:37 0695508d4f2c at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 0695508d4f2c at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) ``` ``` 0695508d4f2c [Nest] 21 - 02/01/2024, 8:12:40 AM ERROR [JobService] Object: 0695508d4f2c { 0695508d4f2c "id": "e7cbe6b7-8647-4401-84f0-2a299a29a24b" 0695508d4f2c } 0695508d4f2c 0695508d4f2c [Nest] 21 - 02/01/2024, 8:12:40 AM ERROR [MediaRepository] ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers 0695508d4f2c built with gcc 12 (Debian 12.2.0-14) 0695508d4f2c configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc 0695508d4f2c libavutil 58. 2.100 / 58. 2.100 0695508d4f2c libavcodec 60. 3.100 / 60. 3.100 0695508d4f2c libavformat 60. 3.100 / 60. 3.100 0695508d4f2c libavdevice 60. 1.100 / 60. 1.100 0695508d4f2c libavfilter 9. 3.100 / 9. 3.100 0695508d4f2c libswscale 7. 1.100 / 7. 1.100 0695508d4f2c libswresample 4. 10.100 / 4. 10.100 0695508d4f2c libpostproc 57. 1.100 / 57. 1.100 0695508d4f2c [matroska,webm @ 0x29018120180] max_analyze_duration 5000000 reached at 5024000 microseconds st:1 0695508d4f2c [matroska,webm @ 0x29018120180] Could not find codec parameters for stream 0 (Video: mpeg4, 1 reference frame, none(left), 1920x1080): unspecified pixel format 0695508d4f2c Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options 0695508d4f2c upload/library/andrew/2017/2017-01-02/00011.mkv: could not seek to position 0.000 0695508d4f2c Input #0, matroska,webm, from 'upload/library/andrew/2017/2017-01-02/00011.mkv': 0695508d4f2c Duration: 00:00:00.03, start: 0.000000, bitrate: 226225 kb/s 0695508d4f2c Stream #0:0(eng): Video: mpeg4, 1 reference frame, none(left), 1920x1080, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn (default) 0695508d4f2c Stream #0:1: Audio: ac3, 48000 Hz, stereo, fltp, 256 kb/s (default) 0695508d4f2c Stream mapping: 0695508d4f2c Stream #0:0 -> #0:0 (mpeg4 (native) -> mjpeg (native)) 0695508d4f2c Press [q] to stop, [?] for help 0695508d4f2c EOF in input file 0 0695508d4f2c Terminating demuxer thread 0 0695508d4f2c Cannot determine format of input stream 0:0 after EOF 0695508d4f2c Error marking filters as finished 0695508d4f2c [AVIOContext @ 0x29018240180] Statistics: 1168387 bytes read, 5 seeks 0695508d4f2c Conversion failed! 0695508d4f2c ``` ## Video Transcoding Likely ~400 of these on my instance. We've always had pixel devices (or "real" cameras) so I think this may be a different bug than the recent #6337. ``` 0695508d4f2c [Nest] 21 - 02/01/2024, 8:27:10 AM ERROR [JobService] Error: ffprobe exited with code 1 0695508d4f2c ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers 0695508d4f2c built with gcc 12 (Debian 12.2.0-14) 0695508d4f2c configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc 0695508d4f2c libavutil 58. 2.100 / 58. 2.100 0695508d4f2c libavcodec 60. 3.100 / 60. 3.100 0695508d4f2c libavformat 60. 3.100 / 60. 3.100 0695508d4f2c libavdevice 60. 1.100 / 60. 1.100 0695508d4f2c libavfilter 9. 3.100 / 9. 3.100 0695508d4f2c libswscale 7. 1.100 / 7. 1.100 0695508d4f2c libswresample 4. 10.100 / 4. 10.100 0695508d4f2c libpostproc 57. 1.100 / 57. 1.100 0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible! 0695508d4f2c [mov,mp4,m4a,3gp,3g2,mj2 @ 0x2539c190180] moov atom not found 0695508d4f2c upload/encoded-video/2350292b-85e4-4e11-9576-1eebd343ccd0/a1/23/a12304f3-e4c2-42c0-8f56-4c1ccd23f0e2-MP.mp4: Invalid data found when processing input 0695508d4f2c 0695508d4f2c at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22) 0695508d4f2c at ChildProcess.emit (node:events:518:28) 0695508d4f2c at ChildProcess._handle.onexit (node:internal/child_process:294:12) 0695508d4f2c [Nest] 21 - 02/01/2024, 8:27:10 AM ERROR [JobService] Object: 0695508d4f2c { 0695508d4f2c "id": "ca4a1c29-9844-4ad1-aabf-e873aee207c5" 0695508d4f2c } ``` ## Other Errors ``` 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:22 PM ERROR [Error: write EPIPE 3c05918f63e6 at afterWriteDispatched (node:internal/stream_base_commons:160:15) 3c05918f63e6 at writeGeneric (node:internal/stream_base_commons:151:3) 3c05918f63e6 at Socket._writeGeneric (node:net:952:11) 3c05918f63e6 at Socket._write (node:net:964:8) 3c05918f63e6 at doWrite (node:internal/streams/writable:590:12) 3c05918f63e6 at clearBuffer (node:internal/streams/writable:773:7) 3c05918f63e6 at Writable.uncork (node:internal/streams/writable:523:7) 3c05918f63e6 at connectionCorkNT (node:_http_outgoing:955:8) 3c05918f63e6 at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Failed to serve file 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:22 PM ERROR [Error: write EPIPE 3c05918f63e6 at afterWriteDispatched (node:internal/stream_base_commons:160:15) 3c05918f63e6 at writeGeneric (node:internal/stream_base_commons:151:3) 3c05918f63e6 at Socket._writeGeneric (node:net:952:11) 3c05918f63e6 at Socket._write (node:net:964:8) 3c05918f63e6 at doWrite (node:internal/streams/writable:590:12) 3c05918f63e6 at clearBuffer (node:internal/streams/writable:773:7) 3c05918f63e6 at Writable.uncork (node:internal/streams/writable:523:7) 3c05918f63e6 at connectionCorkNT (node:_http_outgoing:955:8) 3c05918f63e6 at process.processTicksAndRejections (node:internal/process/task_queues:81:21)] Error: write EPIPE 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:23 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:23 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:26 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:26 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:32 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Failed to serve file 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:00:32 PM ERROR [Error: write EPIPE 3c05918f63e6 at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)] Error: write EPIPE ``` While browsing I see this, maybe nginx headers not configured properly? ``` 3c05918f63e6 [Nest] 3 - 02/01/2024, 12:03:48 PM WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses ``` ### The OS that Immich Server is running on Centos 9 Stream, Podman, systemd ### Version of Immich Server v1.94.1 ### Version of Immich Mobile App v1.94.1.build.121 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile
Author
Owner

@cgeheim commented on GitHub (Feb 1, 2024):

Same problems here

@cgeheim commented on GitHub (Feb 1, 2024): Same problems here
Author
Owner

@dmandrone commented on GitHub (Feb 23, 2024):

Also getting the same issue any time a video is played (see Other Errors section)

@dmandrone commented on GitHub (Feb 23, 2024): Also getting the same issue any time a video is played (see Other Errors section)
Author
Owner

@Pheggas commented on GitHub (Mar 18, 2024):

Bump?? These are serious bugs.

@Pheggas commented on GitHub (Mar 18, 2024): Bump?? These are serious bugs.
Author
Owner

@alextran1502 commented on GitHub (Mar 18, 2024):

@Pheggas, I agree, and we have put more effort into hardening the storage template mechanism. I think you and Andrew was one of the first users of Immich so there could have been artifacts from before we put in the hardening process as well. We keep this open to make sure to revisit this mechanism

cc: @zackpollard

@alextran1502 commented on GitHub (Mar 18, 2024): @Pheggas, I agree, and we have put more effort into hardening the storage template mechanism. I think you and Andrew was one of the first users of Immich so there could have been artifacts from before we put in the hardening process as well. We keep this open to make sure to revisit this mechanism cc: @zackpollard
Author
Owner

@zackpollard commented on GitHub (Mar 18, 2024):

Yea, I would be interested if this issue has gotten worse after our storage migration hardening efforts. On the integrity tooling side, having a automated scan of integrity is something that's on my to-do list. As we have the hash of all files, doing this should be pretty simple overall. The problem is there are many things on my to-do list 😅

@zackpollard commented on GitHub (Mar 18, 2024): Yea, I would be interested if this issue has gotten worse after our storage migration hardening efforts. On the integrity tooling side, having a automated scan of integrity is something that's on my to-do list. As we have the hash of all files, doing this should be pretty simple overall. The problem is there are many things on my to-do list 😅
Author
Owner

@Dmoney517 commented on GitHub (Apr 4, 2024):

I'm seeing the write epipe errors as well. I'm new to the app, is this relates to using the storage template? If I wasn't log watching while setting up and running my imports, I never would have known there was an issue. The videos play and assets load, but it is accompanied by the errors.

@Dmoney517 commented on GitHub (Apr 4, 2024): I'm seeing the write epipe errors as well. I'm new to the app, is this relates to using the storage template? If I wasn't log watching while setting up and running my imports, I never would have known there was an issue. The videos play and assets load, but it is accompanied by the errors.
Author
Owner

@Sid-Sun commented on GitHub (Apr 6, 2024):

I am on v1.101.0
my cluster was set up in August 2023
I was putting labels on my users and doing the migration
even after running full extract metadata job, 3 assets on my server refuse to be migrated:

immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:13 PM   ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration

2 of these have the video transcribe error described here, one does not.

I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on
@zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it.

Thanks <3

@Sid-Sun commented on GitHub (Apr 6, 2024): I am on v1.101.0 my cluster was set up in August 2023 I was putting labels on my users and doing the migration even after running full extract metadata job, 3 assets on my server refuse to be migrated: ``` immich_microservices | [Nest] 8 - 04/06/2024, 7:13:11 PM ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration immich_microservices | [Nest] 8 - 04/06/2024, 7:13:11 PM ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration immich_microservices | [Nest] 8 - 04/06/2024, 7:13:13 PM ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration ``` 2 of these have the video transcribe error described here, one does not. I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on @zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it. Thanks <3
Author
Owner

@zackpollard commented on GitHub (Apr 6, 2024):

I am on v1.101.0
my cluster was set up in August 2023
I was putting labels on my users and doing the migration
even after running full extract metadata job, 3 assets on my server refuse to be migrated:

immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:11 PM   ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration
immich_microservices     | [Nest] 8  - 04/06/2024, 7:13:13 PM   ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration

2 of these have the video transcribe error described here, one does not.

I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on
@zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it.

Thanks <3

Hey, the overall idea is a tool that will incrementally scan all assets on the server over time, recomputing the hash of the asset on disk and comparing it to the hash in the database to make sure it hasn't changed. However before we do this work we wanted to make changes to the current repair page to improve that so it works in the background too, as for big instances this page consistently times out as it computes everything when you load the page.

@zackpollard commented on GitHub (Apr 6, 2024): > I am on v1.101.0 > my cluster was set up in August 2023 > I was putting labels on my users and doing the migration > even after running full extract metadata job, 3 assets on my server refuse to be migrated: > ``` > immich_microservices | [Nest] 8 - 04/06/2024, 7:13:11 PM ERROR [StorageTemplateService] Asset 3e330e45-d0bc-4b92-b169-6b6af999bc54 missing exif info, skipping storage template migration > immich_microservices | [Nest] 8 - 04/06/2024, 7:13:11 PM ERROR [StorageTemplateService] Asset dc6765db-a5ec-426c-813c-a2f0d05a0d8d missing exif info, skipping storage template migration > immich_microservices | [Nest] 8 - 04/06/2024, 7:13:13 PM ERROR [StorageTemplateService] Asset 6eedcbd9-7c2c-4385-ae24-9b428e19fb9a missing exif info, skipping storage template migration > ``` > 2 of these have the video transcribe error described here, one does not. > > I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on > @zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it. > > Thanks <3 Hey, the overall idea is a tool that will incrementally scan all assets on the server over time, recomputing the hash of the asset on disk and comparing it to the hash in the database to make sure it hasn't changed. However before we do this work we wanted to make changes to the current repair page to improve that so it works in the background too, as for big instances this page consistently times out as it computes everything when you load the page.
Author
Owner

@Sid-Sun commented on GitHub (Apr 6, 2024):

I just discovered there is a repair page XD

I see about 413 untracked files on mine at the moment, I have never modified the underlying storage or DB, is this because of the file integrity changing? Is the idea to similarly show the files which have changed integrity over time?

I'll check out how the repair page works & if I have any ideas, thank you! If there is a discussion open you can link me to, I'll catch up on it.

Minor update: One of the assets of mine which couldn't be moved might be a deleted photo that is in the thrash but I haven't visually confirmed it yet.

@Sid-Sun commented on GitHub (Apr 6, 2024): I just discovered there is a repair page XD I see about 413 untracked files on mine at the moment, I have never modified the underlying storage or DB, is this because of the file integrity changing? Is the idea to similarly show the files which have changed integrity over time? I'll check out how the repair page works & if I have any ideas, thank you! If there is a discussion open you can link me to, I'll catch up on it. Minor update: One of the assets of mine which couldn't be moved *_might_* be a deleted photo that is in the thrash but I haven't visually confirmed it yet.
Author
Owner

@brandonkal commented on GitHub (Apr 24, 2024):

I am seeing this EPIPE issue as well. New user since 1.91. I've had to wait 25 seconds for the large image thumbnail to load.

@brandonkal commented on GitHub (Apr 24, 2024): I am seeing this EPIPE issue as well. New user since 1.91. I've had to wait 25 seconds for the large image thumbnail to load.
Author
Owner

@MiguelNdeCarvalho commented on GitHub (Apr 29, 2024):

Hey,

I have created a brand new instance and I have imported all my data by using the immich-go CLI utility and when I run Transcode Videos only for Missing items, almost 1200 items, they never get transcoded and always had them back to the queue whenever I click again, here are the logs that I get.

[Nest] 224  - 04/29/2024, 4:05:21 PM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] moov atom not found
/photos/upload/<redacted>/<redacted>.mov: Invalid data found when processing input

    at ChildProcess.<anonymous> (/app/immich/server/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 224  - 04/29/2024, 4:05:21 PM   ERROR [JobService] Object:
{
  "id": "<redacted>"
}
@MiguelNdeCarvalho commented on GitHub (Apr 29, 2024): Hey, I have created a brand new instance and I have imported all my data by using the `immich-go` CLI utility and when I run **Transcode Videos** only for **Missing items**, almost 1200 items, they never get transcoded and always had them back to the queue whenever I click again, here are the logs that I get. ``` [Nest] 224 - 04/29/2024, 4:05:21 PM ERROR [JobService] Error: ffprobe exited with code 1 ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers built with gcc 12 (Debian 12.2.0-14) configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible! [mov,mp4,m4a,3gp,3g2,mj2 @ 0x310cc170000] moov atom not found /photos/upload/<redacted>/<redacted>.mov: Invalid data found when processing input at ChildProcess.<anonymous> (/app/immich/server/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22) at ChildProcess.emit (node:events:518:28) at ChildProcess._handle.onexit (node:internal/child_process:294:12) [Nest] 224 - 04/29/2024, 4:05:21 PM ERROR [JobService] Object: { "id": "<redacted>" } ```
Author
Owner

@brandonkal commented on GitHub (May 7, 2024):

@MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc)

Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually.

Background: I work in broadcast

@brandonkal commented on GitHub (May 7, 2024): @MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc) Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually. Background: I work in broadcast
Author
Owner

@MiguelNdeCarvalho commented on GitHub (May 7, 2024):

@MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc)

Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually.

Background: I work in broadcast

Thanks for the explanation!

@MiguelNdeCarvalho commented on GitHub (May 7, 2024): > @MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc) > > Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually. > > Background: I work in broadcast > Thanks for the explanation!
Author
Owner

@schmitzkr commented on GitHub (Aug 24, 2024):

I have noticed a few files with this error moov atom not found have made it to my library. Is there a recommended way to remove these files from immich? Can I delete the files only? or do I need to make changes to the database as well?
I have verified outside of immich that the files are very likely corrupted.

@schmitzkr commented on GitHub (Aug 24, 2024): I have noticed a few files with this error `moov atom not found` have made it to my library. Is there a recommended way to remove these files from immich? Can I delete the files only? or do I need to make changes to the database as well? I have verified outside of immich that the files are very likely corrupted.
Author
Owner

@brandonkal commented on GitHub (Aug 24, 2024):

Regarding repairing the files, I’ve had success with Easeus software. You supply it the corrupted files but also known good files in the same format (same recording device, resolution, frame-rate, bitrate etc) and if you are lucky it can recover some of the corrupted files. Because it is corrupted data, it is not a sure thing though.

To remove the files from Immich I believe you should be able to delete them from the UI so they are also marked as removed in the database. If they haven’t been ingested, then a simple disk delete should be fine.

@brandonkal commented on GitHub (Aug 24, 2024): Regarding repairing the files, I’ve had success with Easeus software. You supply it the corrupted files but also known good files in the same format (same recording device, resolution, frame-rate, bitrate etc) and if you are lucky it can recover some of the corrupted files. Because it is corrupted data, it is not a sure thing though. To remove the files from Immich I believe you should be able to delete them from the UI so they are also marked as removed in the database. If they haven’t been ingested, then a simple disk delete should be fine.
Author
Owner

@schmitzkr commented on GitHub (Aug 27, 2024):

Thanks I noticed I can pull them up with the asset Id in the URL, I wasnt able to find them with search so this is helpful.
I have what might be a related question, a lot of my images were taken with a Pixel device and created a lot of motion pictures. Some of them have the corresponding video, others are missing and in the UI the MP play button shows a red exclamation when clicked. I think at some point I lost some of the videos for these MP files, is there a way to decouple the image asset from the video? so they just appears as images?

@schmitzkr commented on GitHub (Aug 27, 2024): Thanks I noticed I can pull them up with the asset Id in the URL, I wasnt able to find them with search so this is helpful. I have what might be a related question, a lot of my images were taken with a Pixel device and created a lot of motion pictures. Some of them have the corresponding video, others are missing and in the UI the MP play button shows a red exclamation when clicked. I think at some point I lost some of the videos for these MP files, is there a way to decouple the image asset from the video? so they just appears as images?
Author
Owner

@jrasm91 commented on GitHub (Sep 4, 2024):

Closing in favor of #12293

@jrasm91 commented on GitHub (Sep 4, 2024): Closing in favor of #12293
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2102