[BUG] Javascript heap out of memory after extract metadata #1272

New Issue

Closed

opened 2026-02-05 01:06:24 +03:00 by OVERLORD · 14 comments

OVERLORD commented 2026-02-05 01:06:24 +03:00

Owner

Copy Link

Originally created by @eygraber on GitHub (Aug 23, 2023).

The bug

I noticed that there were a few duplicate images in my library that were exactly the same except that one looked like it had reverse geocoding information added to it. I though that running the extract metadata job would fix this, so I ran it on All.

At first there were a bunch of lines in the logs like:

immich_microservices     | [Nest] 7  - 08/23/2023, 12:20:49 AM     LOG [MetadataExtractionProcessor] Already extracted asset upload/library/admin/2020/2020-10-07/PXL_20201007_161535958.mp4.

Then there were a bunch of errors that looked like:

immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:02 AM   ERROR [MetadataExtractionProcessor] Failed to extract live photo upload/library/admin/2018/2018-08-12/MVIMG_20180812_191539.jpg: QueryFailedError: duplicate key value violates unique constraint "UQ_userid_checksum"
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] ERROR:  duplicate key value violates unique constraint "UQ_userid_checksum"
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] DETAIL:  Key ("ownerId", checksum)=(d15dd236-5058-43b6-99da-b0c02a2db9d9, \x0d994499ab2f0ddd6906983bd5381f6a2d5680cd) already exists.
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "deviceId", "type", "originalPath", "resizePath", "webpPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "fileCreatedAt", "fileModifiedAt", "isFavorite", "isArchived", "isReadOnly", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, DEFAULT, $10, $11, $12, $13, $14, $15, $16, $17, DEFAULT, $18, $19) RETURNING "id", "webpPath", "encodedVideoPath", "createdAt", "updatedAt", "isFavorite", "isArchived", "isReadOnly", "isVisible"

and

immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG for asset 45496791-74e5-4978-857b-6463fd3974fe at upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM    WARN [MetadataExtractionProcessor] Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG
immich_microservices     |     at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices     |     at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Unable to run job handler: Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG'
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG'
immich_microservices     |     at Object.statSync (node:fs:1690:3)
immich_microservices     |     at MetadataExtractionProcessor.handlePhotoMetadataExtraction (/usr/src/app/dist/microservices/processors/metadata-extraction.processor.js:272:45)
immich_microservices     |     at async /usr/src/app/dist/domain/job/job.service.js:104:37
immich_microservices     |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:342:28)
immich_microservices     |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:525:24)
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Object:
immich_microservices     | {
immich_microservices     |   "id": "45496791-74e5-4978-857b-6463fd3974fe"
immich_microservices     | }
immich_microservices     | 
immich_microservices     | 

After running the job for a while, this gets printed:

microservices heap out of memory

immich_microservices     | Request #1692760955977: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692760955977: Sleeping for 4s and then retrying request...
immich_typesense         | I20230823 03:23:01.688388   730 log.cpp:523] close a full segment. Current first_index: 425757 last_index: 425765 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000425757_00000000000000425765
immich_typesense         | I20230823 03:23:01.924839   730 log.cpp:537] Renamed `/data/state/log/log_inprogress_00000000000000425757' to `/data/state/log/log_00000000000000425757_00000000000000425765'
immich_typesense         | I20230823 03:23:01.924924   730 log.cpp:108] Created new segment `/data/state/log/log_inprogress_00000000000000425766' with fd=174
immich_typesense         | I20230823 03:23:02.298883   645 raft_server.cpp:546] Term: 21, last_index index: 425767, committed_index: 425766, known_applied_index: 425766, applying_index: 0, queued_writes: 0, pending_queue_size: 205, local_sequence: 33327184
immich_typesense         | I20230823 03:23:02.298957   736 raft_server.h:60] Peer refresh succeeded!
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- Last few GCs --->
immich_microservices     | 
immich_microservices     | 
immich_microservices     | [7:0x595e0840000]   126213 ms: Scavenge 4073.0 (4129.7) -> 4066.3 (4130.2) MB, 3.9 / 0.0 ms  (average mu = 0.301, current mu = 0.292) allocation failure; 
immich_microservices     | [7:0x595e0840000]   126222 ms: Scavenge 4073.5 (4130.2) -> 4067.0 (4130.7) MB, 3.1 / 0.0 ms  (average mu = 0.301, current mu = 0.292) allocation failure; 
immich_microservices     | [7:0x595e0840000]   126590 ms: Scavenge 4074.1 (4130.7) -> 4067.4 (4147.2) MB, 362.2 / 0.0 ms  (average mu = 0.301, current mu = 0.292) allocation failure; 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- JS stacktrace --->
immich_microservices     | 
immich_microservices     | 
immich_microservices     | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
immich_microservices     |  1: 0xb83f50 node::Abort() [immich_microservices]
immich_microservices     |  2: 0xa94834  [immich_microservices]
immich_microservices     |  3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  5: 0xf42265  [immich_microservices]
immich_microservices     |  6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
immich_microservices     |  7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     |  8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     |  9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
immich_microservices     | 10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
immich_microservices     | 11: 0x17035b9  [immich_microservices]

postgres giant SELECT

immich_postgres          | 2023-08-22 23:23:02.565 EDT [1083] LOG:  could not send data to client: Connection reset by peer
immich_postgres          | 2023-08-22 23:23:02.565 EDT [1083] STATEMENT:  SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."thumbhash" AS "AssetEntity_thumbhash", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."isReadOnly" AS "AssetEntity_isReadOnly", "AssetEntity"."checksum" AS "AssetEntity_checksum", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity"."sidecarPath" AS "AssetEntity_sidecarPath", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."projectionType" AS "AssetEntity__AssetEntity_exifInfo_projectionType", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "AssetEntity__AssetEntity_faces"."imageWidth" AS "AssetEntity__AssetEntity_faces_imageWidth", "AssetEntity__AssetEntity_faces"."imageHeight" AS "AssetEntity__AssetEntity_faces_imageHeight", "AssetEntity__AssetEntity_faces"."boundingBoxX1" AS "AssetEntity__AssetEntity_faces_boundingBoxX1", "AssetEntity__AssetEntity_faces"."boundingBoxY1" AS "AssetEntity__AssetEntity_faces_boundingBoxY1", "AssetEntity__AssetEntity_faces"."boundingBoxX2" AS "AssetEntity__AssetEntity_faces_boundingBoxX2", "AssetEntity__AssetEntity_faces"."boundingBoxY2" AS "AssetEntity__AssetEntity_faces_boundingBoxY2", "8258e303a73a72cf6abb13d73fb592dde0d68280"."id" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_id", "8258e303a73a72cf6abb13d73fb592dde0d68280"."createdAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_createdAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."updatedAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_updatedAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."ownerId" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_ownerId", "8258e303a73a72cf6abb13d73fb592dde0d68280"."name" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_name", "8258e303a73a72cf6abb13d73fb592dde0d68280"."birthDate" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_birthDate", "8258e303a73a72cf6abb13d73fb592dde0d68280"."thumbnailPath" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_thumbnailPath", "8258e303a73a72cf6abb13d73fb592dde0d68280"."isHidden" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_isHidden" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId"  LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id"  LEFT JOIN "person" "8258e303a73a72cf6abb13d73fb592dde0d68280" ON "8258e303a73a72cf6abb13d73fb592dde0d68280"."id"="AssetEntity__AssetEntity_faces"."personId" WHERE ("AssetEntity"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $50, $51, $52, $53, $54, $55, $56, $57, $58, $59, $60, $61, $62, $63, $64, $65, $66, $67, $68, $69, $70, $71, $72, $73, $74, $75, $76, $77, $78, $79, $80, $81, $82, $83, $84, $85, $86, $87, $88, $89, $90, $91, $92, $93, $94, $95, $96, $97, $98, $99, $100, $101, $102, $103, $104, $105, $106, $107, $108, $109, $110, $111, $112, $113, $114, $115, $116, $117, $118, $119, $120, $121, $122, $123, $124, $125, $126, $127, $128, $129, $130, $131, $132, $133, $134, $135, $136, $137, $138, $139, $140, $141, $142, $143, $144, $145, $146, $147, $148, $149, $150, $151, $152, $153, $154, $155, $156, $157, $158, $159, $160, $161, $162, $163, $164, $165, $166, $167, $168, $169, $170, $171, $172, $173, $174, $175, $176, $177, $178, $179, $180, $181, $182, $183, $184, $185, $186, $187, $188, $189, $190, $191, $192, $193, $194, $195, $196, $197, $198, $199, $200, $201, $202, $203, $204, $205, $206, $207, $208, $209, $210, $211, $212, $213, $214, $215, $216, $217, $218, $219, $220, $221, $222, $223, $224, $225, $226, $227, $228, $229, $230, $231, $232, $233, $234, $235, $236, $237, $238, $239, $240, $241, $242, $243, $244, $245, $246, $247, $248, $249, $250, $251, $252, $253, $254, $255, $256, $257, $258, $259, $260, $261, $262, $263, $264, $265, $266, $267, $268, $269, $270, $271, $272, $273, $274, $275, $276, $277, $278, $279, $280, $281, $282, $283, $284, $285, $286, $287, $288, $289, $290, $291, $292, $293, $294, $295, $296, $297, $298, $299, $300, $301, $302, $303, $304, $305, $306, $307, $308, $309, $310, $311, $312, $313, $314, $315, $316, $317, $318, $319, $320, $321, $322, $323, $324, $325, $326, $327, $328, $329, $330, $331, $332, $333, $334, $335, $336, $337, $338, $339, $340, $341, $342, $343, $344, $345, $346, $347, $348, $349, $350, $351, $352, $353, $354, $355, $356, $357, $358, $359, $360, $361, $362, $363, $364, $365, $366, $367, $368, $369, $370, $371, $372, $373, $374, $375, $376, $377, $378, $379, $380, $381, $382, $383, $384, $385, $386, $387, $388, $389, $390, $391, $392, $393, $394, $395, $396, $397, $398, $399, $400, $401, $402, $403, $404, $405, $406, $407, $408, $409, $410, $411, $412, $413, $414, $415, $416, $417, $418, $419, $420, $421, $422, $423, $424, $425, $426, $427, $428, $429, $430, $431, $432, $433, $434, $435, $436, $437, $438, $439, $440, $441, $442, $443, $444, $445, $446, $447, $448, $449, $450, $451, $452, $453, $454, $455, $456, $457, $458, $459, $460, $461, $462, $463, $464, $465, $466, $467, $468, $469, $470, $471, $472, $473, $474, $475, $476, $477, $478, $479, $480, $481, $482, $483, $484, $485, $486, $487, $488, $489, $490, $491, $492, $493, $494, $495, $496, $497, $498, $499, $500, $501, $502, $503, $504, $505, $506, $507, $508, $509, $510, $511, $512, $513, $514, $515, $516, $517, $518, $519, $520, $521, $522, $523, $524, $525, $526, $527, $528, $529, $530, $531, $532, $533, $534, $535, $536, $537, $538, $539, $540, $541, $542, $543, $544, $545, $546, $547, $548, $549, $550, $551, $552, $553, $554, $555, $556, $557, $558, $559, $560, $561, $562, $563, $564, $565, $566, $567, $568, $569, $570, $571, $572, $573, $574, $575, $576, $577, $578, $579, $580, $581, $582, $583, $584, $585, $586, $587, $588, $589, $590, $591, $592, $593, $594, $595, $596, $597, $598, $599, $600, $601, $602, $603, $604, $605, $606, $607, $608, $609, $610, $611, $612, $613, $614, $615, $616, $617, $618, $619, $620, $621, $622, $623, $624, $625, $626, $627, $628, $629, $630, $631, $632, $633, $634, $635, $636, $637, $638, $639, $640, $641, $642, $643, $644, $645, $646, $647, $648, $649, $650, $651, $652, $653, $654, $655, $656, $657, $658, $659, $660, $661, $662, $663, $664, $665, $666, $667, $668, $669, $670, $671, $672, $673, $674, $675, $676, $677, $678, $679, $680, $681, $682, $683, $684, $685, $686, $687, $688, $689, $690, $691, $692, $693, $694, $695, $696, $697, $698, $699, $700, $701, $702, $703, $704, $705, $706, $707, $708, $709, $710, $711, $712, $713, $714, $715, $716, $717, $718, $719, $720, $721, $722, $723, $724, $725, $726, $727, $728, $729, $730, $731, $732, $733, $734, $735, $736, $737, $738, $739, $740, $741, $742, $743, $744, $745, $746, $747, $748, $749, $750, $751, $752, $753, $754, $755, $756, $757, $758, $759, $760, $761, $762, $763, $764, $765, $766, $767, $768, $769, $770, $771, $772, $773, $774, $775, $776, $777, $778, $779, $780, $781, $782, $783, $784, $785, $786, $787, $788, $789, $790, $791, $792, $793, $794, $795, $796, $797, $798, $799, $800, $801, $802, $803, $804, $805, $806, $807, $808, $809, $810, $811, $812, $813, $814, $815, $816, $817, $818, $819, $820, $821, $822, $823, $824, $825, $826, $827, $828, $829, $830, $831, $832, $833, $834, $835, $836, $837, $838, $839, $840, $841, $842, $843, $844, $845, $846, $847, $848, $849, $850, $851, $852, $853, $854, $855, $856, $857, $858, $859, $860, $861, $862, $863, $864, $865, $866, $867, $868, $869, $870, $871, $872, $873, $874, $875, $876, $877, $878, $879, $880, $881, $882, $883, $884, $885, $886, $887, $888, $889, $890, $891, $892, $893, $894, $895, $896, $897, $898, $899, $900, $901, $902, $903, $904, $905, $906, $907, $908, $909, $910, $911, $912, $913, $914, $915, $916, $917, $918, $919, $920, $921, $922, $923, $924, $925, $926, $927, $928, $929, $930, $931, $932, $933, $934, $935, $936, $937, $938, $939, $940, $941, $942, $943, $944, $945, $946, $947, $948, $949, $950, $951, $952, $953, $954, $955, $956, $957, $958, $959, $960, $961, $962, $963, $964, $965, $966, $967, $968, $969, $970, $971, $972, $973, $974, $975, $976, $977, $978, $979, $980, $981, $982, $983, $984, $985, $986, $987, $988, $989, $990, $991, $992, $993, $994, $995, $996, $997, $998, $999, $1000, $1001, $1002, $1003, $1004, $1005, $1006, $1007, $1008, $1009, $1010, $1011, $1012, $1013, $1014, $1015, $1016, $1017, $1018, $1019, $1020, $1021, $1022, $1023, $1024, $1025, $1026, $1027, $1028, $1029, $1030, $1031, $1032, $1033, $1034, $1035, $1036, $1037, $1038, $1039, $1040, $1041, $1042, $1043, $1044, $1045, $1046, $1047, $1048, $1049, $1050, $1051, $1052, $1053, $1054, $1055, $1056, $1057, $1058, $1059, $1060, $1061, $1062, $1063, $1064, $1065, $1066, $1067, $1068, $1069, $1070, $1071, $1072, $1073, $1074, $1075, $1076, $1077, $1078, $1079, $1080, $1081, $1082, $1083, $1084, $1085, $1086, $1087, $1088, $1089, $1090, $1091, $1092, $1093, $1094, $1095, $1096, $1097, $1098, $1099, $1100, $1101, $1102, $1103, $1104, $1105, $1106, $1107, $1108, $1109, $1110, $1111, $1112, $1113, $1114, $1115, $1116, $1117, $1118, $1119, $1120, $1121, $1122, $1123, $1124, $1125, $1126, $1127, $1128, $1129, $1130, $1131, $1132, $1133, $1134, $1135, $1136, $1137, $1138, $1139, $1140, $1141, $1142, $1143, $1144, $1145, $1146, $1147, $1148, $1149, $1150, $1151, $1152, $1153, $1154, $1155, $1156, $1157, $1158, $1159, $1160, $1161, $1162, $1163, $1164, $1165, $1166, $1167, $1168, $1169, $1170, $1171, $1172, $1173, $1174, $1175, $1176, $1177, $1178, $1179, $1180, $1181, $1182, $1183, $1184, $1185, $1186, $1187, $1188, $1189, $1190, $1191, $1192, $1193, $1194, $1195, $1196, $1197, $1198, $1199, $1200, $1201, $1202, $1203, $1204, $1205, $1206, $1207, $1208, $1209, $1210, $1211, $1212, $1213, $1214, $1215, $1216, $1217, $1218, $1219, $1220, $1221, $1222, $1223, $1224, $1225, $1226, $1227, $1228, $1229, $1230, $1231, $1232, $1233, $1234, $1235, $1236, $1237, $1238, $1239, $1240, $1241, $1242, $1243, $1244, $1245, $1246, $1247, $1248, $1249, $1250, $1251, $1252, $1253, $1254, $1255, $1256, $1257, $1258, $1259, $1260, $1261, $1262, $1263, $1264, $1265, $1266, $1267, $1268, $1269, $1270, $1271, $1272, $1273, $1274, $1275, $1276, $1277, $1278, $1279, $1280, $1281, $1282, $1283, $1284, $1285, $1286, $1287, $1288, $1289, $1290, $1291, $1292, $1293, $1294, $1295, $1296, $1297, $1298, $1299, $1300, $1301, $1302, $1303, $1304, $1305, $1306, $1307, $1308, $1309, $1310, $1311, $1312, $1313, $1314, $1315, $1316, $1317, $1318, $1319, $1320, $1321, $1322, $1323, $1324, $1325, $1326, $1327, $1328, $1329, $1330, $1331, $1332, $1333, $1334, $1335, $1336, $1337, $1338, $1339, $1340, $1341, $1342, $1343, $1344, $1345, $1346, $1347, $1348, $1349, $1350, $1351, $1352, $1353, $1354, $1355, $1356, $1357, $1358, $1359, $1360, $1361, $1362, $1363, $1364, $1365, $1366, $1367, $1368, $1369, $1370, $1371, $1372, $1373, $1374, $1375, $1376, $1377, $1378, $1379, $1380, $1381, $1382, $1383, $1384, $1385, $1386, $1387, $1388, $1389, $1390, $1391, $1392, $1393, $1394, $1395, $1396, $1397, $1398, $1399, $1400, $1401, $1402, $1403, $1404, $1405, $1406, $1407, $1408, $1409, $1410, $1411, $1412, $1413, $1414, $1415, $1416, $1417, $1418, $1419, $1420, $1421, $1422, $1423, $1424, $1425, $1426, $1427, $1428, $1429, $1430, $1431, $1432, $1433, $1434, $1435, $1436, $1437, $1438, $1439, $1440, $1441, $1442, $1443, $1444, $1445, $1446, $1447, $1448, $1449, $1450, $1451, $1452, $1453, $1454, $1455, $1456, $1457, $1458, $1459, $1460, $1461, $1462, $1463, $1464
immich_postgres          | , $1465, $1466, $1467, $1468, $1469, $1470, $1471, $1472, $1473, $1474, $1475, $1476, $1477, $1478, $1479, $1480, $1481, $1482, $1483, $1484, $1485, $1486, $1487, $1488, $1489, $1490, $1491, $1492, $1493, $1494, $1495, $1496, $1497, $1498, $1499, $1500, $1501, $1502, $1503, $1504, $1505, $1506, $1507, $1508, $1509, $1510, $1511, $1512, $1513, $1514, $1515, $1516, $1517, $1518, $1519, $1520, $1521, $1522, $1523, $1524, $1525, $1526, $1527, $1528, $1529, $1530, $1531, $1532, $1533, $1534, $1535, $1536, $1537, $1538, $1539, $1540, $1541, $1542, $1543, $1544, $1545, $1546, $1547, $1548, $1549, $1550, $1551, $1552, $1553, $1554, $1555, $1556, $1557, $1558, $1559, $1560, $1561, $1562, $1563, $1564, $1565, $1566, $1567, $1568, $1569, $1570, $1571))

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

1.74.0

Version of Immich Mobile App

N/A

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    user: 1000:1001
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    user: 1000:1001
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - /disks/storage/immich/volumes/geocoding:/usr/src/app/.reverse-geocoding-dump
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    user: 1000:1001
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /disks/storage/immich/volumes/model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    volumes:
      - /disks/storage/immich/volumes/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    user: 1000:1001
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /disks/storage/immich/volumes/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

PUID=1000
GUID=1001

# The location where your uploaded files are stored
UPLOAD_LOCATION=/disks/storage/immich/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=<>
DB_PASSWORD=<>

TZ=America/New_York

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

### Reproduction steps

```bash
Run extract metadata

Additional information

No response

Originally created by @eygraber on GitHub (Aug 23, 2023). ### The bug I noticed that there were a few duplicate images in my library that were exactly the same except that one looked like it had reverse geocoding information added to it. I though that running the extract metadata job would fix this, so I ran it on All. At first there were a bunch of lines in the logs like: ``` immich_microservices | [Nest] 7 - 08/23/2023, 12:20:49 AM LOG [MetadataExtractionProcessor] Already extracted asset upload/library/admin/2020/2020-10-07/PXL_20201007_161535958.mp4. ``` Then there were a bunch of errors that looked like: ``` immich_microservices | [Nest] 7 - 08/23/2023, 12:23:02 AM ERROR [MetadataExtractionProcessor] Failed to extract live photo upload/library/admin/2018/2018-08-12/MVIMG_20180812_191539.jpg: QueryFailedError: duplicate key value violates unique constraint "UQ_userid_checksum" immich_postgres | 2023-08-23 00:23:02.547 EDT [214] ERROR: duplicate key value violates unique constraint "UQ_userid_checksum" immich_postgres | 2023-08-23 00:23:02.547 EDT [214] DETAIL: Key ("ownerId", checksum)=(d15dd236-5058-43b6-99da-b0c02a2db9d9, \x0d994499ab2f0ddd6906983bd5381f6a2d5680cd) already exists. immich_postgres | 2023-08-23 00:23:02.547 EDT [214] STATEMENT: INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "deviceId", "type", "originalPath", "resizePath", "webpPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "fileCreatedAt", "fileModifiedAt", "isFavorite", "isArchived", "isReadOnly", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, DEFAULT, $10, $11, $12, $13, $14, $15, $16, $17, DEFAULT, $18, $19) RETURNING "id", "webpPath", "encodedVideoPath", "createdAt", "updatedAt", "isFavorite", "isArchived", "isReadOnly", "isVisible" ``` and ``` immich_microservices | [Nest] 7 - 08/23/2023, 12:23:19 AM WARN [MetadataExtractionProcessor] The exifData parsing failed due to Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG for asset 45496791-74e5-4978-857b-6463fd3974fe at upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG immich_microservices | [Nest] 7 - 08/23/2023, 12:23:19 AM WARN [MetadataExtractionProcessor] Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG immich_microservices | at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) immich_microservices | at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) immich_microservices | [Nest] 7 - 08/23/2023, 12:23:19 AM ERROR [JobService] Unable to run job handler: Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG' immich_microservices | [Nest] 7 - 08/23/2023, 12:23:19 AM ERROR [JobService] Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG' immich_microservices | at Object.statSync (node:fs:1690:3) immich_microservices | at MetadataExtractionProcessor.handlePhotoMetadataExtraction (/usr/src/app/dist/microservices/processors/metadata-extraction.processor.js:272:45) immich_microservices | at async /usr/src/app/dist/domain/job/job.service.js:104:37 immich_microservices | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:342:28) immich_microservices | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:525:24) immich_microservices | [Nest] 7 - 08/23/2023, 12:23:19 AM ERROR [JobService] Object: immich_microservices | { immich_microservices | "id": "45496791-74e5-4978-857b-6463fd3974fe" immich_microservices | } immich_microservices | immich_microservices | ``` After running the job for a while, this gets printed: <details> <summary>microservices heap out of memory</summary> ``` immich_microservices | Request #1692760955977: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded" immich_microservices | Request #1692760955977: Sleeping for 4s and then retrying request... immich_typesense | I20230823 03:23:01.688388 730 log.cpp:523] close a full segment. Current first_index: 425757 last_index: 425765 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000425757_00000000000000425765 immich_typesense | I20230823 03:23:01.924839 730 log.cpp:537] Renamed `/data/state/log/log_inprogress_00000000000000425757' to `/data/state/log/log_00000000000000425757_00000000000000425765' immich_typesense | I20230823 03:23:01.924924 730 log.cpp:108] Created new segment `/data/state/log/log_inprogress_00000000000000425766' with fd=174 immich_typesense | I20230823 03:23:02.298883 645 raft_server.cpp:546] Term: 21, last_index index: 425767, committed_index: 425766, known_applied_index: 425766, applying_index: 0, queued_writes: 0, pending_queue_size: 205, local_sequence: 33327184 immich_typesense | I20230823 03:23:02.298957 736 raft_server.h:60] Peer refresh succeeded! immich_microservices | immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | immich_microservices | [7:0x595e0840000] 126213 ms: Scavenge 4073.0 (4129.7) -> 4066.3 (4130.2) MB, 3.9 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | [7:0x595e0840000] 126222 ms: Scavenge 4073.5 (4130.2) -> 4067.0 (4130.7) MB, 3.1 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | [7:0x595e0840000] 126590 ms: Scavenge 4074.1 (4130.7) -> 4067.4 (4147.2) MB, 362.2 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | immich_microservices | immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | immich_microservices | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory immich_microservices | 1: 0xb83f50 node::Abort() [immich_microservices] immich_microservices | 2: 0xa94834 [immich_microservices] immich_microservices | 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 5: 0xf42265 [immich_microservices] immich_microservices | 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] immich_microservices | 7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] immich_microservices | 10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] immich_microservices | 11: 0x17035b9 [immich_microservices] ``` </details> <details> <summary>postgres giant SELECT</summary> ``` immich_postgres | 2023-08-22 23:23:02.565 EDT [1083] LOG: could not send data to client: Connection reset by peer immich_postgres | 2023-08-22 23:23:02.565 EDT [1083] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."thumbhash" AS "AssetEntity_thumbhash", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."isReadOnly" AS "AssetEntity_isReadOnly", "AssetEntity"."checksum" AS "AssetEntity_checksum", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity"."sidecarPath" AS "AssetEntity_sidecarPath", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."projectionType" AS "AssetEntity__AssetEntity_exifInfo_projectionType", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "AssetEntity__AssetEntity_faces"."imageWidth" AS "AssetEntity__AssetEntity_faces_imageWidth", "AssetEntity__AssetEntity_faces"."imageHeight" AS "AssetEntity__AssetEntity_faces_imageHeight", "AssetEntity__AssetEntity_faces"."boundingBoxX1" AS "AssetEntity__AssetEntity_faces_boundingBoxX1", "AssetEntity__AssetEntity_faces"."boundingBoxY1" AS "AssetEntity__AssetEntity_faces_boundingBoxY1", "AssetEntity__AssetEntity_faces"."boundingBoxX2" AS "AssetEntity__AssetEntity_faces_boundingBoxX2", "AssetEntity__AssetEntity_faces"."boundingBoxY2" AS "AssetEntity__AssetEntity_faces_boundingBoxY2", "8258e303a73a72cf6abb13d73fb592dde0d68280"."id" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_id", "8258e303a73a72cf6abb13d73fb592dde0d68280"."createdAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_createdAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."updatedAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_updatedAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."ownerId" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_ownerId", "8258e303a73a72cf6abb13d73fb592dde0d68280"."name" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_name", "8258e303a73a72cf6abb13d73fb592dde0d68280"."birthDate" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_birthDate", "8258e303a73a72cf6abb13d73fb592dde0d68280"."thumbnailPath" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_thumbnailPath", "8258e303a73a72cf6abb13d73fb592dde0d68280"."isHidden" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_isHidden" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "8258e303a73a72cf6abb13d73fb592dde0d68280" ON "8258e303a73a72cf6abb13d73fb592dde0d68280"."id"="AssetEntity__AssetEntity_faces"."personId" WHERE ("AssetEntity"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $50, $51, $52, $53, $54, $55, $56, $57, $58, $59, $60, $61, $62, $63, $64, $65, $66, $67, $68, $69, $70, $71, $72, $73, $74, $75, $76, $77, $78, $79, $80, $81, $82, $83, $84, $85, $86, $87, $88, $89, $90, $91, $92, $93, $94, $95, $96, $97, $98, $99, $100, $101, $102, $103, $104, $105, $106, $107, $108, $109, $110, $111, $112, $113, $114, $115, $116, $117, $118, $119, $120, $121, $122, $123, $124, $125, $126, $127, $128, $129, $130, $131, $132, $133, $134, $135, $136, $137, $138, $139, $140, $141, $142, $143, $144, $145, $146, $147, $148, $149, $150, $151, $152, $153, $154, $155, $156, $157, $158, $159, $160, $161, $162, $163, $164, $165, $166, $167, $168, $169, $170, $171, $172, $173, $174, $175, $176, $177, $178, $179, $180, $181, $182, $183, $184, $185, $186, $187, $188, $189, $190, $191, $192, $193, $194, $195, $196, $197, $198, $199, $200, $201, $202, $203, $204, $205, $206, $207, $208, $209, $210, $211, $212, $213, $214, $215, $216, $217, $218, $219, $220, $221, $222, $223, $224, $225, $226, $227, $228, $229, $230, $231, $232, $233, $234, $235, $236, $237, $238, $239, $240, $241, $242, $243, $244, $245, $246, $247, $248, $249, $250, $251, $252, $253, $254, $255, $256, $257, $258, $259, $260, $261, $262, $263, $264, $265, $266, $267, $268, $269, $270, $271, $272, $273, $274, $275, $276, $277, $278, $279, $280, $281, $282, $283, $284, $285, $286, $287, $288, $289, $290, $291, $292, $293, $294, $295, $296, $297, $298, $299, $300, $301, $302, $303, $304, $305, $306, $307, $308, $309, $310, $311, $312, $313, $314, $315, $316, $317, $318, $319, $320, $321, $322, $323, $324, $325, $326, $327, $328, $329, $330, $331, $332, $333, $334, $335, $336, $337, $338, $339, $340, $341, $342, $343, $344, $345, $346, $347, $348, $349, $350, $351, $352, $353, $354, $355, $356, $357, $358, $359, $360, $361, $362, $363, $364, $365, $366, $367, $368, $369, $370, $371, $372, $373, $374, $375, $376, $377, $378, $379, $380, $381, $382, $383, $384, $385, $386, $387, $388, $389, $390, $391, $392, $393, $394, $395, $396, $397, $398, $399, $400, $401, $402, $403, $404, $405, $406, $407, $408, $409, $410, $411, $412, $413, $414, $415, $416, $417, $418, $419, $420, $421, $422, $423, $424, $425, $426, $427, $428, $429, $430, $431, $432, $433, $434, $435, $436, $437, $438, $439, $440, $441, $442, $443, $444, $445, $446, $447, $448, $449, $450, $451, $452, $453, $454, $455, $456, $457, $458, $459, $460, $461, $462, $463, $464, $465, $466, $467, $468, $469, $470, $471, $472, $473, $474, $475, $476, $477, $478, $479, $480, $481, $482, $483, $484, $485, $486, $487, $488, $489, $490, $491, $492, $493, $494, $495, $496, $497, $498, $499, $500, $501, $502, $503, $504, $505, $506, $507, $508, $509, $510, $511, $512, $513, $514, $515, $516, $517, $518, $519, $520, $521, $522, $523, $524, $525, $526, $527, $528, $529, $530, $531, $532, $533, $534, $535, $536, $537, $538, $539, $540, $541, $542, $543, $544, $545, $546, $547, $548, $549, $550, $551, $552, $553, $554, $555, $556, $557, $558, $559, $560, $561, $562, $563, $564, $565, $566, $567, $568, $569, $570, $571, $572, $573, $574, $575, $576, $577, $578, $579, $580, $581, $582, $583, $584, $585, $586, $587, $588, $589, $590, $591, $592, $593, $594, $595, $596, $597, $598, $599, $600, $601, $602, $603, $604, $605, $606, $607, $608, $609, $610, $611, $612, $613, $614, $615, $616, $617, $618, $619, $620, $621, $622, $623, $624, $625, $626, $627, $628, $629, $630, $631, $632, $633, $634, $635, $636, $637, $638, $639, $640, $641, $642, $643, $644, $645, $646, $647, $648, $649, $650, $651, $652, $653, $654, $655, $656, $657, $658, $659, $660, $661, $662, $663, $664, $665, $666, $667, $668, $669, $670, $671, $672, $673, $674, $675, $676, $677, $678, $679, $680, $681, $682, $683, $684, $685, $686, $687, $688, $689, $690, $691, $692, $693, $694, $695, $696, $697, $698, $699, $700, $701, $702, $703, $704, $705, $706, $707, $708, $709, $710, $711, $712, $713, $714, $715, $716, $717, $718, $719, $720, $721, $722, $723, $724, $725, $726, $727, $728, $729, $730, $731, $732, $733, $734, $735, $736, $737, $738, $739, $740, $741, $742, $743, $744, $745, $746, $747, $748, $749, $750, $751, $752, $753, $754, $755, $756, $757, $758, $759, $760, $761, $762, $763, $764, $765, $766, $767, $768, $769, $770, $771, $772, $773, $774, $775, $776, $777, $778, $779, $780, $781, $782, $783, $784, $785, $786, $787, $788, $789, $790, $791, $792, $793, $794, $795, $796, $797, $798, $799, $800, $801, $802, $803, $804, $805, $806, $807, $808, $809, $810, $811, $812, $813, $814, $815, $816, $817, $818, $819, $820, $821, $822, $823, $824, $825, $826, $827, $828, $829, $830, $831, $832, $833, $834, $835, $836, $837, $838, $839, $840, $841, $842, $843, $844, $845, $846, $847, $848, $849, $850, $851, $852, $853, $854, $855, $856, $857, $858, $859, $860, $861, $862, $863, $864, $865, $866, $867, $868, $869, $870, $871, $872, $873, $874, $875, $876, $877, $878, $879, $880, $881, $882, $883, $884, $885, $886, $887, $888, $889, $890, $891, $892, $893, $894, $895, $896, $897, $898, $899, $900, $901, $902, $903, $904, $905, $906, $907, $908, $909, $910, $911, $912, $913, $914, $915, $916, $917, $918, $919, $920, $921, $922, $923, $924, $925, $926, $927, $928, $929, $930, $931, $932, $933, $934, $935, $936, $937, $938, $939, $940, $941, $942, $943, $944, $945, $946, $947, $948, $949, $950, $951, $952, $953, $954, $955, $956, $957, $958, $959, $960, $961, $962, $963, $964, $965, $966, $967, $968, $969, $970, $971, $972, $973, $974, $975, $976, $977, $978, $979, $980, $981, $982, $983, $984, $985, $986, $987, $988, $989, $990, $991, $992, $993, $994, $995, $996, $997, $998, $999, $1000, $1001, $1002, $1003, $1004, $1005, $1006, $1007, $1008, $1009, $1010, $1011, $1012, $1013, $1014, $1015, $1016, $1017, $1018, $1019, $1020, $1021, $1022, $1023, $1024, $1025, $1026, $1027, $1028, $1029, $1030, $1031, $1032, $1033, $1034, $1035, $1036, $1037, $1038, $1039, $1040, $1041, $1042, $1043, $1044, $1045, $1046, $1047, $1048, $1049, $1050, $1051, $1052, $1053, $1054, $1055, $1056, $1057, $1058, $1059, $1060, $1061, $1062, $1063, $1064, $1065, $1066, $1067, $1068, $1069, $1070, $1071, $1072, $1073, $1074, $1075, $1076, $1077, $1078, $1079, $1080, $1081, $1082, $1083, $1084, $1085, $1086, $1087, $1088, $1089, $1090, $1091, $1092, $1093, $1094, $1095, $1096, $1097, $1098, $1099, $1100, $1101, $1102, $1103, $1104, $1105, $1106, $1107, $1108, $1109, $1110, $1111, $1112, $1113, $1114, $1115, $1116, $1117, $1118, $1119, $1120, $1121, $1122, $1123, $1124, $1125, $1126, $1127, $1128, $1129, $1130, $1131, $1132, $1133, $1134, $1135, $1136, $1137, $1138, $1139, $1140, $1141, $1142, $1143, $1144, $1145, $1146, $1147, $1148, $1149, $1150, $1151, $1152, $1153, $1154, $1155, $1156, $1157, $1158, $1159, $1160, $1161, $1162, $1163, $1164, $1165, $1166, $1167, $1168, $1169, $1170, $1171, $1172, $1173, $1174, $1175, $1176, $1177, $1178, $1179, $1180, $1181, $1182, $1183, $1184, $1185, $1186, $1187, $1188, $1189, $1190, $1191, $1192, $1193, $1194, $1195, $1196, $1197, $1198, $1199, $1200, $1201, $1202, $1203, $1204, $1205, $1206, $1207, $1208, $1209, $1210, $1211, $1212, $1213, $1214, $1215, $1216, $1217, $1218, $1219, $1220, $1221, $1222, $1223, $1224, $1225, $1226, $1227, $1228, $1229, $1230, $1231, $1232, $1233, $1234, $1235, $1236, $1237, $1238, $1239, $1240, $1241, $1242, $1243, $1244, $1245, $1246, $1247, $1248, $1249, $1250, $1251, $1252, $1253, $1254, $1255, $1256, $1257, $1258, $1259, $1260, $1261, $1262, $1263, $1264, $1265, $1266, $1267, $1268, $1269, $1270, $1271, $1272, $1273, $1274, $1275, $1276, $1277, $1278, $1279, $1280, $1281, $1282, $1283, $1284, $1285, $1286, $1287, $1288, $1289, $1290, $1291, $1292, $1293, $1294, $1295, $1296, $1297, $1298, $1299, $1300, $1301, $1302, $1303, $1304, $1305, $1306, $1307, $1308, $1309, $1310, $1311, $1312, $1313, $1314, $1315, $1316, $1317, $1318, $1319, $1320, $1321, $1322, $1323, $1324, $1325, $1326, $1327, $1328, $1329, $1330, $1331, $1332, $1333, $1334, $1335, $1336, $1337, $1338, $1339, $1340, $1341, $1342, $1343, $1344, $1345, $1346, $1347, $1348, $1349, $1350, $1351, $1352, $1353, $1354, $1355, $1356, $1357, $1358, $1359, $1360, $1361, $1362, $1363, $1364, $1365, $1366, $1367, $1368, $1369, $1370, $1371, $1372, $1373, $1374, $1375, $1376, $1377, $1378, $1379, $1380, $1381, $1382, $1383, $1384, $1385, $1386, $1387, $1388, $1389, $1390, $1391, $1392, $1393, $1394, $1395, $1396, $1397, $1398, $1399, $1400, $1401, $1402, $1403, $1404, $1405, $1406, $1407, $1408, $1409, $1410, $1411, $1412, $1413, $1414, $1415, $1416, $1417, $1418, $1419, $1420, $1421, $1422, $1423, $1424, $1425, $1426, $1427, $1428, $1429, $1430, $1431, $1432, $1433, $1434, $1435, $1436, $1437, $1438, $1439, $1440, $1441, $1442, $1443, $1444, $1445, $1446, $1447, $1448, $1449, $1450, $1451, $1452, $1453, $1454, $1455, $1456, $1457, $1458, $1459, $1460, $1461, $1462, $1463, $1464 immich_postgres | , $1465, $1466, $1467, $1468, $1469, $1470, $1471, $1472, $1473, $1474, $1475, $1476, $1477, $1478, $1479, $1480, $1481, $1482, $1483, $1484, $1485, $1486, $1487, $1488, $1489, $1490, $1491, $1492, $1493, $1494, $1495, $1496, $1497, $1498, $1499, $1500, $1501, $1502, $1503, $1504, $1505, $1506, $1507, $1508, $1509, $1510, $1511, $1512, $1513, $1514, $1515, $1516, $1517, $1518, $1519, $1520, $1521, $1522, $1523, $1524, $1525, $1526, $1527, $1528, $1529, $1530, $1531, $1532, $1533, $1534, $1535, $1536, $1537, $1538, $1539, $1540, $1541, $1542, $1543, $1544, $1545, $1546, $1547, $1548, $1549, $1550, $1551, $1552, $1553, $1554, $1555, $1556, $1557, $1558, $1559, $1560, $1561, $1562, $1563, $1564, $1565, $1566, $1567, $1568, $1569, $1570, $1571)) ``` </details> ### The OS that Immich Server is running on Ubuntu 22.04 ### Version of Immich Server 1.74.0 ### Version of Immich Mobile App N/A ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: user: 1000:1001 container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env depends_on: - redis - database - typesense restart: always immich-microservices: user: 1000:1001 container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - /disks/storage/immich/volumes/geocoding:/usr/src/app/.reverse-geocoding-dump - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env depends_on: - redis - database - typesense restart: always immich-machine-learning: user: 1000:1001 container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - /disks/storage/immich/volumes/model-cache:/cache env_file: - .env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - .env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data volumes: - /disks/storage/immich/volumes/tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: user: 1000:1001 container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - /disks/storage/immich/volumes/pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 depends_on: - immich-server - immich-web restart: always ``` ### Your .env content ```Shell PUID=1000 GUID=1001 # The location where your uploaded files are stored UPLOAD_LOCATION=/disks/storage/immich/library # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=<> DB_PASSWORD=<> TZ=America/New_York # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ``` ### Reproduction steps ```bash Run extract metadata ``` ### Additional information _No response_

OVERLORD closed this issue 2026-02-05 01:06:25 +03:00

OVERLORD commented 2026-02-05 01:06:29 +03:00

Author

Owner

Copy Link

@mertalev commented on GitHub (Aug 23, 2023):

Seems like a memory leak in the handler for Pixel motion photos. My hunch is that the buffers for extracting videos are using too much memory, possibly not being freed if there's an error, or not being freed quickly enough anyway.

@mertalev commented on GitHub (Aug 23, 2023): Seems like a memory leak in the handler for Pixel motion photos. My hunch is that the buffers for extracting videos are using too much memory, possibly not being freed if there's an error, or not being freed quickly enough anyway.

OVERLORD commented 2026-02-05 01:06:31 +03:00

Author

Owner

Copy Link

@jrasm91 commented on GitHub (Aug 23, 2023):

I'm thinking it is recursively creating assets 😅

@jrasm91 commented on GitHub (Aug 23, 2023): I'm thinking it is recursively creating assets 😅

OVERLORD commented 2026-02-05 01:06:33 +03:00

Author

Owner

Copy Link

@francesquini commented on GitHub (Oct 3, 2023):

Not sure if the problem is exactly the same, but I'm getting a similar error followed by a core dump:

immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout
immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
immich_microservices     |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_microservices     |     at Object.onceWrapper (node:events:628:28)
immich_microservices     |     at Connection.emit (node:events:514:28)
immich_microservices     |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_microservices     |     at Socket.emit (node:events:514:28)
immich_microservices     |     at TCP.<anonymous> (node:net:323:12)
immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Object:
immich_microservices     | {}
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- Last few GCs --->
immich_microservices     | 
immich_microservices     | [8:0x5f254840000]   329587 ms: Scavenge (reduce) 1941.5 (1983.0) -> 1940.7 (1983.0) MB, 15.1 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | [8:0x5f254840000]   329612 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.7 (1983.0) MB, 19.8 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | [8:0x5f254840000]   329623 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.8 (1983.2) MB, 6.8 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- JS stacktrace --->
immich_microservices     | 
immich_microservices     | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices     |  1: 0xb83f50 node::Abort() [immich_microservices]
immich_microservices     |  2: 0xa94834  [immich_microservices]
immich_microservices     |  3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  5: 0xf42265  [immich_microservices]
immich_microservices     |  6: 0xf43168 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
immich_microservices     |  7: 0xf53673  [immich_microservices]
immich_microservices     |  8: 0xf544e8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
immich_microservices     |  9: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     | 10: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     | 11: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
immich_microservices     | 12: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
immich_microservices     | 13: 0x17035b9  [immich_microservices]
immich_postgres          | 2023-10-03 19:25:46.778 UTC [624] LOG:  could not send data to client: Connection reset by peer
immich_postgres          | 2023-10-03 19:25:46.778 UTC [624] STATEMENT:  SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId"
immich_postgres          | 2023-10-03 19:25:46.779 UTC [624] FATAL:  connection to client lost
immich_postgres          | 2023-10-03 19:25:46.779 UTC [624] STATEMENT:  SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId"
immich_microservices     | Aborted (core dumped)
immich_microservices exited with code 0

This is consistent and happens at every execution of the migration job.

@francesquini commented on GitHub (Oct 3, 2023): Not sure if the problem is exactly the same, but I'm getting a similar error followed by a core dump: ``` immich_microservices | [Nest] 8 - 10/03/2023, 7:23:47 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout immich_microservices | [Nest] 8 - 10/03/2023, 7:23:47 PM ERROR [JobService] Error: Connection terminated due to connection timeout immich_microservices | at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) immich_microservices | at Object.onceWrapper (node:events:628:28) immich_microservices | at Connection.emit (node:events:514:28) immich_microservices | at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) immich_microservices | at Socket.emit (node:events:514:28) immich_microservices | at TCP.<anonymous> (node:net:323:12) immich_microservices | [Nest] 8 - 10/03/2023, 7:23:47 PM ERROR [JobService] Object: immich_microservices | {} immich_microservices | immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | [8:0x5f254840000] 329587 ms: Scavenge (reduce) 1941.5 (1983.0) -> 1940.7 (1983.0) MB, 15.1 / 0.0 ms (average mu = 0.261, current mu = 0.209) allocation failure; immich_microservices | [8:0x5f254840000] 329612 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.7 (1983.0) MB, 19.8 / 0.0 ms (average mu = 0.261, current mu = 0.209) allocation failure; immich_microservices | [8:0x5f254840000] 329623 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.8 (1983.2) MB, 6.8 / 0.0 ms (average mu = 0.261, current mu = 0.209) allocation failure; immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory immich_microservices | 1: 0xb83f50 node::Abort() [immich_microservices] immich_microservices | 2: 0xa94834 [immich_microservices] immich_microservices | 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 5: 0xf42265 [immich_microservices] immich_microservices | 6: 0xf43168 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] immich_microservices | 7: 0xf53673 [immich_microservices] immich_microservices | 8: 0xf544e8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] immich_microservices | 9: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 10: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 11: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] immich_microservices | 12: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] immich_microservices | 13: 0x17035b9 [immich_microservices] immich_postgres | 2023-10-03 19:25:46.778 UTC [624] LOG: could not send data to client: Connection reset by peer immich_postgres | 2023-10-03 19:25:46.778 UTC [624] STATEMENT: SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId" immich_postgres | 2023-10-03 19:25:46.779 UTC [624] FATAL: connection to client lost immich_postgres | 2023-10-03 19:25:46.779 UTC [624] STATEMENT: SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId" immich_microservices | Aborted (core dumped) immich_microservices exited with code 0 ``` This is consistent and happens at every execution of the migration job.

OVERLORD commented 2026-02-05 01:06:36 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 11, 2023):

I got the same issues in k3s for the microservices parts for truechart

[Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:18.789042417+02:00 [Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:18.789057486+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:18.789069085+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:18.789114401+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:18.789133310+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:18.789144668+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:18.789159909+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:18.789170851+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:18.789213775+02:00 [Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] Object:
2023-10-11T19:13:18.789226330+02:00 {
2023-10-11T19:13:18.789238126+02:00   "id": "400de142-13c7-4424-9fa6-cce0e10d7a73"
2023-10-11T19:13:18.789249081+02:00 }
2023-10-11T19:13:18.789259087+02:00 
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 5fe9c003-3506-44d4-9264-16e1266abdc9
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset db23aeb2-e35f-4c9e-8bc2-0af4b4476bc1
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 8cfb274e-3c99-45f6-9678-68d84aa3944c
[Nest] 7  - 10/11/2023, 7:13:20 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset b84207e4-ab96-41a0-92c6-855e19fbf857
[Nest] 7  - 10/11/2023, 7:13:21 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset fba4826a-11e6-493b-ab27-6a83ad3d536e
[Nest] 7  - 10/11/2023, 7:13:21 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset a988e32a-84d9-4cfb-9610-79220147de94
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout of 10000ms exceeded
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Error: timeout of 10000ms exceeded
    at createError (/usr/src/app/node_modules/typesense/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/usr/src/app/node_modules/typesense/node_modules/axios/lib/adapters/http.js:369:16)
    at RedirectableRequest.emit (node:events:514:28)
    at Timeout.<anonymous> (/usr/src/app/node_modules/follow-redirects/index.js:169:12)
2023-10-11T19:13:21.786904883+02:00     at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Object:
{
  "id": "83b72b63-b869-40f0-80f4-d1aea477b1d5"
}
2023-10-11T19:13:21.786923019+02:00 
[Nest] 7  - 10/11/2023, 7:13:24 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset bd25f755-3da6-4a76-ac31-ca58581dc380
[Nest] 7  - 10/11/2023, 7:13:27 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 7ba9f06f-b342-4dfe-bbf1-c77217894f27
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:27.613029534+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:27.613045059+02:00   "id": "ea8e152f-42b6-43d2-9ad8-a595e6b61798"
2023-10-11T19:13:27.613048055+02:00 }
2023-10-11T19:13:27.613050837+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
{
  "id": "a56c1370-5bd1-450b-8de1-e4da00e79672"
}

[Nest] 7  - 10/11/2023, 7:13:28 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 2b794421-20c3-40b6-a98b-8d6a63e6fe32
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.431743273+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.431746795+02:00     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.431749391+02:00     at runNextTicks (node:internal/process/task_queues:64:3)
2023-10-11T19:13:28.431752058+02:00     at process.processImmediate (node:internal/timers:449:9)
2023-10-11T19:13:28.431754629+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.431757205+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.431759672+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.431762635+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.431765171+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.432117622+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.432129611+02:00 {
2023-10-11T19:13:28.432133520+02:00   "id": "e58d5d60-9ea3-44b9-a3c9-fe7defa6d867"
2023-10-11T19:13:28.432136263+02:00 }
2023-10-11T19:13:28.432138727+02:00 
2023-10-11T19:13:28.432719161+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
2023-10-11T19:13:28.432823783+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.432830041+02:00     at Object.fetch (node:internal/deps/undici/und
ici:13220:11)
2023-10-11T19:13:28.432833106+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.432835650+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.432838258+02:00     at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
2023-10-11T19:13:28.432840714+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.432843403+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.432845819+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.432936251+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.432941285+02:00 {
2023-10-11T19:13:28.432944105+02:00   "id": "7994bfd3-53ac-4f7e-9649-0829250f9d41"
2023-10-11T19:13:28.432946595+02:00 }
2023-10-11T19:13:28.432948911+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.579157643+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.579161423+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.579171899+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.579295756+02:00 {
  "id": "5b396809-39dd-4c09-a369-177bebd81a27"
2023-10-11T19:13:28.579302705+02:00 }

[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.624674626+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.624681721+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.624684526+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.624696932+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.624700461+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.624703093+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.624823264+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.624829223+02:00 {
2023-10-11T19:13:28.624832882+02:00   "id": "7120cb7b-6163-4dae-82bc-1ac29914b138"
2023-10-11T19:13:28.624835929+02:00 }
2023-10-11T19:13:28.624838648+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset cf4ec11f-c6d6-49c5-9064-58cb3d1be707
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.783314243+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.783350662+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.783363858+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.783387750+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:28.783424842+02:00   "id": "4589c464-8307-455e-83f6-e19b76a3a424"
}

[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:28.840767205+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.840794602+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.840808677+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.840820622+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.840831492+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:28.840842777+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.840854979+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.841024661+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.841045353+02:00 {
2023-10-11T19:13:28.841059213+02:00   "id": "e81f8a58-0e8a-491d-a9cc-f28bb4eea2b6"
2023-10-11T19:13:28.841070781+02:00 }
2023-10-11T19:13:28.841110845+02:00 
[Nest] 7  - 10/11/2023, 7:13:29 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 80a79908-c49b-487f-bead-a224b3453cd7
[Nest] 7  - 10/11/2023, 7:13:30 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 5b471cd1-3933-4ca4-ab60-229807e3e860
[Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-thumbhash-thumbnail): Error: Connection terminated due to connection timeout
[Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-10-11T19:13:31.482596099+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
2023-10-11T19:13:31.482603215+02:00     at Connection.emit (node:events:514:28)
2023-10-11T19:13:31.482606685+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-10-11T19:13:31.482609283+02:00     at Socket.emit (node:events:514:28)
2023-10-11T19:13:31.482611884+02:00     at TCP.<anonymous> (node:net:337:12)
2023-10-11T19:13:31.482651487+02:00 [Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:31.482660906+02:00   "id": "627e6394-f677-429b-b444-9399354d24b3"
2023-10-11T19:13:31.482663691+02:00 }

[Nest] 7  - 10/11/2023, 7:13:31 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 627e6394-f677-429b-b444-9399354d24b3
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:32.198116303+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] Object:
{
  "id": "3dd9c9a9-349e-4c1e-94c3-44d1b7076a29"
}

[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset f6dccf33-3230-4c2b-9031-b966c884d613
[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 024b537c-c30c-4e27-9234-8138f0338861
[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 288eb4c7-366d-4b0b-8ba0-a4f420a45882
[Nest] 7  - 10/11/2023, 7:13:33 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 986f85df-01c5-4e48-88c3-e3e84e353c1b
[Nest] 7  - 10/11/2023, 7:13:37 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 72935da0-66e4-46ef-8825-8c77d18c37a5
[Nest] 7  - 10/11/2023, 7:13:39 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 355746ab-8b70-49d1-af8a-90f9d7712b91
[Nest] 7  - 10/11/2023, 7:13:39 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 6b59b01c-aaa1-4f53-adc5-4a271ae15f50
[Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:40.431657438+02:00 [Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:40.431662824+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:40.431666226+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:40.431671922+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:40.431674737+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:40.431677687+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:40.431680727+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:40.431713510+02:00 [Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] Object:
2023-10-11T19:13:40.431719093+02:00 {
2023-10-11T19:13:40.431722445+02:00   "id": "641554c5-1502-4e1f-8559-591c844acce7"
2023-10-11T19:13:40.431725012+02:00 }
2023-10-11T19:13:40.431727377+02:00 
[Nest] 7  - 10/11/2023, 7:13:40 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 984e81dc-4b5d-4d89-89d5-afc48563b3d5
[Nest] 7  - 10/11/2023, 7:13:42 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset db3f355f-b4f3-481d-98b7-5f919261b7dd
[Nest] 7  - 10/11/2023, 7:13:42 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 56141756-dca1-4f43-917b-c0f0f2c01327
[Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
2023-10-11T19:13:44.972488668+02:00 [Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:44.972501987+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:44.972506083+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:44.972508814+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:44.972511586+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:44.972514865+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:44.972517493+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:44.972615981+02:00 [Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] Object:
2023-10-11T19:13:44.972622001+02:00 {
2023-10-11T19:13:44.972625531+02:00   "id": "62f8ee52-90b3-4251-a1fa-81b84436fde6"
2023-10-11T19:13:44.972628553+02:00 }
2023-10-11T19:13:44.972631259+02:00 
[Nest] 7  - 10/11/2023, 7:13:47 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset c27a8827-c65e-48c0-a7fc-4bece1c49ae1
/usr/src/app/dist/microservices/app.service.js:85
2023-10-11T19:13:48.529060637+02:00                 throw error;
2023-10-11T19:13:48.529065008+02:00                 ^
2023-10-11T19:13:48.529067863+02:00 
2023-10-11T19:13:48.529070587+02:00 Error: Error downloading GeoNames cities500 data: Error: aborted
2023-10-11T19:13:48.529092264+02:00     at PassThrough.<anonymous> (/usr/src/app/node_modules/local-reverse-geocoder/index.js:251:19)
2023-10-11T19:13:48.529098259+02:00     at PassThrough.emit (node:events:526:35)
2023-10-11T19:13:48.529100933+02:00     at emitErrorNT (node:internal/streams/destroy:151:8)
2023-10-11T19:13:48.529103442+02:00     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
2023-10-11T19:13:48.529108595+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2023-10-11T19:13:48.529110877+02:00 
2023-10-11T19:13:48.529113559+02:00 Node.js v20.8.0

@yodatak commented on GitHub (Oct 11, 2023): I got the same issues in k3s for the microservices parts for truechart ``` [Nest] 7 - 10/11/2023, 7:13:18 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed 2023-10-11T19:13:18.789042417+02:00 [Nest] 7 - 10/11/2023, 7:13:18 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:18.789057486+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:18.789069085+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-10-11T19:13:18.789114401+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:18.789133310+02:00 at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31) 2023-10-11T19:13:18.789144668+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:18.789159909+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:18.789170851+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:18.789213775+02:00 [Nest] 7 - 10/11/2023, 7:13:18 PM ERROR [JobService] Object: 2023-10-11T19:13:18.789226330+02:00 { 2023-10-11T19:13:18.789238126+02:00 "id": "400de142-13c7-4424-9fa6-cce0e10d7a73" 2023-10-11T19:13:18.789249081+02:00 } 2023-10-11T19:13:18.789259087+02:00 [Nest] 7 - 10/11/2023, 7:13:19 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 5fe9c003-3506-44d4-9264-16e1266abdc9 [Nest] 7 - 10/11/2023, 7:13:19 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset db23aeb2-e35f-4c9e-8bc2-0af4b4476bc1 [Nest] 7 - 10/11/2023, 7:13:19 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 8cfb274e-3c99-45f6-9678-68d84aa3944c [Nest] 7 - 10/11/2023, 7:13:20 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset b84207e4-ab96-41a0-92c6-855e19fbf857 [Nest] 7 - 10/11/2023, 7:13:21 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset fba4826a-11e6-493b-ab27-6a83ad3d536e [Nest] 7 - 10/11/2023, 7:13:21 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset a988e32a-84d9-4cfb-9610-79220147de94 [Nest] 7 - 10/11/2023, 7:13:21 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout of 10000ms exceeded [Nest] 7 - 10/11/2023, 7:13:21 PM ERROR [JobService] Error: timeout of 10000ms exceeded at createError (/usr/src/app/node_modules/typesense/node_modules/axios/lib/core/createError.js:16:15) at RedirectableRequest.handleRequestTimeout (/usr/src/app/node_modules/typesense/node_modules/axios/lib/adapters/http.js:369:16) at RedirectableRequest.emit (node:events:514:28) at Timeout.<anonymous> (/usr/src/app/node_modules/follow-redirects/index.js:169:12) 2023-10-11T19:13:21.786904883+02:00 at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 10/11/2023, 7:13:21 PM ERROR [JobService] Object: { "id": "83b72b63-b869-40f0-80f4-d1aea477b1d5" } 2023-10-11T19:13:21.786923019+02:00 [Nest] 7 - 10/11/2023, 7:13:24 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset bd25f755-3da6-4a76-ac31-ca58581dc380 [Nest] 7 - 10/11/2023, 7:13:27 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 7ba9f06f-b342-4dfe-bbf1-c77217894f27 [Nest] 7 - 10/11/2023, 7:13:27 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:27 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:13220:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31) at async /usr/src/app/dist/domain/job/job.service.js:109:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:27.613029534+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 7 - 10/11/2023, 7:13:27 PM ERROR [JobService] Object: { 2023-10-11T19:13:27.613045059+02:00 "id": "ea8e152f-42b6-43d2-9ad8-a595e6b61798" 2023-10-11T19:13:27.613048055+02:00 } 2023-10-11T19:13:27.613050837+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:13220:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31) at async /usr/src/app/dist/domain/job/job.service.js:109:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: { "id": "a56c1370-5bd1-450b-8de1-e4da00e79672" } [Nest] 7 - 10/11/2023, 7:13:28 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 2b794421-20c3-40b6-a98b-8d6a63e6fe32 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.431743273+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:28.431746795+02:00 at processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-10-11T19:13:28.431749391+02:00 at runNextTicks (node:internal/process/task_queues:64:3) 2023-10-11T19:13:28.431752058+02:00 at process.processImmediate (node:internal/timers:449:9) 2023-10-11T19:13:28.431754629+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:28.431757205+02:00 at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22) 2023-10-11T19:13:28.431759672+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:28.431762635+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:28.431765171+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:28.432117622+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: 2023-10-11T19:13:28.432129611+02:00 { 2023-10-11T19:13:28.432133520+02:00 "id": "e58d5d60-9ea3-44b9-a3c9-fe7defa6d867" 2023-10-11T19:13:28.432136263+02:00 } 2023-10-11T19:13:28.432138727+02:00 2023-10-11T19:13:28.432719161+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed 2023-10-11T19:13:28.432823783+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.432830041+02:00 at Object.fetch (node:internal/deps/undici/und ici:13220:11) 2023-10-11T19:13:28.432833106+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-10-11T19:13:28.432835650+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:28.432838258+02:00 at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23) 2023-10-11T19:13:28.432840714+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:28.432843403+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:28.432845819+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:28.432936251+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: 2023-10-11T19:13:28.432941285+02:00 { 2023-10-11T19:13:28.432944105+02:00 "id": "7994bfd3-53ac-4f7e-9649-0829250f9d41" 2023-10-11T19:13:28.432946595+02:00 } 2023-10-11T19:13:28.432948911+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.579157643+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:28.579161423+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23) at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:28.579171899+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: 2023-10-11T19:13:28.579295756+02:00 { "id": "5b396809-39dd-4c09-a369-177bebd81a27" 2023-10-11T19:13:28.579302705+02:00 } [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.624674626+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-10-11T19:13:28.624681721+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:28.624684526+02:00 at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22) 2023-10-11T19:13:28.624696932+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:28.624700461+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:28.624703093+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:28.624823264+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: 2023-10-11T19:13:28.624829223+02:00 { 2023-10-11T19:13:28.624832882+02:00 "id": "7120cb7b-6163-4dae-82bc-1ac29914b138" 2023-10-11T19:13:28.624835929+02:00 } 2023-10-11T19:13:28.624838648+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset cf4ec11f-c6d6-49c5-9064-58cb3d1be707 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.783314243+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:28.783350662+02:00 at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22) 2023-10-11T19:13:28.783363858+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:28.783387750+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: { 2023-10-11T19:13:28.783424842+02:00 "id": "4589c464-8307-455e-83f6-e19b76a3a424" } [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed 2023-10-11T19:13:28.840767205+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:28.840794602+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:28.840808677+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-10-11T19:13:28.840820622+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:28.840831492+02:00 at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31) 2023-10-11T19:13:28.840842777+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:28.840854979+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:28.841024661+02:00 [Nest] 7 - 10/11/2023, 7:13:28 PM ERROR [JobService] Object: 2023-10-11T19:13:28.841045353+02:00 { 2023-10-11T19:13:28.841059213+02:00 "id": "e81f8a58-0e8a-491d-a9cc-f28bb4eea2b6" 2023-10-11T19:13:28.841070781+02:00 } 2023-10-11T19:13:28.841110845+02:00 [Nest] 7 - 10/11/2023, 7:13:29 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 80a79908-c49b-487f-bead-a224b3453cd7 [Nest] 7 - 10/11/2023, 7:13:30 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 5b471cd1-3933-4ca4-ab60-229807e3e860 [Nest] 7 - 10/11/2023, 7:13:31 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-thumbhash-thumbnail): Error: Connection terminated due to connection timeout [Nest] 7 - 10/11/2023, 7:13:31 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-10-11T19:13:31.482596099+02:00 at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) 2023-10-11T19:13:31.482603215+02:00 at Connection.emit (node:events:514:28) 2023-10-11T19:13:31.482606685+02:00 at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) 2023-10-11T19:13:31.482609283+02:00 at Socket.emit (node:events:514:28) 2023-10-11T19:13:31.482611884+02:00 at TCP.<anonymous> (node:net:337:12) 2023-10-11T19:13:31.482651487+02:00 [Nest] 7 - 10/11/2023, 7:13:31 PM ERROR [JobService] Object: { 2023-10-11T19:13:31.482660906+02:00 "id": "627e6394-f677-429b-b444-9399354d24b3" 2023-10-11T19:13:31.482663691+02:00 } [Nest] 7 - 10/11/2023, 7:13:31 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 627e6394-f677-429b-b444-9399354d24b3 [Nest] 7 - 10/11/2023, 7:13:32 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 10/11/2023, 7:13:32 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:32.198116303+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23) at async /usr/src/app/dist/domain/job/job.service.js:109:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 7 - 10/11/2023, 7:13:32 PM ERROR [JobService] Object: { "id": "3dd9c9a9-349e-4c1e-94c3-44d1b7076a29" } [Nest] 7 - 10/11/2023, 7:13:32 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset f6dccf33-3230-4c2b-9031-b966c884d613 [Nest] 7 - 10/11/2023, 7:13:32 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 024b537c-c30c-4e27-9234-8138f0338861 [Nest] 7 - 10/11/2023, 7:13:32 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 288eb4c7-366d-4b0b-8ba0-a4f420a45882 [Nest] 7 - 10/11/2023, 7:13:33 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 986f85df-01c5-4e48-88c3-e3e84e353c1b [Nest] 7 - 10/11/2023, 7:13:37 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 72935da0-66e4-46ef-8825-8c77d18c37a5 [Nest] 7 - 10/11/2023, 7:13:39 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 355746ab-8b70-49d1-af8a-90f9d7712b91 [Nest] 7 - 10/11/2023, 7:13:39 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 6b59b01c-aaa1-4f53-adc5-4a271ae15f50 [Nest] 7 - 10/11/2023, 7:13:40 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed 2023-10-11T19:13:40.431657438+02:00 [Nest] 7 - 10/11/2023, 7:13:40 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:40.431662824+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:40.431666226+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:40.431671922+02:00 at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31) 2023-10-11T19:13:40.431674737+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:40.431677687+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:40.431680727+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:40.431713510+02:00 [Nest] 7 - 10/11/2023, 7:13:40 PM ERROR [JobService] Object: 2023-10-11T19:13:40.431719093+02:00 { 2023-10-11T19:13:40.431722445+02:00 "id": "641554c5-1502-4e1f-8559-591c844acce7" 2023-10-11T19:13:40.431725012+02:00 } 2023-10-11T19:13:40.431727377+02:00 [Nest] 7 - 10/11/2023, 7:13:40 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset 984e81dc-4b5d-4d89-89d5-afc48563b3d5 [Nest] 7 - 10/11/2023, 7:13:42 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset db3f355f-b4f3-481d-98b7-5f919261b7dd [Nest] 7 - 10/11/2023, 7:13:42 PM LOG [MediaService] Successfully generated WEBP image thumbnail for asset 56141756-dca1-4f43-917b-c0f0f2c01327 [Nest] 7 - 10/11/2023, 7:13:44 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed 2023-10-11T19:13:44.972488668+02:00 [Nest] 7 - 10/11/2023, 7:13:44 PM ERROR [JobService] TypeError: fetch failed 2023-10-11T19:13:44.972501987+02:00 at Object.fetch (node:internal/deps/undici/undici:13220:11) 2023-10-11T19:13:44.972506083+02:00 at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) 2023-10-11T19:13:44.972508814+02:00 at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22) 2023-10-11T19:13:44.972511586+02:00 at async /usr/src/app/dist/domain/job/job.service.js:109:37 2023-10-11T19:13:44.972514865+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-10-11T19:13:44.972517493+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-10-11T19:13:44.972615981+02:00 [Nest] 7 - 10/11/2023, 7:13:44 PM ERROR [JobService] Object: 2023-10-11T19:13:44.972622001+02:00 { 2023-10-11T19:13:44.972625531+02:00 "id": "62f8ee52-90b3-4251-a1fa-81b84436fde6" 2023-10-11T19:13:44.972628553+02:00 } 2023-10-11T19:13:44.972631259+02:00 [Nest] 7 - 10/11/2023, 7:13:47 PM LOG [MediaService] Successfully generated WEBP video thumbnail for asset c27a8827-c65e-48c0-a7fc-4bece1c49ae1 /usr/src/app/dist/microservices/app.service.js:85 2023-10-11T19:13:48.529060637+02:00 throw error; 2023-10-11T19:13:48.529065008+02:00 ^ 2023-10-11T19:13:48.529067863+02:00 2023-10-11T19:13:48.529070587+02:00 Error: Error downloading GeoNames cities500 data: Error: aborted 2023-10-11T19:13:48.529092264+02:00 at PassThrough.<anonymous> (/usr/src/app/node_modules/local-reverse-geocoder/index.js:251:19) 2023-10-11T19:13:48.529098259+02:00 at PassThrough.emit (node:events:526:35) 2023-10-11T19:13:48.529100933+02:00 at emitErrorNT (node:internal/streams/destroy:151:8) 2023-10-11T19:13:48.529103442+02:00 at emitErrorCloseNT (node:internal/streams/destroy:116:3) 2023-10-11T19:13:48.529108595+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:82:21) 2023-10-11T19:13:48.529110877+02:00 2023-10-11T19:13:48.529113559+02:00 Node.js v20.8.0 ```

OVERLORD commented 2026-02-05 01:06:40 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 13, 2023):

I cannot use immich anymore to make thumbernails or anythings that need microservices because of this bug , so immich its unusable

@yodatak commented on GitHub (Oct 13, 2023): I cannot use immich anymore to make thumbernails or anythings that need microservices because of this bug , so immich its unusable

OVERLORD commented 2026-02-05 01:06:43 +03:00

Author

Owner

Copy Link

@jrasm91 commented on GitHub (Oct 14, 2023):

Can you try adjusting some of the job concurrencies to be lower values? It seems something is taking up all the available resources (possible disk i/o).

@jrasm91 commented on GitHub (Oct 14, 2023): Can you try adjusting some of the job concurrencies to be lower values? It seems something is taking up all the available resources (possible disk i/o).

OVERLORD commented 2026-02-05 01:06:46 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 14, 2023):

Whith only one in concurent job I got the same issue

@yodatak commented on GitHub (Oct 14, 2023): Whith only one in concurent job I got the same issue

OVERLORD commented 2026-02-05 01:06:49 +03:00

Author

Owner

Copy Link

@alextran1502 commented on GitHub (Oct 14, 2023):

@yodatak how much resource are you allocating for the Immich instance?

@alextran1502 commented on GitHub (Oct 14, 2023): @yodatak how much resource are you allocating for the Immich instance?

OVERLORD commented 2026-02-05 01:06:54 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 15, 2023):

Some more error: with
tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c
4000 mcpu limit and 8GO limit of ram

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-name: immich
    meta.helm.sh/release-namespace: photos
  labels:
    app: immich-8.1.24
    app.kubernetes.io/instance: immich
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: immich
    app.kubernetes.io/version: 1.81.1
    helm-revision: '7'
    helm.sh/chart: immich-8.1.24
    helm.toolkit.fluxcd.io/name: immich
    helm.toolkit.fluxcd.io/namespace: photos
    release: immich
  name: immich-microservices
  namespace: XXXX
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: immich
      app.kubernetes.io/name: immich
      pod.name: microservices
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        rollme: EDIHZ
      creationTimestamp: null
      labels:
        app: immich-8.1.24
        app.kubernetes.io/instance: immich
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: immich
        app.kubernetes.io/version: 1.81.1
        helm-revision: '7'
        helm.sh/chart: immich-8.1.24
        pod.name: microservices
        release: immich
    spec:
      automountServiceAccountToken: false
      containers:
        - args:
            - start-microservices.sh
          env:
            - name: TZ
              value: Europe/Paris
            - name: UMASK
              value: '0022'
            - name: UMASK_SET
              value: '0022'
            - name: S6_READ_ONLY_ROOT
              value: '1'
          envFrom:
            - secretRef:
                name: immich-secret
            - secretRef:
                name: immich-deps-secret
            - configMapRef:
                name: immich-common-config
            - configMapRef:
                name: immich-micro-config
          image: >-
            tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 10004
            timeoutSeconds: 5
          name: immich
          ports:
            - containerPort: 10004
              name: microservices
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 2
            tcpSocket:
              port: 10004
            timeoutSeconds: 5
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - ALL
            privileged: true
            readOnlyRootFilesystem: false
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          startupProbe:
            failureThreshold: 60
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            tcpSocket:
              port: 10004
            timeoutSeconds: 2
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: devshm
            - mountPath: /media
              name: externallibrary
            - mountPath: /usr/src/app/upload/library
              name: library
            - mountPath: /microcache
              name: microcache
            - mountPath: /mnt/revolution/Photos Videos
              name: originals-photos
            - mountPath: /usr/src/app/upload/profile
              name: profile
            - mountPath: /shared
              name: shared
            - mountPath: /usr/src/app/upload/thumbs
              name: thumbs
            - mountPath: /tmp
              name: tmp
            - mountPath: /usr/src/app/upload/upload
              name: uploads
            - mountPath: /var/logs
              name: varlogs
            - mountPath: /var/run
              name: varrun
            - mountPath: /usr/src/app/upload/encoded-video
              name: video
      dnsConfig:
        options:
          - name: ndots
            value: '1'
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      initContainers:
        - command:
            - /bin/ash
            - '-c'
            - >
              echo "Pinging [http://immich-server:10001/server-info/ping] until
              it is ready..."

              until wget --spider --quiet
              "http://immich-server:10001/server-info/ping"; do
                echo "Waiting for [http://immich-server:10001/server-info/ping] to be ready..."
                sleep 2
              done

              echo "URL [http://immich-server:10001/server-info/ping] is ready!"
          env:
            - name: TZ
              value: Europe/Paris
            - name: UMASK
              value: '0022'
            - name: UMASK_SET
              value: '0022'
            - name: S6_READ_ONLY_ROOT
              value: '1'
          image: >-
            tccr.io/truecharts/alpine:v3.18.2@sha256:a661597195a991150580f4915018663dd18bc36f5fef1e88e00c7a3e7b93e127
          imagePullPolicy: IfNotPresent
          name: immich-init-wait-server
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - ALL
            privileged: true
            readOnlyRootFilesystem: false
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: devshm
            - mountPath: /mnt/XXXX/Photos Videos
              name: originals-photos
            - mountPath: /shared
              name: shared
            - mountPath: /tmp
              name: tmp
            - mountPath: /var/logs
              name: varlogs
            - mountPath: /var/run
              name: varrun
      restartPolicy: Always
      runtimeClassName: nvidia
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch
        supplementalGroups:
          - 568
          - 1000
          - 0
          - 568
      serviceAccount: default
      serviceAccountName: default
      shareProcessNamespace: false
      terminationGracePeriodSeconds: 60
      volumes:
        - emptyDir:
            medium: Memory
          name: devshm
        - name: externallibrary
          persistentVolumeClaim:
            claimName: immich-externallibrary
        - hostPath:
            path: /mnt/XXXXX/immich/library
            type: Directory
          name: library
        - emptyDir: {}
          name: microcache
        - hostPath:
            path: /mnt/XXXX/Photos
            type: Directory
          name: originals-photos
        - hostPath:
            path: /mnt/XXXX/immich/profile
            type: Directory
          name: profile
        - emptyDir: {}
          name: shared
        - hostPath:
            path: /opt/XXXX/thumbs
            type: Directory
          name: thumbs
        - emptyDir: {}
          name: tmp
        - hostPath:
            path: /mnt/XXXXX/immich/upload
            type: Directory
          name: uploads
        - emptyDir: {}
          name: varlogs
        - emptyDir:
            medium: Memory
          name: varrun
        - hostPath:
            path: /mnt/XXXX/immich/encoded-video
            type: Directory
          name: video

Container: immich
Disconnected
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [NestFactory] Starting Nest application...
2023-10-15T16:58:55.056797643+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +28ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +110ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] InfraModule dependencies initialized +5ms
2023-10-15T16:58:55.186795380+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DomainModule dependencies initialized +9ms
[Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/undefined): TypeError: handler is not a function
[Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] TypeError: handler is not a function
at /usr/src/app/dist/domain/job/job.service.js:109:43
at Worker.workerHandler [as processFn] (/usr/src/app/dist/infra/repositories/job.repository.js:27:46)
2023-10-15T16:59:05.580804341+02:00 at Worker.callProcessJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:106:21)
at Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:39)
at /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:70
at Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:30)
at Worker.run (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:45)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Object:
{}
2023-10-15T16:59:05.580845846+02:00

@yodatak commented on GitHub (Oct 15, 2023): Some more error: with tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c 4000 mcpu limit and 8GO limit of ram ```yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: meta.helm.sh/release-name: immich meta.helm.sh/release-namespace: photos labels: app: immich-8.1.24 app.kubernetes.io/instance: immich app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: immich app.kubernetes.io/version: 1.81.1 helm-revision: '7' helm.sh/chart: immich-8.1.24 helm.toolkit.fluxcd.io/name: immich helm.toolkit.fluxcd.io/namespace: photos release: immich name: immich-microservices namespace: XXXX spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 3 selector: matchLabels: app.kubernetes.io/instance: immich app.kubernetes.io/name: immich pod.name: microservices strategy: type: Recreate template: metadata: annotations: rollme: EDIHZ creationTimestamp: null labels: app: immich-8.1.24 app.kubernetes.io/instance: immich app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: immich app.kubernetes.io/version: 1.81.1 helm-revision: '7' helm.sh/chart: immich-8.1.24 pod.name: microservices release: immich spec: automountServiceAccountToken: false containers: - args: - start-microservices.sh env: - name: TZ value: Europe/Paris - name: UMASK value: '0022' - name: UMASK_SET value: '0022' - name: S6_READ_ONLY_ROOT value: '1' envFrom: - secretRef: name: immich-secret - secretRef: name: immich-deps-secret - configMapRef: name: immich-common-config - configMapRef: name: immich-micro-config image: >- tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 10004 timeoutSeconds: 5 name: immich ports: - containerPort: 10004 name: microservices protocol: TCP readinessProbe: failureThreshold: 5 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 2 tcpSocket: port: 10004 timeoutSeconds: 5 resources: limits: cpu: '4' memory: 8Gi requests: cpu: 10m memory: 50Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - ALL privileged: true readOnlyRootFilesystem: false runAsGroup: 1000 runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault startupProbe: failureThreshold: 60 initialDelaySeconds: 10 periodSeconds: 5 successThreshold: 1 tcpSocket: port: 10004 timeoutSeconds: 2 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: devshm - mountPath: /media name: externallibrary - mountPath: /usr/src/app/upload/library name: library - mountPath: /microcache name: microcache - mountPath: /mnt/revolution/Photos Videos name: originals-photos - mountPath: /usr/src/app/upload/profile name: profile - mountPath: /shared name: shared - mountPath: /usr/src/app/upload/thumbs name: thumbs - mountPath: /tmp name: tmp - mountPath: /usr/src/app/upload/upload name: uploads - mountPath: /var/logs name: varlogs - mountPath: /var/run name: varrun - mountPath: /usr/src/app/upload/encoded-video name: video dnsConfig: options: - name: ndots value: '1' dnsPolicy: ClusterFirst enableServiceLinks: false initContainers: - command: - /bin/ash - '-c' - > echo "Pinging [http://immich-server:10001/server-info/ping] until it is ready..." until wget --spider --quiet "http://immich-server:10001/server-info/ping"; do echo "Waiting for [http://immich-server:10001/server-info/ping] to be ready..." sleep 2 done echo "URL [http://immich-server:10001/server-info/ping] is ready!" env: - name: TZ value: Europe/Paris - name: UMASK value: '0022' - name: UMASK_SET value: '0022' - name: S6_READ_ONLY_ROOT value: '1' image: >- tccr.io/truecharts/alpine:v3.18.2@sha256:a661597195a991150580f4915018663dd18bc36f5fef1e88e00c7a3e7b93e127 imagePullPolicy: IfNotPresent name: immich-init-wait-server resources: limits: cpu: '4' memory: 8Gi requests: cpu: 10m memory: 50Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - ALL privileged: true readOnlyRootFilesystem: false runAsGroup: 1000 runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: devshm - mountPath: /mnt/XXXX/Photos Videos name: originals-photos - mountPath: /shared name: shared - mountPath: /tmp name: tmp - mountPath: /var/logs name: varlogs - mountPath: /var/run name: varrun restartPolicy: Always runtimeClassName: nvidia schedulerName: default-scheduler securityContext: fsGroup: 100 fsGroupChangePolicy: OnRootMismatch supplementalGroups: - 568 - 1000 - 0 - 568 serviceAccount: default serviceAccountName: default shareProcessNamespace: false terminationGracePeriodSeconds: 60 volumes: - emptyDir: medium: Memory name: devshm - name: externallibrary persistentVolumeClaim: claimName: immich-externallibrary - hostPath: path: /mnt/XXXXX/immich/library type: Directory name: library - emptyDir: {} name: microcache - hostPath: path: /mnt/XXXX/Photos type: Directory name: originals-photos - hostPath: path: /mnt/XXXX/immich/profile type: Directory name: profile - emptyDir: {} name: shared - hostPath: path: /opt/XXXX/thumbs type: Directory name: thumbs - emptyDir: {} name: tmp - hostPath: path: /mnt/XXXXX/immich/upload type: Directory name: uploads - emptyDir: {} name: varlogs - emptyDir: medium: Memory name: varrun - hostPath: path: /mnt/XXXX/immich/encoded-video type: Directory name: video ```` Container: immich Disconnected [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [NestFactory] Starting Nest application... 2023-10-15T16:58:55.056797643+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +28ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +110ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] InfraModule dependencies initialized +5ms 2023-10-15T16:58:55.186795380+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DomainModule dependencies initialized +9ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/undefined): TypeError: handler is not a function [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] TypeError: handler is not a function at /usr/src/app/dist/domain/job/job.service.js:109:43 at Worker.workerHandler [as processFn] (/usr/src/app/dist/infra/repositories/job.repository.js:27:46) 2023-10-15T16:59:05.580804341+02:00 at Worker.callProcessJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:106:21) at Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:39) at /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:70 at Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:30) at Worker.run (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:45) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Object: {} 2023-10-15T16:59:05.580845846+02:00

OVERLORD commented 2026-02-05 01:06:56 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 15, 2023):

I found for me that the issue is i have PREFER WIDE GAMUT enabled if i disabled it it work for me my issue is dependend of https://github.com/immich-app/immich/pull/4438

@yodatak commented on GitHub (Oct 15, 2023): I found for me that the issue is i have PREFER WIDE GAMUT enabled if i disabled it it work for me my issue is dependend of https://github.com/immich-app/immich/pull/4438

OVERLORD commented 2026-02-05 01:07:02 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 26, 2023):

i still have the issue :/

@yodatak commented on GitHub (Oct 26, 2023): i still have the issue :/

OVERLORD commented 2026-02-05 01:07:05 +03:00

Author

Owner

Copy Link

@yodatak commented on GitHub (Oct 26, 2023):

[Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout
[Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Error: Connection terminated due to connection timeout
2023-10-26T20:36:54.847256916+02:00 at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-10-26T20:36:54.847259004+02:00 at Object.onceWrapper (node:events:628:28)
2023-10-26T20:36:54.847260689+02:00 at Connection.emit (node:events:514:28)
at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
at Socket.emit (node:events:514:28)
2023-10-26T20:36:54.847266141+02:00 at TCP. (node:net:337:12)

@yodatak commented on GitHub (Oct 26, 2023): [Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout [Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-10-26T20:36:54.847256916+02:00 at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) 2023-10-26T20:36:54.847259004+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:36:54.847260689+02:00 at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) 2023-10-26T20:36:54.847266141+02:00 at TCP.<anonymous> (node:net:337:12)

OVERLORD commented 2026-02-05 01:07:10 +03:00

Author

Owner

Copy Link

@traktuner commented on GitHub (Nov 9, 2023):

I have the same problem.
I clicked "refresh" for the library tasks (400k images) and not long after that:

<--- Last few GCs --->

[1022:0x30904830000]   377636 ms: Mark-sweep 4049.0 (4136.8) -> 4034.4 (4138.1) MB, 5284.4 / 0.0 ms  (average mu = 0.110, current mu = 0.010) allocation failure; scavenge might not succeed

[1022:0x30904830000]   383081 ms: Mark-sweep 4050.3 (4138.1) -> 4035.5 (4139.1) MB, 5382.1 / 0.0 ms  (average mu = 0.063, current mu = 0.011) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

 1: 0xb85bc0 node::Abort() [immich_microservices]

 2: 0xa94834  [immich_microservices]

 3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices]

 4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices]

 5: 0xf447c5  [immich_microservices]

 6: 0xf456c8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]

 7: 0xf55bd3  [immich_microservices]

 8: 0xf56a48 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]

 9: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]

10: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]

11: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]

12: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]

13: 0x1705b39  [immich_microservices]

@traktuner commented on GitHub (Nov 9, 2023): I have the same problem. I clicked "refresh" for the library tasks (400k images) and not long after that: ``` <--- Last few GCs ---> [1022:0x30904830000] 377636 ms: Mark-sweep 4049.0 (4136.8) -> 4034.4 (4138.1) MB, 5284.4 / 0.0 ms (average mu = 0.110, current mu = 0.010) allocation failure; scavenge might not succeed [1022:0x30904830000] 383081 ms: Mark-sweep 4050.3 (4138.1) -> 4035.5 (4139.1) MB, 5382.1 / 0.0 ms (average mu = 0.063, current mu = 0.011) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xb85bc0 node::Abort() [immich_microservices] 2: 0xa94834 [immich_microservices] 3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices] 4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices] 5: 0xf447c5 [immich_microservices] 6: 0xf456c8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0xf55bd3 [immich_microservices] 8: 0xf56a48 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x1705b39 [immich_microservices] ```

OVERLORD commented 2026-02-05 01:07:13 +03:00

Author

Owner

Copy Link

@zackpollard commented on GitHub (Nov 26, 2023):

This issue seems to have diverged from the original problem. I'm not sure all the reported problems are the same. Out of memory exceptions are quite often caused by the library we used for reverse geocoding. In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!

@zackpollard commented on GitHub (Nov 26, 2023): This issue seems to have diverged from the original problem. I'm not sure all the reported problems are the same. Out of memory exceptions are quite often caused by the library we used for reverse geocoding. In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

feat/asset-file-apis

chore/translations

fix/web-switch-label-clickable

fix/web-people-hidden-state

renovate/typescript-projects

release/next

fix/timezones

fix/time-zone-upserts

midzelis/wip

push-zpwsovysllvn

push-nwxlpmyzkyrl

push-nvnkszuqwppm

renovate/github-actions

push-smstsuupsowp

refactor/adaptive_image

push-olwpzvrxnomt

push-lmxsupnmxspl

renovate/machine-learning

feat/web-chromecast-video-looping

feat/use-native-clients

renovate/flutter

fix/create-face-edited

fix/mobile-ios-mtls

docs/contributing

docs/mise-mobile

renovate/grafana-monorepo

feature/bottom-buttons-order

feat/immich-mobile-ui-showcase

refactor/consolidate-image-requests

renovate/connectivity_plus-7.x

renovate/major-vitest-monorepo

renovate/pypi-python-multipart-vulnerability

fix/mobile-people-query

sqlite_thumbs

feat/html-text

chore/no-macro-validation

refactor/purchase-store

uhthomas/mobile-fix-app-bar-fade

uhthomas/mobile-fix-asset-jump

feat/pano-ocr

feat/shared-link-login

fix/database-backup-db-names

fix-keep-correct-ios-shared-album-asset

fix-memory-generation-and-display

feat/verify-permissions

refactor/album-service-small-tests

fix/ml-rocm-build

fix/flipped-dimensions-mobile

push-vpxwmwwxwnvw

fix-migration-width-height

refactor/more-queries

revert/prettier-translations

refactor/asset-service-queries

fix/locale-settings-desc

chore/add-debug-log

feat/edit-filters

shared-deep-link-handler

feat/mobile-editing

feat/thumbnail-native-clients

feat/platform-clients

feat/integrity-checks-izzy

fix/foreground-cloud-sync

feat/dynamic-layout

filter-by-person

feat/csp

refactor/sidebar

fix/disable-editing

fix/view-timeline-deeplink

image-zoom-on-slow-connection

fix-consider-dar-for-video-dimension

fix/merged-edited-assets

perf/optimize-album-sort

open-api-fix

feat/create-job-with-dto

use-toast-primary

feat/vitest-4

feat/ios-fastlane-match

match-signing

fix-update-time-update-timeline

chore/translation-keys

feat/modal-routes

feat/panorama-tiles

feature/mobile-view-asset-owner

feat/system-settings

feature/show-activity-count

better-info-in-asset-viewer

fix/all-people-count

feat/location-favorites

feature/rearrange-buttons-2

fix/download-storage-template

feat/kb-shortcuts-mobile

fix/people-count

push-qolzzzzxrvvn

chore/originals-in-asset-files

feat/asset-size-columns

ben/tree-a11y

new-search-filter-ui

refactor/expectSelectedReadonly

refactor/mobile-grdb

push-qvuktpxmkknu

feat/mobile-native-local-sync

refactor/timeline_ops

fix/scrubber_end

feat/version.txt

feat/context-menus

feat/server-chunked-uploads

refactor/virtualsegment

refactor/rename_daymonth_groups

fix/restrict-android-bg-worker

feat/android-periodic-worker

fix-remote-sync-clean-up

refactor/timeline_move_ops

renovate/mapbox-mapbox-gl-rtl-text-0.x

fix/timeline_split_selectable

feat/keyboard_actions_help_modal

feat/static_frontend

feat/notification-warnign-android

feat/plugins2

feat/plugins

test/create-workflow-token-action

fix/docs-force

debug/search-result-similarity

debug/cf-chunked-uploads

feat/eslint_rule

feat/search-filter-album/web

refactor/timeline_photostream

refactor/timelineasset_asset

feat/session-permissions

feat/timeline_photostream_assetnav

feat/timeline_minor_optimize

feat/timeline_perf_nocomp

feat/timeline_search_results_actions

feat/timeline_search_results_page

fix/timeline_padding

fix/timeline_search_reactivity_warnings

feat/timeline_scrollbar

feat/timeline_stream_withviewer

fix/timeline_back_forth_nav

refactor/timeline_photostream_component

fix/generated-files-checks

fix/locate-button-local

chore/base-image-mimalloc

refactor/timeline_assetlayout

refactor/timeline_selectable

refactor/timeline_aware_actions

refactor/timeline_monthsegment

feat/remove-old-pages

chore/deps-gradle

tmp_photostream

tmp/lcms

feat/mobile-dynamic-thumbnails

fix/mobile-finer-thumbnail-concurrency

refactor/timeline1

refactor/extract_photostream

refactor/rename_load_api

refactor/timeline2

refactor/timeline3

feat/multi-select-asset-viewer

feat-no-thumbhash-cache

refactor/asset_grid

feat/faster-access-checks

fix/18991

fix/19543

chore/temp-remove

fix/21419

feat/mobile-hdr-images

chore/update-mise-lockfile

feat/mise-server-checks

feat/mise-ci

feat/windows-2025

feat/dev_cli

refactor/mobile-migrate-clients

fix/map-theme

fix/require-checkbox

chore/use_swc

feat/efficient-thumbnail-decoding

refactor/mobile-thumbhash

refactor/mobile-thumbhash-new

fix/mobile-uncached-zoom

feat/beta-background-upload

fix/beta-timeline-memories-setting

fix/failed-uploads-not-removed

feat/mobile-shared-album

feat/groups

drift-map-page

drift-auth-user-sync

fix/disable-memory

feat/add-to-album-action

edit-date-time-action

drift-people-page

sqlite-remove-isIn

feat/inline-storage-columns

chore/required-reviewers

refact/asset-manager

fix/folder-sort

pnpm

feat/widget-multiple-server-urls

chore/medium-tests-dbname

fix/web-no-iterator-find

fix/map-pan-interruption

track-livephotos

timeline_events

chore/oxlint-migration

feat/maintenance-worker

feat/dav

chore/demo-snapshot

refactor/server-side-dedupe

feat/integrity-checks

dev/recognition-eval

lighter_buckets_test

perf/postgres-queue

postgres-queue

focus_rings

refactor/web-stores-1

refactor/add-to-taken

feat/sort-places

feat/sidecar-asset-file

vet

tmp/demo-snapshot-preview

fix/server-migration-file-extension

refactor/mobile-v2

fix/asset-update-race-condition

rknn-toolkit-lite2

refactor/mobile-split-up-search-page

feature/Add-rocm-support-for-machine-learning

feat/rocm

chore/async-hash-file

feat/shared-link-view-count

feat/rotation

feat/graphql

feat/job-ids

feat/ignore-library-permission-error

feat/docker-compose-builder

feat/kysely-typeorm

mobile/onboarding

no-video-player

fix/server-qsv-output-format

chore/server-geodata-tweaks

mobile/native-video-player-no-hero

feat/xxhash

fix/docs-concurrency

feat/preload-ml-textual-model

feat/local-tileserver

refactor/exif-orientation

original-path-infix

refactor/mobile/login-form-1

feat/server-editor-endpoints

fix/server-qsv-vbr

fix-mobile-db-problems

feat/ml-armnn-conversion

feat/mobile/backup-with-album-info

feat/fast-initial-sync-1

chore/handle-output_dims

feat/server-more-robust-generation

feat/unassign-faces

feat/shortcuts-on-asset-grid

feat/background-upload

feat/capacitor-mobile-app-poc

feat/server-nvenc-hw-decoding

release/v1.105

fix/mobile-fetch-non-archive

feat/fine-grained-access-controls

web/automation-ui

feat/mobile-server-endpoint-save-dropdown

feat/blurhash-thumbnail

object-storage

feat/memories-animations

dev/metrics

ml/tflite

feat/ml-export-cli

v2.5.3

v2.5.2

v2.5.1

v2.5.0

v2.4.1

v2.4.0

v2.3.1

v2.3.0

v2.2.3

v2.2.2

v2.2.1

v2.2.0

v2.1.0

v2.0.1

v2.0.0

v1.144.1

v1.144.0

v1.143.1

v1.143.0

v1.142.1

v1.142.0

v1.141.1

v1.141.0

v1.140.1

v1.140.0

v1.139.4

v1.139.3

v1.139.2

v1.139.1

v1.139.0

v1.138.1

v1.138.0

v1.137.3

v1.137.2

v1.137.1

v1.137.0

v1.136.0

v1.135.3

v1.135.2

v1.135.1

v1.135.0

v1.134.0

v1.133.1

v1.133.0

v1.132.3

v1.132.2

v1.132.1

v1.132.0

v1.131.3

v1.131.2

v1.131.1

v1.131.0

v1.130.3

v1.130.2

v1.130.1

v1.130.0

v1.129.0

v1.128.0

v1.127.0

v1.126.1

v1.126.0

v1.125.7

v1.125.6

v1.125.5

v1.125.4

v1.125.3

v1.125.2

v1.125.1

v1.125.0

v1.124.2

v1.124.1

v1.124.0

v1.123.0

v1.122.3

v1.122.2

v1.122.1

v1.122.0

v1.121.0

v1.120.2

v1.120.1

v1.120.0

v1.119.1

v1.119.0

v1.118.2

v1.118.1

v1.118.0

v1.117.0

v1.116.2

v1.116.1

v1.116.0

v1.115.0

v1.114.0

v1.113.1

v1.113.0

v1.112.1

v1.112.0

v1.111.0

v1.110.0

v1.109.2

v1.109.1

v1.109.0

v1.108.0

v1.107.2

v1.107.1

v1.107.0

v1.106.4

v1.106.3

v1.106.2

v1.106.1

v1.106.0

v1.105.1

v1.105.0

v1.104.0

v1.103.1

v1.103.0

v1.102.3

v1.102.2

v1.102.1

v1.102.0

v1.101.0

v1.100.0

v1.99.0

v1.98.2

v1.98.1

v1.98.0

v1.97.0

v1.96.0

v1.95.1

v1.95.0

v1.94.1

v1.94.0

v1.93.3

v1.93.2

v1.93.1

v1.93.0

v1.92.1

v1.92.0

v1.91.4

v1.91.3

v1.91.2

v1.91.1

v1.91.0

v1.90.2

v1.90.1

v1.90.0

v1.89.0

v1.88.2

v1.88.1

v1.88.0

v1.87.0

v1.86.0

v1.85.0

v1.84.0

v1.83.0

v1.82.1

v1.82.0

v1.81.1

v1.81.0

v1.80.0

v1.79.1

v1.79.0

v1.78.1

v1.78.0

v1.77.0

v1.76.1

v1.76.0

v1.75.2

v1.75.1

v1.75.0

v1.74.0

v1.73.0

v1.72.2

v1.72.1

v1.72.0

v1.71.0

v1.70.0

v1.69.0

v1.68.0

v1.67.2

v1.67.1

v1.67.0

v1.66.1

v1.66.0

v1.65.0

v1.64.0

v1.63.2

v1.63.1

v1.63.0

v1.62.1

v1.62.0

v1.61.0

v1.60.0

v1.59.1

v1.59.0

v1.58.0

v1.57.1

v1.57.0

v1.56.2

v1.56.1

v1.56.0

v1.55.1

v1.55.0

v1.54.1

v1.54.0

v1.53.0

v1.52.1

v1.52.0

v1.51.2

v1.51.1

v1.51.0

v1.50.1

v1.50.0

v1.49.0

v1.48.1

v1.48.0

v1.47.3

v1.47.2

v1.47.1

v1.47.0

v1.46.1

v1.46.0

v1.45.0

v1.44.0

v1.43.1

v1.43.0

v1.42.0_65-dev

v1.41.1_64-dev

v1.41.0_64-dev

v1.40.1_63-dev

v1.40.0_63-dev

v1.39.0_61-dev

v1.38.2_60-dev

v1.38.1_60-dev

v1.38.0_60-dev

v1.37.0_58-dev

v1.36.2_56-dev

v1.36.1_55-dev

v1.36.0_55-dev

v1.35.0_54-dev

v1.34.0_53-dev

v1.33.1_52-dev

v1.33.0_52-dev

v1.32.1_51-dev

v1.32.0_50-dev

v1.31.1_49-dev

v1.31.0_49-dev

v1.30.2_48-dev

v1.30.0_46-dev

v1.29.6_45-dev

v1.29.6_44-dev

v1.29.5_44-dev

v1.29.4_44-dev

v1.29.3_43-dev

v1.29.2_43-dev

v1.29.1_43-dev

v1.29.0_42-dev

v1.28.4_41-dev

v1.28.4_42-dev

v1.28.3_41-dev

v1.28.2_40-dev

v1.28.1_39-dev

v1.28.0_38-dev

v1.27.0_37-dev

v1.26.0_36-dev

v1.25.0_35-dev

v1.24.0_34-dev

v1.23.0_33-dev

v1.22.0_32-dev

v1.21.1_31-dev

v1.21.0_31-dev

v1.20.3_30-dev

v1.20.2_30-dev

v1.20.1_30-dev

v1.20.0_30-dev

v1.19.1_29-dev

v1.19.0_29-dev

v1.18.0_27-dev

v1.17.0_25-dev

v1.16.0_23-dev

v1.15.1_21-dev

v1.15.0_21-dev

v1.14.0_21-dev

v1.13.0_20-dev

v1.12.0_18-dev

v1.11.0_17-dev

v1.10.0_15-dev

v1.9.1_14-dev

v1.9.0_13-dev

v1.8.0_12-dev

v1.7.0_11-dev

v1.6.0_10-dev

v1.5.1+9-dev

v1.5.0+8-dev

v1.4.0+7-dev

v1.4.0+6-dev

v1.4.0-dev

v1.3.0-dev

v1.3.1-dev

v0.6-dev

v0.5-dev

v0.4-dev

v0.3-dev

v0.2-dev

first-android-release

Labels

Clear labels

accessibility

changelog:enhancement

changelog:security

changelog:translation

cli

date-time

documentation

external-library

format

good first issue

needs-answer

nice to have

sharing

tech-debt

📱mobile

🖥️web

🗄️server

🧠machine-learning

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

OVERLORD

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: immich-app/immich#1272