[BUG] Running the face recognization job results in "Failed 1" after a minute or so. #869

Closed
opened 2026-02-04 23:11:41 +03:00 by OVERLORD · 16 comments
Owner

Originally created by @florihupf on GitHub (May 21, 2023).

The bug

Nothing in the logs as I can see.

Retrying results in the same thing.

The OS that Immich Server is running on

Truenas Scale

Version of Immich Server

v1.56.2

Version of Immich Mobile App

v1.56.0 (96)

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

Stock

Your .env content

Stock

Reproduction steps

- After upgrading from v1.55.0 kick of a Face Recognition Job from the Admin panel.
- Wait for a minute or so.
- Now it says "Failed 1".

Note that there also was just a single job queued, not any more. Pointing out, because the other job types usually queue more jobs at once.

Additional information

Screenshot 2023-05-21 at 15 51 26
Originally created by @florihupf on GitHub (May 21, 2023). ### The bug Nothing in the logs as I can see. Retrying results in the same thing. ### The OS that Immich Server is running on Truenas Scale ### Version of Immich Server v1.56.2 ### Version of Immich Mobile App v1.56.0 (96) ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML Stock ``` ### Your .env content ```Shell Stock ``` ### Reproduction steps ```bash - After upgrading from v1.55.0 kick of a Face Recognition Job from the Admin panel. - Wait for a minute or so. - Now it says "Failed 1". ``` Note that there also was just a single job queued, not any more. Pointing out, because the other job types usually queue more jobs at once. ### Additional information <img width="836" alt="Screenshot 2023-05-21 at 15 51 26" src="https://github.com/immich-app/immich/assets/4993621/2908c5c3-305d-4895-8fb1-1d7ec323ddef">
Author
Owner

@alextran1502 commented on GitHub (May 21, 2023):

Can you help check your microservices log and machine learning container log?

@alextran1502 commented on GitHub (May 21, 2023): Can you help check your microservices log and machine learning container log?
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

Right now I see

INFO:     Waiting for application startup.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
  warnings.warn(
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/genderage.onnx genderage
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit)

In the machine learning container. Will see if anything gets added.

@florihupf commented on GitHub (May 21, 2023): Right now I see ``` INFO: Waiting for application startup. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider' warnings.warn( Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/models/buffalo_l/genderage.onnx genderage Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 set det-size: (640, 640) INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit) ``` In the machine learning container. Will see if anything gets added.
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

The micro services container says:

[Nest] 1  - 05/21/2023, 4:45:11 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized

<--- Last few GCs --->

[1:0x7f406a599300]   201293 ms: Mark-sweep 4045.5 (4132.9) -> 4030.9 (4134.2) MB, 4208.1 / 0.0 ms  (average mu = 0.083, current mu = 0.008) allocation failure; scavenge might not succeed
[1:0x7f406a599300]   205588 ms: Mark-sweep 4046.8 (4134.2) -> 4032.2 (4135.4) MB, 4243.3 / 0.0 ms  (average mu = 0.048, current mu = 0.012) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Having 8GB allocated for the docker environment. Maybe I need more?

@florihupf commented on GitHub (May 21, 2023): The micro services container says: ``` [Nest] 1 - 05/21/2023, 4:45:11 PM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized <--- Last few GCs ---> [1:0x7f406a599300] 201293 ms: Mark-sweep 4045.5 (4132.9) -> 4030.9 (4134.2) MB, 4208.1 / 0.0 ms (average mu = 0.083, current mu = 0.008) allocation failure; scavenge might not succeed [1:0x7f406a599300] 205588 ms: Mark-sweep 4046.8 (4134.2) -> 4032.2 (4135.4) MB, 4243.3 / 0.0 ms (average mu = 0.048, current mu = 0.012) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory ``` Having 8GB allocated for the docker environment. Maybe I need more?
Author
Owner

@alextran1502 commented on GitHub (May 21, 2023):

Hmm @jrasm91 any idea? Are you running docker-compose file or all-in-one image on your Truenas?

@alextran1502 commented on GitHub (May 21, 2023): Hmm @jrasm91 any idea? Are you running docker-compose file or all-in-one image on your Truenas?
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

I am running in a docker container from TrueCharts that provides a docker compose environment, ie. docker inside docker.

I could try the all-in-one image. Let me find it.

@florihupf commented on GitHub (May 21, 2023): I am running in a docker container from TrueCharts that provides a docker compose environment, ie. docker inside docker. I could try the all-in-one image. Let me find it.
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

Right now I see

INFO:     Waiting for application startup.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
  warnings.warn(
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/models/buffalo_l/genderage.onnx genderage
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit)

In the machine learning container. Will see if anything gets added.

After a while these logs resulted in "Failed 2" on the admin panel.

@florihupf commented on GitHub (May 21, 2023): > Right now I see > > ``` > INFO: Waiting for application startup. > Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. > The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`. > Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. > /opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider' > warnings.warn( > Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} > model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68 > Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} > model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106 > Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} > find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 > Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} > model ignore: /cache/models/buffalo_l/genderage.onnx genderage > Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} > find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 > set det-size: (640, 640) > INFO: Application startup complete. > INFO: Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit) > ``` > > In the machine learning container. Will see if anything gets added. After a while these logs resulted in "Failed 2" on the admin panel.
Author
Owner

@alextran1502 commented on GitHub (May 21, 2023):

I suggest you run the docker-compose route since it is the official support installation method. Truechart and the all-in-one image is community image.

@alextran1502 commented on GitHub (May 21, 2023): I suggest you run the `docker-compose` route since it is the official support installation method. Truechart and the all-in-one image is community image.
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

Yeah, for docker-compose to work on Truenas Scale, you need the docker-compose image provided by TrueCharts, because Truenas Scale doesn't support docker-compose :(

@florihupf commented on GitHub (May 21, 2023): Yeah, for docker-compose to work on Truenas Scale, you need the docker-compose image provided by TrueCharts, because Truenas Scale doesn't support docker-compose :(
Author
Owner

@jrasm91 commented on GitHub (May 21, 2023):

Right now the way the "all" jobs work it needs to load every single asset into memory and then queue them. It looks like it runs out of memory doing that. How big is your library (number of assets)?

I think allocating more RAM would eventually fix it. We have a backlog todo to redo how this works so that we can load and process chunks at a time, which would reduce the need for so much ram. Other than allocating more RAM I'm not sure what else you can do in the short term.

@jrasm91 commented on GitHub (May 21, 2023): Right now the way the "all" jobs work it needs to load every single asset into memory and then queue them. It looks like it runs out of memory doing that. How big is your library (number of assets)? I think allocating more RAM would eventually fix it. We have a backlog todo to redo how this works so that we can load and process chunks at a time, which would reduce the need for so much ram. Other than allocating more RAM I'm not sure what else you can do in the short term.
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

Right, my library is pretty large. I think more than 300GB and 10.000 assets.

@florihupf commented on GitHub (May 21, 2023): Right, my library is pretty large. I think more than 300GB and 10.000 assets.
Author
Owner

@florihupf commented on GitHub (May 21, 2023):

Error: Just checked 112k assets.

@florihupf commented on GitHub (May 21, 2023): Error: Just checked 112k assets.
Author
Owner

@jrasm91 commented on GitHub (May 21, 2023):

Yeah, unfortunately this is a limitation right now. If you have the ram for it you can temporarily increase it to queue the job and then reduce it again afterwards

@jrasm91 commented on GitHub (May 21, 2023): Yeah, unfortunately this is a limitation right now. If you have the ram for it you can temporarily increase it to queue the job and then reduce it again afterwards
Author
Owner

@ShinasShaji commented on GitHub (May 22, 2023):

Yeah, unfortunately this is a limitation right now. If you have the ram for it you can temporarily increase it to queue the job and then reduce it again afterwards

Can confirm that doing this works.

I had this issue - see my comment at #2468

I eventually got it working by allocating more RAM than I used to - 16 GB did the trick for me.

Also see my comment here. I don't know too much about it to confirm or verify, but immich_microservices doesn't seem to be using more than 20-25% of the memory available to the Docker container.

Here are my logs; you can see what is probably the database losing connection when immich_microservices crashes, and also see immich_microservices restarting in the end. I've had thumbnail generation failures before (probably due to corrupt image files), so I don't think that is strictly related to the issue.

immich_postgres          | 2023-05-19 04:12:22.715 UTC [44] LOG:  could not send data to client: Connection reset by peer
immich_postgres          | 2023-05-19 04:12:22.715 UTC [44] STATEMENT:  SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId"  LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id"  LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId"
immich_postgres          | 2023-05-19 04:12:22.715 UTC [44] FATAL:  connection to client lost
immich_postgres          | 2023-05-19 04:12:22.715 UTC [44] STATEMENT:  SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId"  LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id"  LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId"
immich_postgres          | 2023-05-19 04:14:37.181 UTC [48] LOG:  could not send data to client: Connection reset by peer
immich_postgres          | 2023-05-19 04:14:37.181 UTC [48] STATEMENT:  SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId"  LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id"  LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId"
immich_postgres          | 2023-05-19 04:14:37.183 UTC [48] FATAL:  connection to client lost
immich_postgres          | 2023-05-19 04:14:37.183 UTC [48] STATEMENT:  SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id"  LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId"  LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id"  LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId"
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [NestFactory] Starting Nest application...
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +61ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +7ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:14 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +226ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] InfraModule dependencies initialized +43ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] DomainModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [NestApplication] Nest application successfully started +16ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:15 AM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002
immich_microservices     | [Nest] 1  - 05/19/2023, 4:09:33 AM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:31 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" for assetId 5239556c-9a6c-4439-a1a3-213efa45e288 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/IMG_9775.jpg
immich_microservices     | QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
immich_microservices     |     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
immich_microservices     |     at runMicrotasks (<anonymous>)
immich_microservices     |     at processTicksAndRejections (node:internal/process/task_queues:96:5)
immich_microservices     |     at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33)
immich_microservices     |     at async updateSubject (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:380:38)
immich_microservices     |     at async Promise.all (index 0)
immich_microservices     |     at async SubjectExecutor.executeUpdateOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:422:9)
immich_microservices     |     at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:99:9)
immich_microservices     |     at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:148:21)
immich_microservices     |     at async AssetRepository.save (/usr/src/app/dist/apps/microservices/libs/infra/src/repositories/asset.repository.js:59:24)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:32 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId f84a6e14-2527-4f4b-9abf-83444c1094e1 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/57FE4E88-C045-4A4C-952A-89C7381CA2F1.jpg
immich_microservices     | TypeError: exifDate.toDate is not a function
immich_microservices     |     at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33)
immich_microservices     |     at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:32 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId 03b1f0d5-fe69-468b-8cf3-686969f93e1b at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/7F104CDE-AA65-4348-8671-24D3597368BD.jpg
immich_microservices     | TypeError: exifDate.toDate is not a function
immich_microservices     |     at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33)
immich_microservices     |     at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:32 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId accba6fe-7635-43bd-b739-98b06113da39 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/19D8942E-80AD-45B0-A110-5E256DDFDD0D.jpg
immich_microservices     | TypeError: exifDate.toDate is not a function
immich_microservices     |     at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33)
immich_microservices     |     at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:32 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId a95ad113-6144-41d4-927f-6a286076ff93 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/CB0B24F5-2023-408C-922D-51942AF86984.jpg
immich_microservices     | TypeError: exifDate.toDate is not a function
immich_microservices     |     at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33)
immich_microservices     |     at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:32 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId 07a8ed60-95ae-42f5-9d8b-fe8e9a6a45de at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/61FD7963-2E51-48F0-ACC8-760812DC2259.jpg
immich_microservices     | TypeError: exifDate.toDate is not a function
immich_microservices     |     at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33)
immich_microservices     |     at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM    WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=1ead6d25-a189-4ea3-bd92-094437880c94)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM    WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=d66a42e1-dfd3-4511-a340-2cf285dff5a0)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM    WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=f71e04da-735a-47fc-8a1c-ff14f705046f)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM   ERROR [MediaService] Failed to generate thumbnail for asset: 1ead6d25-a189-4ea3-bd92-094437880c94/IMAGE
immich_microservices     | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/1ead6d25-a189-4ea3-bd92-094437880c94.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-21/IMG_5875.heic
immich_microservices     |     at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices     |     at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM   ERROR [MediaService] Failed to generate thumbnail for asset: d66a42e1-dfd3-4511-a340-2cf285dff5a0/IMAGE
immich_microservices     | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/d66a42e1-dfd3-4511-a340-2cf285dff5a0.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/B7A0575E-BF32-49E4-B629-63E6A9C9616E.heic
immich_microservices     |     at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices     |     at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices     | [Nest] 1  - 05/19/2023, 4:10:37 AM   ERROR [MediaService] Failed to generate thumbnail for asset: f71e04da-735a-47fc-8a1c-ff14f705046f/IMAGE
immich_microservices     | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/f71e04da-735a-47fc-8a1c-ff14f705046f.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-01-14/IMG_6597.heic
immich_microservices     |     at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices     |     at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices     | 
immich_microservices     | <--- Last few GCs --->
immich_microservices     | 
immich_microservices     | [1:0x7fc8d4b82330]   189511 ms: Scavenge (reduce) 2042.2 (2081.2) -> 2041.4 (2081.2) MB, 4.8 / 0.0 ms  (average mu = 0.202, current mu = 0.137) allocation failure 
immich_microservices     | [1:0x7fc8d4b82330]   189583 ms: Scavenge (reduce) 2042.2 (2081.2) -> 2041.4 (2081.2) MB, 4.7 / 0.0 ms  (average mu = 0.202, current mu = 0.137) allocation failure 
immich_microservices     | [1:0x7fc8d4b82330]   189659 ms: Scavenge (reduce) 2042.3 (2081.2) -> 2041.5 (2081.4) MB, 4.4 / 0.0 ms  (average mu = 0.202, current mu = 0.137) allocation failure 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- JS stacktrace --->
immich_microservices     | 
immich_microservices     | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [NestFactory] Starting Nest application...
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +48ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +8ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:25 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:26 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +166ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:26 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:26 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:26 AM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:26 AM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [InstanceLoader] InfraModule dependencies initialized +6336ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [InstanceLoader] DomainModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [NestApplication] Nest application successfully started +16ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002
immich_microservices     | [Nest] 1  - 05/19/2023, 4:12:32 AM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
immich_microservices     | 
immich_microservices     | <--- Last few GCs --->
immich_microservices     | 
immich_microservices     | [1:0x7f6c43fc4330]   133079 ms: Scavenge (reduce) 2040.0 (2081.7) -> 2039.2 (2081.9) MB, 4.5 / 0.0 ms  (average mu = 0.251, current mu = 0.264) allocation failure 
immich_microservices     | [1:0x7f6c43fc4330]   133147 ms: Scavenge (reduce) 2040.0 (2081.9) -> 2039.2 (2081.9) MB, 4.3 / 0.0 ms  (average mu = 0.251, current mu = 0.264) allocation failure 
immich_microservices     | [1:0x7f6c43fc4330]   133203 ms: Scavenge (reduce) 2040.1 (2081.9) -> 2039.3 (2081.9) MB, 4.3 / 0.0 ms  (average mu = 0.251, current mu = 0.264) allocation failure 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- JS stacktrace --->
immich_microservices     | 
immich_microservices     | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [NestFactory] Starting Nest application...
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +63ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +6ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:39 AM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [InstanceLoader] InfraModule dependencies initialized +6187ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [InstanceLoader] DomainModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [NestApplication] Nest application successfully started +14ms
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002
immich_microservices     | [Nest] 1  - 05/19/2023, 4:14:46 AM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized


@ShinasShaji commented on GitHub (May 22, 2023): > Yeah, unfortunately this is a limitation right now. If you have the ram for it you can temporarily increase it to queue the job and then reduce it again afterwards Can confirm that doing this works. I had this issue - see my comment at [#2468](https://github.com/immich-app/immich/discussions/2468#discussioncomment-5944529) I eventually got it working by allocating more RAM than I used to - 16 GB did the trick for me. Also see my comment [here](https://github.com/immich-app/immich/discussions/2468#discussioncomment-5950462). I don't know too much about it to confirm or verify, but immich_microservices doesn't seem to be using more than 20-25% of the memory available to the Docker container. Here are my logs; you can see what is probably the database losing connection when immich_microservices crashes, and also see immich_microservices restarting in the end. I've had thumbnail generation failures before (probably due to corrupt image files), so I don't think that is strictly related to the issue. ``` immich_postgres | 2023-05-19 04:12:22.715 UTC [44] LOG: could not send data to client: Connection reset by peer immich_postgres | 2023-05-19 04:12:22.715 UTC [44] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId" immich_postgres | 2023-05-19 04:12:22.715 UTC [44] FATAL: connection to client lost immich_postgres | 2023-05-19 04:12:22.715 UTC [44] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId" immich_postgres | 2023-05-19 04:14:37.181 UTC [48] LOG: could not send data to client: Connection reset by peer immich_postgres | 2023-05-19 04:14:37.181 UTC [48] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId" immich_postgres | 2023-05-19 04:14:37.183 UTC [48] FATAL: connection to client lost immich_postgres | 2023-05-19 04:14:37.183 UTC [48] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."mimeType" AS "AssetEntity_mimeType", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id" AS "33fbcd7ce858667c52e23abedc8270391efd944359d0faafa3a2231d2f8ea46", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."createdAt" AS "3cdfc1759bd649af199210e81688698eeabbe313b6b7dfd4e2264944f72dea1", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."updatedAt" AS "852105b9390e1bad3d77486094f8c49ad01d3be0391b4571a362273e5651a14", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."ownerId" AS "a296dd914287f4e402b7c079114c7ec37e5469e54398b094a01386e0d2bcc95", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."name" AS "ecea6b95bdfc1cde4cfde443d83ce25263233613434d74f8e97a69a2544f7d8", "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."thumbnailPath" AS "42338c4e7b16a4c811985f947dcf19079ef3e8a98e8b48263f02983c9952d74" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38" ON "49c3f422502d7076bb0c9f0ba572c2cb24d40371305c77ca010160bebff8c38"."id"="AssetEntity__AssetEntity_faces"."personId" immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +61ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] ConfigModule dependencies initialized +7ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:14 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +226ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] InfraModule dependencies initialized +43ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] DomainModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [NestApplication] Nest application successfully started +16ms immich_microservices | [Nest] 1 - 05/19/2023, 4:09:15 AM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002 immich_microservices | [Nest] 1 - 05/19/2023, 4:09:33 AM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized immich_microservices | [Nest] 1 - 05/19/2023, 4:10:31 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" for assetId 5239556c-9a6c-4439-a1a3-213efa45e288 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/IMG_9775.jpg immich_microservices | QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" immich_microservices | at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) immich_microservices | at runMicrotasks (<anonymous>) immich_microservices | at processTicksAndRejections (node:internal/process/task_queues:96:5) immich_microservices | at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33) immich_microservices | at async updateSubject (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:380:38) immich_microservices | at async Promise.all (index 0) immich_microservices | at async SubjectExecutor.executeUpdateOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:422:9) immich_microservices | at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:99:9) immich_microservices | at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:148:21) immich_microservices | at async AssetRepository.save (/usr/src/app/dist/apps/microservices/libs/infra/src/repositories/asset.repository.js:59:24) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:32 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId f84a6e14-2527-4f4b-9abf-83444c1094e1 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/57FE4E88-C045-4A4C-952A-89C7381CA2F1.jpg immich_microservices | TypeError: exifDate.toDate is not a function immich_microservices | at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33) immich_microservices | at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:32 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId 03b1f0d5-fe69-468b-8cf3-686969f93e1b at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/7F104CDE-AA65-4348-8671-24D3597368BD.jpg immich_microservices | TypeError: exifDate.toDate is not a function immich_microservices | at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33) immich_microservices | at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:32 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId accba6fe-7635-43bd-b739-98b06113da39 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/19D8942E-80AD-45B0-A110-5E256DDFDD0D.jpg immich_microservices | TypeError: exifDate.toDate is not a function immich_microservices | at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33) immich_microservices | at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:32 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId a95ad113-6144-41d4-927f-6a286076ff93 at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/CB0B24F5-2023-408C-922D-51942AF86984.jpg immich_microservices | TypeError: exifDate.toDate is not a function immich_microservices | at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33) immich_microservices | at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:32 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF TypeError: exifDate.toDate is not a function for assetId 07a8ed60-95ae-42f5-9d8b-fe8e9a6a45de at upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/61FD7963-2E51-48F0-ACC8-760812DC2259.jpg immich_microservices | TypeError: exifDate.toDate is not a function immich_microservices | at exifToDate (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:92:33) immich_microservices | at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:105:36) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=1ead6d25-a189-4ea3-bd92-094437880c94) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=d66a42e1-dfd3-4511-a340-2cf285dff5a0) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM WARN [MediaService] Failed to generate jpeg thumbnail using sharp, trying with exiftool-vendored (asset=f71e04da-735a-47fc-8a1c-ff14f705046f) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM ERROR [MediaService] Failed to generate thumbnail for asset: 1ead6d25-a189-4ea3-bd92-094437880c94/IMAGE immich_microservices | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/1ead6d25-a189-4ea3-bd92-094437880c94.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-21/IMG_5875.heic immich_microservices | at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) immich_microservices | at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM ERROR [MediaService] Failed to generate thumbnail for asset: d66a42e1-dfd3-4511-a340-2cf285dff5a0/IMAGE immich_microservices | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/d66a42e1-dfd3-4511-a340-2cf285dff5a0.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-02-19/B7A0575E-BF32-49E4-B629-63E6A9C9616E.heic immich_microservices | at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) immich_microservices | at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) immich_microservices | [Nest] 1 - 05/19/2023, 4:10:37 AM ERROR [MediaService] Failed to generate thumbnail for asset: f71e04da-735a-47fc-8a1c-ff14f705046f/IMAGE immich_microservices | Error: Output file /usr/src/app/upload/thumbs/f61c1583-ae2d-4262-96ff-e896b0ef9370/f71e04da-735a-47fc-8a1c-ff14f705046f.jpeg already exists for /usr/src/app/upload/library/f61c1583-ae2d-4262-96ff-e896b0ef9370/2023/2023-01-14/IMG_6597.heic immich_microservices | at BinaryExtractionTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29) immich_microservices | at BinaryExtractionTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40) immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | [1:0x7fc8d4b82330] 189511 ms: Scavenge (reduce) 2042.2 (2081.2) -> 2041.4 (2081.2) MB, 4.8 / 0.0 ms (average mu = 0.202, current mu = 0.137) allocation failure immich_microservices | [1:0x7fc8d4b82330] 189583 ms: Scavenge (reduce) 2042.2 (2081.2) -> 2041.4 (2081.2) MB, 4.7 / 0.0 ms (average mu = 0.202, current mu = 0.137) allocation failure immich_microservices | [1:0x7fc8d4b82330] 189659 ms: Scavenge (reduce) 2042.3 (2081.2) -> 2041.5 (2081.4) MB, 4.4 / 0.0 ms (average mu = 0.202, current mu = 0.137) allocation failure immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +48ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] ConfigModule dependencies initialized +8ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:25 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:26 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +166ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:26 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:26 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:26 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled immich_microservices | [Nest] 1 - 05/19/2023, 4:12:26 AM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [InstanceLoader] InfraModule dependencies initialized +6336ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [InstanceLoader] DomainModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [NestApplication] Nest application successfully started +16ms immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002 immich_microservices | [Nest] 1 - 05/19/2023, 4:12:32 AM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | [1:0x7f6c43fc4330] 133079 ms: Scavenge (reduce) 2040.0 (2081.7) -> 2039.2 (2081.9) MB, 4.5 / 0.0 ms (average mu = 0.251, current mu = 0.264) allocation failure immich_microservices | [1:0x7f6c43fc4330] 133147 ms: Scavenge (reduce) 2040.0 (2081.9) -> 2039.2 (2081.9) MB, 4.3 / 0.0 ms (average mu = 0.251, current mu = 0.264) allocation failure immich_microservices | [1:0x7f6c43fc4330] 133203 ms: Scavenge (reduce) 2040.1 (2081.9) -> 2039.3 (2081.9) MB, 4.3 / 0.0 ms (average mu = 0.251, current mu = 0.264) allocation failure immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +63ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] ConfigModule dependencies initialized +6ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +176ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled immich_microservices | [Nest] 1 - 05/19/2023, 4:14:39 AM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [InstanceLoader] InfraModule dependencies initialized +6187ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [InstanceLoader] DomainModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [NestApplication] Nest application successfully started +14ms immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.56.0 - Listening on port: 3002 immich_microservices | [Nest] 1 - 05/19/2023, 4:14:46 AM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized ```
Author
Owner

@MattPhilpot commented on GitHub (May 22, 2023):

I eventually got it working by allocating more RAM than I used to - 16 GB did the trick for me.

I will confirm increasing memory is the solution as well. Not the same configuration as @florihupf - using k3s - but similar'ish library size of 80k assets and ~1.1TB of storage.

Struggles with 8GB, running fine when I reserve 12GB. After the face recognition job is done, I'll probably up that reservation to 16-20GB.

@MattPhilpot commented on GitHub (May 22, 2023): > > > I eventually got it working by allocating more RAM than I used to - 16 GB did the trick for me. > I will confirm increasing memory is the solution as well. Not the same configuration as @florihupf - using k3s - but similar'ish library size of 80k assets and ~1.1TB of storage. Struggles with 8GB, running fine when I reserve 12GB. After the face recognition job is done, I'll probably up that reservation to 16-20GB.
Author
Owner

@jrasm91 commented on GitHub (May 22, 2023):

Looks like @michelheusschen is implementing pagination for these "all" jobs, which should significantly reduce the memory consumption, at least for the all jobs. I believe it is still high in some situations (like full mobile sync)

@jrasm91 commented on GitHub (May 22, 2023): Looks like @michelheusschen is implementing pagination for these "all" jobs, which should significantly reduce the memory consumption, at least for the all jobs. I believe it is still high in some situations (like full mobile sync)
Author
Owner

@florihupf commented on GitHub (May 24, 2023):

Super thanks for the quick fix, works great!

@florihupf commented on GitHub (May 24, 2023): Super thanks for the quick fix, works great!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#869