immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 13 Packages Projects Releases 10 Wiki Activity

[BUG] Files stranded in upload/ folder if error occurs during metadata extraction #787

New Issue
Closed
opened 2026-02-04 22:38:31 +03:00 by OVERLORD · 7 comments
OVERLORD commented 2026-02-04 22:38:31 +03:00
Owner
Copy Link

Originally created by @raisinbear on GitHub (Mar 31, 2023).

The bug

Hi,

if image metadata extraction fails due to corrupt data (example below), the uploaded files stay in the upload folder. IMO because the job STORAGE_TEMPLATE_MIGRATION_SINGLE would be enqueued after the exception had been raised. I'd expect the same behavior for video metadata extraction.
If that is intended behavior, please ignore. But in 1.51.x files were migrated regardless.

Log:

immich_microservices  | [Nest] 1  - 03/30/2023, 7:27:33 AM   ERROR [MetadataExtractionProcessor] Error extracting EXIF QueryFailedError: invalid input syntax for type integer: "{"20"}"
immich_microservices  | QueryFailedError: invalid input syntax for type integer: "{"20"}"
immich_microservices  |     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
immich_microservices  |     at runMicrotasks (<anonymous>)
immich_microservices  |     at processTicksAndRejections (node:internal/process/task_queues:96:5)
immich_microservices  |     at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
immich_microservices  |     at async MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:181:13)
immich_postgres       | 2023-03-30 05:27:35.417 UTC [321] ERROR:  invalid input syntax for type integer: "{"25"}"
immich_postgres       | 2023-03-30 05:27:35.417 UTC [321] CONTEXT:  unnamed portal parameter $17 = '...'
immich_postgres       | 2023-03-30 05:27:35.417 UTC [321] STATEMENT:  INSERT INTO "exif"("assetId", "description", "exifImageWidth", "exifImageHeight", "fileSizeInByte", "orientation", "dateTimeOriginal", "modifyDate", "latitude", "longitude", "city", "livePhotoCID", "state", "country", "make", "model", "imageName", "lensModel", "fNumber", "focalLength", "iso", "exposureTime", "fps", "exifTextSearchableColumn") VALUES ($1, DEFAULT, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, $10, DEFAULT, DEFAULT, $11, $12, $13, $14, $15, $16, $17, $18, DEFAULT, DEFAULT) ON CONFLICT ( "assetId" ) DO UPDATE SET "assetId" = EXCLUDED."assetId", "exifImageWidth" = EXCLUDED."exifImageWidth", "exifImageHeight" = EXCLUDED."exifImageHeight", "fileSizeInByte" = EXCLUDED."fileSizeInByte", "orientation" = EXCLUDED."orientation", "dateTimeOriginal" = EXCLUDED."dateTimeOriginal", "modifyDate" = EXCLUDED."modifyDate", "latitude" = EXCLUDED."latitude", "longitude" = EXCLUDED."longitude", "livePhotoCID" = EXCLUDED."livePhotoCID", "make" = EXCLUDED."make", "model" = EXCLUDED."model", "imageName" = EXCLUDED."imageName", "lensModel" = EXCLUDED."lensModel", "fNumber" = EXCLUDED."fNumber", "focalLength" = EXCLUDED."focalLength", "iso" = EXCLUDED."iso", "exposureTime" = EXCLUDED."exposureTime"  RETURNING "description"

The OS that Immich Server is running on

Debian 11

Version of Immich Server

v1.52.1

Version of Immich Mobile App

v1.52.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
      - typesense
    restart: always

#  immich-machine-learning:
#    container_name: immich_machine_learning
#    image: ghcr.io/immich-app/immich-machine-learning:release
#    volumes:
#      - ${UPLOAD_LOCATION}:/usr/src/app/upload
#      - model-cache:/cache
#    env_file:
#      - .env
#    environment:
#      - NODE_ENV=production
#    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/home/immich/data

###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=[...removed...]
# TYPESENSE_ENABLED=false

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=False

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

Reproduction steps

1. Upload file with invalid / corrupt exif data.
2. Check `upload/` folder for files not being moved to `library/` hierarchy.

Additional information

No response

Originally created by @raisinbear on GitHub (Mar 31, 2023). ### The bug Hi, if image metadata extraction fails due to corrupt data (example below), the uploaded files stay in the upload folder. IMO because the job `STORAGE_TEMPLATE_MIGRATION_SINGLE` would be enqueued after the exception had been raised. I'd expect the same behavior for video metadata extraction. If that is intended behavior, please ignore. But in 1.51.x files were migrated regardless. Log: ``` immich_microservices | [Nest] 1 - 03/30/2023, 7:27:33 AM ERROR [MetadataExtractionProcessor] Error extracting EXIF QueryFailedError: invalid input syntax for type integer: "{"20"}" immich_microservices | QueryFailedError: invalid input syntax for type integer: "{"20"}" immich_microservices | at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) immich_microservices | at runMicrotasks (<anonymous>) immich_microservices | at processTicksAndRejections (node:internal/process/task_queues:96:5) immich_microservices | at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33) immich_microservices | at async MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:181:13) immich_postgres | 2023-03-30 05:27:35.417 UTC [321] ERROR: invalid input syntax for type integer: "{"25"}" immich_postgres | 2023-03-30 05:27:35.417 UTC [321] CONTEXT: unnamed portal parameter $17 = '...' immich_postgres | 2023-03-30 05:27:35.417 UTC [321] STATEMENT: INSERT INTO "exif"("assetId", "description", "exifImageWidth", "exifImageHeight", "fileSizeInByte", "orientation", "dateTimeOriginal", "modifyDate", "latitude", "longitude", "city", "livePhotoCID", "state", "country", "make", "model", "imageName", "lensModel", "fNumber", "focalLength", "iso", "exposureTime", "fps", "exifTextSearchableColumn") VALUES ($1, DEFAULT, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, $10, DEFAULT, DEFAULT, $11, $12, $13, $14, $15, $16, $17, $18, DEFAULT, DEFAULT) ON CONFLICT ( "assetId" ) DO UPDATE SET "assetId" = EXCLUDED."assetId", "exifImageWidth" = EXCLUDED."exifImageWidth", "exifImageHeight" = EXCLUDED."exifImageHeight", "fileSizeInByte" = EXCLUDED."fileSizeInByte", "orientation" = EXCLUDED."orientation", "dateTimeOriginal" = EXCLUDED."dateTimeOriginal", "modifyDate" = EXCLUDED."modifyDate", "latitude" = EXCLUDED."latitude", "longitude" = EXCLUDED."longitude", "livePhotoCID" = EXCLUDED."livePhotoCID", "make" = EXCLUDED."make", "model" = EXCLUDED."model", "imageName" = EXCLUDED."imageName", "lensModel" = EXCLUDED."lensModel", "fNumber" = EXCLUDED."fNumber", "focalLength" = EXCLUDED."focalLength", "iso" = EXCLUDED."iso", "exposureTime" = EXCLUDED."exposureTime" RETURNING "description" ``` ### The OS that Immich Server is running on Debian 11 ### Version of Immich Server v1.52.1 ### Version of Immich Mobile App v1.52.1 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-server.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:release entrypoint: ["/bin/sh", "./start-microservices.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env environment: - NODE_ENV=production depends_on: - redis - database - typesense restart: always # immich-machine-learning: # container_name: immich_machine_learning # image: ghcr.io/immich-app/immich-machine-learning:release # volumes: # - ${UPLOAD_LOCATION}:/usr/src/app/upload # - model-cache:/cache # env_file: # - .env # environment: # - NODE_ENV=production # restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:release entrypoint: ["/bin/sh", "./entrypoint.sh"] env_file: - .env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.0 environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data logging: driver: none volumes: - tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2 restart: always database: container_name: immich_postgres image: postgres:14 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:release environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 logging: driver: none depends_on: - immich-server restart: always volumes: pgdata: model-cache: tsdata: ``` ### Your .env content ```Shell ################################################################################### # Database ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_PASSWORD=postgres DB_DATABASE_NAME=immich # Optional Database settings: # DB_PORT=5432 ################################################################################### # Redis ################################################################################### REDIS_HOSTNAME=immich_redis # Optional Redis settings: # REDIS_PORT=6379 # REDIS_DBINDEX=0 # REDIS_PASSWORD= # REDIS_SOCKET= ################################################################################### # Upload File Location # # This is the location where uploaded files are stored. ################################################################################### UPLOAD_LOCATION=/home/immich/data ################################################################################### # Typesense ################################################################################### TYPESENSE_API_KEY=[...removed...] # TYPESENSE_ENABLED=false ################################################################################### # Reverse Geocoding # # Reverse geocoding is done locally which has a small impact on memory usage # This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable # This ranges from 0-3 with 3 being the most precise # 3 - Cities > 500 population: ~200MB RAM # 2 - Cities > 1000 population: ~150MB RAM # 1 - Cities > 5000 population: ~80MB RAM # 0 - Cities > 15000 population: ~40MB RAM #################################################################################### # DISABLE_REVERSE_GEOCODING=false # REVERSE_GEOCODING_PRECISION=3 #################################################################################### # WEB - Optional # # Custom message on the login page, should be written in HTML form. # For example: # PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>" #################################################################################### PUBLIC_LOGIN_PAGE_MESSAGE= #################################################################################### # Alternative Service Addresses - Optional # # This is an advanced feature for users who may be running their immich services on different hosts. # It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers. # Note: immich-microservices is bound to 3002, but no references are made #################################################################################### IMMICH_WEB_URL=http://immich-web:3000 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=False #################################################################################### # Alternative API's External Address - Optional # # This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery. # You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash. # NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api # Examples: http://localhost:3001, http://immich-api.example.com, etc #################################################################################### #IMMICH_API_URL_EXTERNAL=http://localhost:3001 ``` ### Reproduction steps ```bash 1. Upload file with invalid / corrupt exif data. 2. Check `upload/` folder for files not being moved to `library/` hierarchy. ``` ### Additional information _No response_
OVERLORD commented 2026-02-04 22:38:40 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 31, 2023):

@jrasm91 FYI

@alextran1502 commented on GitHub (Mar 31, 2023): @jrasm91 FYI
OVERLORD commented 2026-02-04 22:38:45 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

This is an accurate observation. Migration happens on success of exif extraction now, where it previously happened before. It was not intended, but seems like not a bad feature 😛. Basically files stay in the upload folder unless extraction is successful. If they have a problem, it there is probably some intervention required and they should be easier to locate now.

@jrasm91 commented on GitHub (Mar 31, 2023): This is an accurate observation. Migration happens on success of exif extraction now, where it previously happened before. It was not intended, but seems like not a bad feature :stuck_out_tongue:. Basically files stay in the upload folder unless extraction is successful. If they have a problem, it there is probably some intervention required and they should be easier to locate now.
OVERLORD commented 2026-02-04 22:38:50 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

I'm happy with the new functionality and unless there's some other downside to this I think we should keep it as is.

@jrasm91 commented on GitHub (Mar 31, 2023): I'm happy with the new functionality and unless there's some other downside to this I think we should keep it as is.
OVERLORD commented 2026-02-04 22:38:57 +03:00
Author
Owner
Copy Link

@raisinbear commented on GitHub (Mar 31, 2023):

I'm happy with the new functionality and unless there's some other downside to this I think we should keep it as is.

It‘s actually not a bad idea to have such files separated. It merely got in the way of me regularly cleaning up uploads/ which usually is full of half uploaded junk from interrupted background uploads or iOS setting my device to sleep during foreground backup 😅

@raisinbear commented on GitHub (Mar 31, 2023): > I'm happy with the new functionality and unless there's some other downside to this I think we should keep it as is. It‘s actually not a bad idea to have such files separated. It merely got in the way of me regularly cleaning up uploads/ which usually is full of half uploaded junk from interrupted background uploads or iOS setting my device to sleep during foreground backup 😅
OVERLORD commented 2026-02-04 22:39:01 +03:00
Author
Owner
Copy Link

@raisinbear commented on GitHub (Mar 31, 2023):

Maybe in the long run a separate folder and space in web / mobile app could be beneficial to point people at files having issues

@raisinbear commented on GitHub (Mar 31, 2023): Maybe in the long run a separate folder and space in web / mobile app could be beneficial to point people at files having issues
OVERLORD commented 2026-02-04 22:39:05 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 31, 2023):

Maybe in the long run a separate folder and space in web / mobile app could be beneficial to point people at files having issues

That's another good idea.

@jrasm91 commented on GitHub (Mar 31, 2023): > Maybe in the long run a separate folder and space in web / mobile app could be beneficial to point people at files having issues That's another good idea.
OVERLORD commented 2026-02-04 22:39:09 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Apr 1, 2023):

Close this one as the expected behavior

@alextran1502 commented on GitHub (Apr 1, 2023): Close this one as the expected behavior
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#787
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?