mirror of
https://github.com/immich-app/immich.git
synced 2026-02-25 19:08:38 +03:00
A couple integrity questions #2102
Closed
opened 2026-02-05 05:07:08 +03:00 by OVERLORD
·
17 comments
No Branch/Tag Specified
main
chore/translations
feat/mobile-ocr
feat/library-offline-stats
release/next
fix/timeline-rtl
feat/integrity-checks-izzy
uhthomas/feat-mobile-search-results
uhthomas/fix-mobile-search-results
renovate/flutter
feat/splash-screen-error
feat/mobile-edit-2-server-sync-entity
update-pwa
refactor/star-rating
uhthomas/feat-sort-smart-search
renovate/github-cqlabs-homebrew-dcm-1.x
push-vxwxqoulmxun
push-zlzxxyywnmtr
chore/deduplicate-storage-template-example
fix/download-button
fix/maintenance-reload
feat/video-player
feat/mobile-editing
feat/use-native-clients
refactor/remove-replace-with-upload
push-snrprxmlposz
push-okmnxsumoyzr
uhthomas/chore-mobile-maplibre
uhthomas/mobile-fix-asset-details-album-pop
feat/crawl-wrapper
feat/open-in-browser
push-skvzqoozqkpl
feat/custom-date-range
feat/edit-filters
fix/locale-settings-desc
push-xyozownmuwqp
push-lvyturrtwkrq
push-mvnsqpxklmnu
push-ztrmyrpuwvow
push-rsywxvptwxuv
push-pvvtwywwqzvy
postgres-socketio
feat/pg-queue
proposal/zod
refactor/asset-upload
renovate/connectivity_plus-7.x
better-project-structure
uhthomas/mobile-feat-asset-viewer-details
fix/ml-rocm-build
fix/25803
feat/asset-file-apis
midzelis/wip
push-zpwsovysllvn
push-nwxlpmyzkyrl
feature/bottom-buttons-order
sqlite_thumbs
fix-keep-correct-ios-shared-album-asset
fix-memory-generation-and-display
push-vpxwmwwxwnvw
fix-migration-width-height
revert/prettier-translations
shared-deep-link-handler
feat/thumbnail-native-clients
feat/platform-clients
fix/foreground-cloud-sync
filter-by-person
feat/csp
refactor/sidebar
fix/disable-editing
fix/view-timeline-deeplink
image-zoom-on-slow-connection
fix-consider-dar-for-video-dimension
fix/merged-edited-assets
open-api-fix
feat/create-job-with-dto
use-toast-primary
feat/vitest-4
feat/ios-fastlane-match
match-signing
fix-update-time-update-timeline
feat/modal-routes
feat/panorama-tiles
feature/mobile-view-asset-owner
feat/system-settings
feature/show-activity-count
better-info-in-asset-viewer
fix/all-people-count
feat/location-favorites
feature/rearrange-buttons-2
fix/download-storage-template
feat/kb-shortcuts-mobile
fix/people-count
push-qolzzzzxrvvn
chore/originals-in-asset-files
feat/asset-size-columns
ben/tree-a11y
new-search-filter-ui
refactor/expectSelectedReadonly
refactor/mobile-grdb
push-qvuktpxmkknu
feat/mobile-native-local-sync
refactor/timeline_ops
fix/scrubber_end
feat/version.txt
feat/context-menus
feat/server-chunked-uploads
refactor/virtualsegment
refactor/rename_daymonth_groups
fix/restrict-android-bg-worker
feat/android-periodic-worker
fix-remote-sync-clean-up
refactor/timeline_move_ops
fix/timeline_split_selectable
feat/keyboard_actions_help_modal
feat/static_frontend
feat/notification-warnign-android
feat/plugins2
feat/plugins
test/create-workflow-token-action
fix/docs-force
debug/search-result-similarity
debug/cf-chunked-uploads
feat/eslint_rule
feat/search-filter-album/web
refactor/timeline_photostream
refactor/timelineasset_asset
feat/session-permissions
feat/timeline_photostream_assetnav
feat/timeline_minor_optimize
feat/timeline_perf_nocomp
feat/timeline_search_results_actions
feat/timeline_search_results_page
fix/timeline_padding
fix/timeline_search_reactivity_warnings
feat/timeline_scrollbar
feat/timeline_stream_withviewer
fix/timeline_back_forth_nav
refactor/timeline_photostream_component
fix/generated-files-checks
fix/locate-button-local
chore/base-image-mimalloc
refactor/timeline_assetlayout
refactor/timeline_selectable
refactor/timeline_aware_actions
refactor/timeline_monthsegment
feat/remove-old-pages
chore/deps-gradle
tmp_photostream
tmp/lcms
feat/mobile-dynamic-thumbnails
fix/mobile-finer-thumbnail-concurrency
refactor/timeline1
refactor/extract_photostream
refactor/rename_load_api
refactor/timeline2
refactor/timeline3
feat/multi-select-asset-viewer
feat-no-thumbhash-cache
refactor/asset_grid
feat/faster-access-checks
fix/18991
fix/19543
chore/temp-remove
fix/21419
feat/mobile-hdr-images
chore/update-mise-lockfile
feat/mise-server-checks
feat/mise-ci
feat/windows-2025
feat/dev_cli
refactor/mobile-migrate-clients
fix/map-theme
fix/require-checkbox
chore/use_swc
feat/efficient-thumbnail-decoding
refactor/mobile-thumbhash
refactor/mobile-thumbhash-new
feat/beta-background-upload
fix/beta-timeline-memories-setting
fix/failed-uploads-not-removed
feat/mobile-shared-album
feat/groups
drift-map-page
drift-auth-user-sync
fix/disable-memory
feat/add-to-album-action
edit-date-time-action
drift-people-page
sqlite-remove-isIn
chore/required-reviewers
refact/asset-manager
fix/folder-sort
pnpm
feat/widget-multiple-server-urls
chore/medium-tests-dbname
fix/web-no-iterator-find
fix/map-pan-interruption
track-livephotos
timeline_events
chore/oxlint-migration
feat/maintenance-worker
feat/dav
chore/demo-snapshot
refactor/server-side-dedupe
feat/integrity-checks
dev/recognition-eval
lighter_buckets_test
perf/postgres-queue
postgres-queue
focus_rings
refactor/web-stores-1
refactor/add-to-taken
feat/sort-places
vet
tmp/demo-snapshot-preview
fix/server-migration-file-extension
fix/asset-update-race-condition
rknn-toolkit-lite2
refactor/mobile-split-up-search-page
feature/Add-rocm-support-for-machine-learning
feat/rocm
chore/async-hash-file
feat/shared-link-view-count
feat/rotation
feat/graphql
feat/job-ids
feat/ignore-library-permission-error
feat/docker-compose-builder
feat/kysely-typeorm
mobile/onboarding
no-video-player
fix/server-qsv-output-format
chore/server-geodata-tweaks
mobile/native-video-player-no-hero
feat/xxhash
fix/docs-concurrency
feat/local-tileserver
refactor/exif-orientation
original-path-infix
refactor/mobile/login-form-1
feat/server-editor-endpoints
fix/server-qsv-vbr
fix-mobile-db-problems
feat/ml-armnn-conversion
feat/mobile/backup-with-album-info
feat/fast-initial-sync-1
chore/handle-output_dims
feat/unassign-faces
feat/shortcuts-on-asset-grid
feat/capacitor-mobile-app-poc
feat/server-nvenc-hw-decoding
fix/mobile-fetch-non-archive
web/automation-ui
feat/mobile-server-endpoint-save-dropdown
object-storage
feat/memories-animations
dev/metrics
ml/tflite
feat/ml-export-cli
v2.5.6
v2.5.5
v2.5.4
v2.5.3
v2.5.2
v2.5.1
v2.5.0
v2.4.1
v2.4.0
v2.3.1
v2.3.0
v2.2.3
v2.2.2
v2.2.1
v2.2.0
v2.1.0
v2.0.1
v2.0.0
v1.144.1
v1.144.0
v1.143.1
v1.143.0
v1.142.1
v1.142.0
v1.141.1
v1.141.0
v1.140.1
v1.140.0
v1.139.4
v1.139.3
v1.139.2
v1.139.1
v1.139.0
v1.138.1
v1.138.0
v1.137.3
v1.137.2
v1.137.1
v1.137.0
v1.136.0
v1.135.3
v1.135.2
v1.135.1
v1.135.0
v1.134.0
v1.133.1
v1.133.0
v1.132.3
v1.132.2
v1.132.1
v1.132.0
v1.131.3
v1.131.2
v1.131.1
v1.131.0
v1.130.3
v1.130.2
v1.130.1
v1.130.0
v1.129.0
v1.128.0
v1.127.0
v1.126.1
v1.126.0
v1.125.7
v1.125.6
v1.125.5
v1.125.4
v1.125.3
v1.125.2
v1.125.1
v1.125.0
v1.124.2
v1.124.1
v1.124.0
v1.123.0
v1.122.3
v1.122.2
v1.122.1
v1.122.0
v1.121.0
v1.120.2
v1.120.1
v1.120.0
v1.119.1
v1.119.0
v1.118.2
v1.118.1
v1.118.0
v1.117.0
v1.116.2
v1.116.1
v1.116.0
v1.115.0
v1.114.0
v1.113.1
v1.113.0
v1.112.1
v1.112.0
v1.111.0
v1.110.0
v1.109.2
v1.109.1
v1.109.0
v1.108.0
v1.107.2
v1.107.1
v1.107.0
v1.106.4
v1.106.3
v1.106.2
v1.106.1
v1.106.0
v1.105.1
v1.105.0
v1.104.0
v1.103.1
v1.103.0
v1.102.3
v1.102.2
v1.102.1
v1.102.0
v1.101.0
v1.100.0
v1.99.0
v1.98.2
v1.98.1
v1.98.0
v1.97.0
v1.96.0
v1.95.1
v1.95.0
v1.94.1
v1.94.0
v1.93.3
v1.93.2
v1.93.1
v1.93.0
v1.92.1
v1.92.0
v1.91.4
v1.91.3
v1.91.2
v1.91.1
v1.91.0
v1.90.2
v1.90.1
v1.90.0
v1.89.0
v1.88.2
v1.88.1
v1.88.0
v1.87.0
v1.86.0
v1.85.0
v1.84.0
v1.83.0
v1.82.1
v1.82.0
v1.81.1
v1.81.0
v1.80.0
v1.79.1
v1.79.0
v1.78.1
v1.78.0
v1.77.0
v1.76.1
v1.76.0
v1.75.2
v1.75.1
v1.75.0
v1.74.0
v1.73.0
v1.72.2
v1.72.1
v1.72.0
v1.71.0
v1.70.0
v1.69.0
v1.68.0
v1.67.2
v1.67.1
v1.67.0
v1.66.1
v1.66.0
v1.65.0
v1.64.0
v1.63.2
v1.63.1
v1.63.0
v1.62.1
v1.62.0
v1.61.0
v1.60.0
v1.59.1
v1.59.0
v1.58.0
v1.57.1
v1.57.0
v1.56.2
v1.56.1
v1.56.0
v1.55.1
v1.55.0
v1.54.1
v1.54.0
v1.53.0
v1.52.1
v1.52.0
v1.51.2
v1.51.1
v1.51.0
v1.50.1
v1.50.0
v1.49.0
v1.48.1
v1.48.0
v1.47.3
v1.47.2
v1.47.1
v1.47.0
v1.46.1
v1.46.0
v1.45.0
v1.44.0
v1.43.1
v1.43.0
v1.42.0_65-dev
v1.41.1_64-dev
v1.41.0_64-dev
v1.40.1_63-dev
v1.40.0_63-dev
v1.39.0_61-dev
v1.38.2_60-dev
v1.38.1_60-dev
v1.38.0_60-dev
v1.37.0_58-dev
v1.36.2_56-dev
v1.36.1_55-dev
v1.36.0_55-dev
v1.35.0_54-dev
v1.34.0_53-dev
v1.33.1_52-dev
v1.33.0_52-dev
v1.32.1_51-dev
v1.32.0_50-dev
v1.31.1_49-dev
v1.31.0_49-dev
v1.30.2_48-dev
v1.30.0_46-dev
v1.29.6_45-dev
v1.29.6_44-dev
v1.29.5_44-dev
v1.29.4_44-dev
v1.29.3_43-dev
v1.29.2_43-dev
v1.29.1_43-dev
v1.29.0_42-dev
v1.28.4_41-dev
v1.28.4_42-dev
v1.28.3_41-dev
v1.28.2_40-dev
v1.28.1_39-dev
v1.28.0_38-dev
v1.27.0_37-dev
v1.26.0_36-dev
v1.25.0_35-dev
v1.24.0_34-dev
v1.23.0_33-dev
v1.22.0_32-dev
v1.21.1_31-dev
v1.21.0_31-dev
v1.20.3_30-dev
v1.20.2_30-dev
v1.20.1_30-dev
v1.20.0_30-dev
v1.19.1_29-dev
v1.19.0_29-dev
v1.18.0_27-dev
v1.17.0_25-dev
v1.16.0_23-dev
v1.15.1_21-dev
v1.15.0_21-dev
v1.14.0_21-dev
v1.13.0_20-dev
v1.12.0_18-dev
v1.11.0_17-dev
v1.10.0_15-dev
v1.9.1_14-dev
v1.9.0_13-dev
v1.8.0_12-dev
v1.7.0_11-dev
v1.6.0_10-dev
v1.5.1+9-dev
v1.5.0+8-dev
v1.4.0+7-dev
v1.4.0+6-dev
v1.4.0-dev
v1.3.0-dev
v1.3.1-dev
v0.6-dev
v0.5-dev
v0.4-dev
v0.3-dev
v0.2-dev
first-android-release
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: immich-app/immich#2102
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @andrewdunndev on GitHub (Feb 1, 2024).
The bug
In discussion with @jrasm91 wanted to enumerate a couple integrity related log errors on my instance, these are likely instance specific "holes" that can be fallen into (especially if you were an early adopter). Tooling for examining instance integrity would be fantastic but realizing that it's potentially a lower priority than, say, search at this time.
On this instance all jobs have been re-run fully on the entire dataset.
I have never manipulated the underlying files or database, these errors in state were achieved through likely having a long running instance and not getting on top of these errors when they initially emerged.
Most likely these are all missing files that have "ghosted" over time. Some excerpt errors, will happily dig into each of these in more detail. There appears to be many (~300 within a ~200k) instances of these two types of errors which looks like missing assets or potentially originating file formats that immich isn't prepared for (e.g. VHS conversions of home movies where I may have ffmpeg'ed poorly):
Thumbnail Generation
Video Transcoding
Likely ~400 of these on my instance. We've always had pixel devices (or "real" cameras) so I think this may be a different bug than the recent #6337.
Other Errors
While browsing I see this, maybe nginx headers not configured properly?
The OS that Immich Server is running on
Centos 9 Stream, Podman, systemd
Version of Immich Server
v1.94.1
Version of Immich Mobile App
v1.94.1.build.121
Platform with the issue
@cgeheim commented on GitHub (Feb 1, 2024):
Same problems here
@dmandrone commented on GitHub (Feb 23, 2024):
Also getting the same issue any time a video is played (see Other Errors section)
@Pheggas commented on GitHub (Mar 18, 2024):
Bump?? These are serious bugs.
@alextran1502 commented on GitHub (Mar 18, 2024):
@Pheggas, I agree, and we have put more effort into hardening the storage template mechanism. I think you and Andrew was one of the first users of Immich so there could have been artifacts from before we put in the hardening process as well. We keep this open to make sure to revisit this mechanism
cc: @zackpollard
@zackpollard commented on GitHub (Mar 18, 2024):
Yea, I would be interested if this issue has gotten worse after our storage migration hardening efforts. On the integrity tooling side, having a automated scan of integrity is something that's on my to-do list. As we have the hash of all files, doing this should be pretty simple overall. The problem is there are many things on my to-do list 😅
@Dmoney517 commented on GitHub (Apr 4, 2024):
I'm seeing the write epipe errors as well. I'm new to the app, is this relates to using the storage template? If I wasn't log watching while setting up and running my imports, I never would have known there was an issue. The videos play and assets load, but it is accompanied by the errors.
@Sid-Sun commented on GitHub (Apr 6, 2024):
I am on v1.101.0
my cluster was set up in August 2023
I was putting labels on my users and doing the migration
even after running full extract metadata job, 3 assets on my server refuse to be migrated:
2 of these have the video transcribe error described here, one does not.
I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on
@zackpollard If there is documentation you could point me toward integrity and give some hints to what the state is and what the integrity tooling would need to address, I can try to take a stab at it.
Thanks <3
@zackpollard commented on GitHub (Apr 6, 2024):
Hey, the overall idea is a tool that will incrementally scan all assets on the server over time, recomputing the hash of the asset on disk and comparing it to the hash in the database to make sure it hasn't changed. However before we do this work we wanted to make changes to the current repair page to improve that so it works in the background too, as for big instances this page consistently times out as it computes everything when you load the page.
@Sid-Sun commented on GitHub (Apr 6, 2024):
I just discovered there is a repair page XD
I see about 413 untracked files on mine at the moment, I have never modified the underlying storage or DB, is this because of the file integrity changing? Is the idea to similarly show the files which have changed integrity over time?
I'll check out how the repair page works & if I have any ideas, thank you! If there is a discussion open you can link me to, I'll catch up on it.
Minor update: One of the assets of mine which couldn't be moved might be a deleted photo that is in the thrash but I haven't visually confirmed it yet.
@brandonkal commented on GitHub (Apr 24, 2024):
I am seeing this EPIPE issue as well. New user since 1.91. I've had to wait 25 seconds for the large image thumbnail to load.
@MiguelNdeCarvalho commented on GitHub (Apr 29, 2024):
Hey,
I have created a brand new instance and I have imported all my data by using the
immich-goCLI utility and when I run Transcode Videos only for Missing items, almost 1200 items, they never get transcoded and always had them back to the queue whenever I click again, here are the logs that I get.@brandonkal commented on GitHub (May 7, 2024):
@MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc)
Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually.
Background: I work in broadcast
@MiguelNdeCarvalho commented on GitHub (May 7, 2024):
Thanks for the explanation!
@schmitzkr commented on GitHub (Aug 24, 2024):
I have noticed a few files with this error
moov atom not foundhave made it to my library. Is there a recommended way to remove these files from immich? Can I delete the files only? or do I need to make changes to the database as well?I have verified outside of immich that the files are very likely corrupted.
@brandonkal commented on GitHub (Aug 24, 2024):
Regarding repairing the files, I’ve had success with Easeus software. You supply it the corrupted files but also known good files in the same format (same recording device, resolution, frame-rate, bitrate etc) and if you are lucky it can recover some of the corrupted files. Because it is corrupted data, it is not a sure thing though.
To remove the files from Immich I believe you should be able to delete them from the UI so they are also marked as removed in the database. If they haven’t been ingested, then a simple disk delete should be fine.
@schmitzkr commented on GitHub (Aug 27, 2024):
Thanks I noticed I can pull them up with the asset Id in the URL, I wasnt able to find them with search so this is helpful.
I have what might be a related question, a lot of my images were taken with a Pixel device and created a lot of motion pictures. Some of them have the corresponding video, others are missing and in the UI the MP play button shows a red exclamation when clicked. I think at some point I lost some of the videos for these MP files, is there a way to decouple the image asset from the video? so they just appears as images?
@jrasm91 commented on GitHub (Sep 4, 2024):
Closing in favor of #12293