immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

Server: Storage layer refactor #442

New Issue
Closed
opened 2026-02-04 20:29:41 +03:00 by OVERLORD · 9 comments
OVERLORD commented 2026-02-04 20:29:41 +03:00
Owner
Copy Link

Originally created by @bo0tzz on GitHub (Nov 23, 2022).

Feature detail

At the moment, Immich does not have an abstracted storage layer. On upload, files are stored in the semi-hardcoded library path with a randomly generated filename, their path is stored in the database, and in any future (read) operations this stored path is used (file serving, thumbnail generation, etc).
For several of the features we're meaning to (potentially) implement in the future (eg #34, #418, #451), it will be very helpful to refactor and abstract the storage layer. For some of them, like supporting multiple storage backends, it will be necessary entirely. In this issue I want to propose a design, although it will need some more discussion and refinement before it will be complete.

As mentioned above, currently the storage path for a file is generated once and stored in the database. I propose that we instead move to a model where storage paths are built on the fly based on the data we have for an asset. We already use some of that data to build the path on upload right now:

const originalUploadFolder = join(basePath, req.user.id, 'original', sanitizedDeviceId);

Instead, when trying to write or read an asset, the storage layer would expose a function for that which accepts the AssetEntity (or a more limited set of data, if desired). The storage implementation then uses that internally, together with some configuration, to build the actual path. That way, things like the storage path become an implementation detail that does not need to be exposed to the rest of Immich.

I think it would be good to keep the storage providers as self-contained as we can, and avoid having it do things like access the database. Instead, it would take in a configuration when initializing (eg, the root path where to store files, S3 access credentials, or a template for the filename). That configuration can of course be read from the database by whatever code initializes the provider.

This will allow for a multitude of nice things:

  • Storage backends become swappable. We can then relatively easily add support for storing files on S3, etc.
  • We can use multiple storage backends for different file types (original file, thumbnail, etc), even on different mediums
    (eg thumbnails on disk, originals on S3)
  • Code for things like path templating can be cleanly contained inside a single storage provider
  • Migrations between storage types! Just instantiate both the old and the new configuration, and copy between them. No need to even update the database.
  • Probably other stuff :)

tbd:

  1. What interface does a storage provider need? Probably at least create, delete and stat. How about something like S3, which might be able to provide URLs for direct access (bypassing immich)?
  2. The description here (partly) covers multiple features. What is the exact scope of the initial refactor (and how do we anticipate those future features in it)?
  3. ???

Platform

Server

Originally created by @bo0tzz on GitHub (Nov 23, 2022). ### Feature detail At the moment, Immich does not have an abstracted storage layer. On upload, [files are stored](https://github.com/immich-app/immich/blob/main/server/apps/immich/src/config/asset-upload.config.ts) in the semi-hardcoded library path with a randomly generated filename, their path is stored in the database, and in any future (read) operations this stored path is used ([file serving](https://github.com/immich-app/immich/blob/main/server/apps/immich/src/api-v1/asset/asset.service.ts#L388), [thumbnail generation](https://github.com/immich-app/immich/blob/main/server/apps/microservices/src/processors/thumbnail.processor.ts#L68), etc). For several of the features we're meaning to (potentially) implement in the future (eg #34, #418, #451), it will be very helpful to refactor and abstract the storage layer. For some of them, like supporting multiple storage backends, it will be necessary entirely. In this issue I want to propose a design, although it will need some more discussion and refinement before it will be complete. As mentioned above, currently the storage path for a file is generated once and stored in the database. I propose that we instead move to a model where storage paths are built on the fly based on the data we have for an asset. We already use some of that data to build the path on upload right now: ```typescript const originalUploadFolder = join(basePath, req.user.id, 'original', sanitizedDeviceId); ``` Instead, when trying to write or read an asset, the storage layer would expose a function for that which accepts the `AssetEntity` (or a more limited set of data, if desired). The storage implementation then uses that internally, together with some configuration, to build the actual path. That way, things like the storage path become an implementation detail that does not need to be exposed to the rest of Immich. I think it would be good to keep the storage providers as self-contained as we can, and avoid having it do things like access the database. Instead, it would take in a configuration when initializing (eg, the root path where to store files, S3 access credentials, or a template for the filename). That configuration can of course be read from the database by whatever code initializes the provider. This will allow for a multitude of nice things: * Storage backends become swappable. We can then relatively easily add support for storing files on S3, etc. * We can use multiple storage backends for different file types (original file, thumbnail, etc), even on different mediums (eg thumbnails on disk, originals on S3) * Code for things like path templating can be cleanly contained inside a single storage provider * Migrations between storage types! Just instantiate both the old and the new configuration, and copy between them. No need to even update the database. * Probably other stuff :) tbd: 1. What interface does a storage provider need? Probably at least `create`, `delete` and `stat`. How about something like S3, which might be able to provide URLs for direct access (bypassing immich)? 2. The description here (partly) covers multiple features. What is the exact scope of the initial refactor (and how do we anticipate those future features in it)? 3. ??? ### Platform Server
OVERLORD added the 🗄️server label 2026-02-04 20:29:41 +03:00
OVERLORD commented 2026-02-04 20:29:45 +03:00
Author
Owner
Copy Link

@Cellivar commented on GitHub (Dec 8, 2022):

If I might be so bold as to add some unsolicited advice..

You're very close to the concept of a generic blob storage interface, where filesystem storage just a slightly weird looking blob storage API. Blob storage for large files, like images, is a very common design pattern for modern networked systems. Though my link hasn't been updated for a few years it's a good example of the way you may want to head in with your implementation. I suspect you can find more modern options for TypeScript out there, searching from my phone is difficult.

Your abstraction layer you described becomes blob operations, which then translate into actual blob API calls (filesystem write, s3 write, NFS share write, etc).

@Cellivar commented on GitHub (Dec 8, 2022): If I might be so bold as to add some unsolicited advice.. You're very close to the concept of [a generic blob storage interface](https://github.com/maxogden/abstract-blob-store), where filesystem storage just a slightly weird looking blob storage API. Blob storage for large files, like images, is a very common design pattern for modern networked systems. Though my link hasn't been updated for a few years it's a good example of the way you may want to head in with your implementation. I suspect you can find more modern options for TypeScript out there, searching from my phone is difficult. Your abstraction layer you described becomes blob operations, which then translate into actual blob API calls (filesystem write, s3 write, NFS share write, etc).
OVERLORD commented 2026-02-04 20:29:49 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 8, 2022):

That's very helpful, thank you!

@bo0tzz commented on GitHub (Dec 8, 2022): That's very helpful, thank you!
OVERLORD commented 2026-02-04 20:30:01 +03:00
Author
Owner
Copy Link

@pinpox commented on GitHub (Jul 11, 2023):

Does immich support S3 Storage currently? I saw a related pr was merged some time ago, but can't figure out how to set it up.

@pinpox commented on GitHub (Jul 11, 2023): Does immich support S3 Storage currently? I saw a related pr was merged some time ago, but can't figure out how to set it up.
OVERLORD commented 2026-02-04 20:30:08 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Jul 11, 2023):

This is probably what you are thinking of: https://github.com/immich-app/immich/discussions/1683#discussioncomment-6206105

@jrasm91 commented on GitHub (Jul 11, 2023): This is probably what you are thinking of: https://github.com/immich-app/immich/discussions/1683#discussioncomment-6206105
OVERLORD commented 2026-02-04 20:30:11 +03:00
Author
Owner
Copy Link

@ibotty commented on GitHub (Aug 28, 2024):

I just wanted to point to Apache OpenDAL which is used in the big data ecosystem quiet a bit. It is a unified storage layer supporting many different storage systems, among it s3 and local posix file systems.

It also has node bindings.

https://opendal.apache.org/docs/nodejs/

@ibotty commented on GitHub (Aug 28, 2024): I just wanted to point to Apache OpenDAL which is used in the big data ecosystem quiet a bit. It is a unified storage layer supporting many different storage systems, among it s3 and local posix file systems. It also has node bindings. https://opendal.apache.org/docs/nodejs/
OVERLORD commented 2026-02-04 20:30:18 +03:00
Author
Owner
Copy Link

@algora-pbc commented on GitHub (Aug 2, 2025):

💎 nycterent is offering a $100 bounty for this issue. View and reward the bounty at algora.io/immich-app/immich/issues/1011

👉 Got a pull request resolving this? Claim the bounty by commenting /claim #1011 in your PR and joining algora.io

@algora-pbc commented on GitHub (Aug 2, 2025): 💎 **nycterent** is offering a **$100** bounty for this issue. View and reward the bounty at `algora.io/immich-app/immich/issues/1011` 👉 Got a pull request resolving this? Claim the bounty by commenting `/claim #1011` in your PR and joining `algora.io`
OVERLORD commented 2026-02-04 20:30:20 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Aug 3, 2025):

We don't want open bounties like this, so I've hidden the above comment. The zig project explains it better than I could at https://ziglang.org/news/bounties-damage-open-source-projects/.

@bo0tzz commented on GitHub (Aug 3, 2025): We don't want open bounties like this, so I've hidden the above comment. The zig project explains it better than I could at https://ziglang.org/news/bounties-damage-open-source-projects/.
OVERLORD commented 2026-02-04 20:30:27 +03:00
Author
Owner
Copy Link

@mikes1991gh commented on GitHub (Aug 7, 2025):

💯

It's been about a year since any update here or for https://github.com/immich-app/immich/issues/4445 so I think a lot of people would appreciate any kind of update. Not necessarily that it is happening tomorrow but some sign if this (new storage layer and S3/backblaze/etc compat in particular) is something that might be considered in the medium term.

@mikes1991gh commented on GitHub (Aug 7, 2025): 💯 It's been about a year since any update here or for https://github.com/immich-app/immich/issues/4445 so I think a lot of people would appreciate any kind of update. Not necessarily that it is happening tomorrow but some sign if this (new storage layer and S3/backblaze/etc compat in particular) is something that might be considered in the medium term.
OVERLORD commented 2026-02-04 20:30:28 +03:00
Author
Owner
Copy Link

@unedited-despair commented on GitHub (Aug 19, 2025):

The ownCloud OCIS server comes with a pretty advanced thougt model / architecture / implementation of a storage layer.
It's a different technology stack, but could still serve as good inspiration:

  1. Functional concept - Federated storage: This could be directly applicable to Immich
  2. Runtime concept - Cloud Storage Services for Synchronization and Sharing (CS3): The OCIS implementation revolves around REVA, the reference client for CS3. REVA supports local filesystems as well as S3 for example. See for example: 20febffaa7/services/storage-users/pkg/revaconfig/drivers.go (L271)
  3. Example: Storage config for S3: https://doc.owncloud.com/ocis/next/deployment/storage/s3.html
  4. Full list of REVA's storage drivers incl. explanatory remarks / upsides / downsides: https://github.com/cs3org/reva/wiki/storages#storage-driver-implementations-in-reva

I second @mikes1991gh's feedback, would be nice to get some hint about the plans in regard to the storage layer refactor.

@unedited-despair commented on GitHub (Aug 19, 2025): The ownCloud OCIS server comes with a pretty advanced thougt model / architecture / implementation of a storage layer. It's a different technology stack, but could still serve as good inspiration: 1. Functional concept - [Federated storage](https://doc.owncloud.com/ocis/next/architecture/architecture.html#federated-storage): _This could be directly applicable to Immich_ 2. Runtime concept - [Cloud Storage Services for Synchronization and Sharing (CS3)](https://doc.owncloud.com/ocis/next/architecture/architecture.html#reva-and-cs3): The OCIS implementation revolves around REVA, the reference client for CS3. REVA supports local filesystems as well as S3 for example. See for example: https://github.com/owncloud/ocis/blob/20febffaa7f68bfc0ced1ca76b5414df8afc6bf7/services/storage-users/pkg/revaconfig/drivers.go#L271 3. Example: Storage config for S3: https://doc.owncloud.com/ocis/next/deployment/storage/s3.html 4. Full list of REVA's storage drivers incl. explanatory remarks / upsides / downsides: https://github.com/cs3org/reva/wiki/storages#storage-driver-implementations-in-reva I second @mikes1991gh's feedback, would be nice to get some hint about the plans in regard to the storage layer refactor.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#442
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?