Commit Graph

1337 Commits

Author SHA1 Message Date
Jonathan Jogenfors
068e703e88 feat(server): Automatic watching of library folders (#6192)
* feat: initial watch support

* allow offline files

* chore: ignore query errors when resetting e2e db

* revert db query

* add savepoint

* guard the user query

* chore: openapi and db migration

* wip

* support multiple libraries

* fix tests

* wip

* can now cleanup chokidar watchers

* fix unit tests

* add library watch queue

* add missing init from merge

* wip

* can now filter file extensions

* remove watch api from non job client

* Fix e2e test

* watch library with updated import path and exclusion pattern

* add library watch frontend ui

* case sensitive watching extensions

* can auto watch libraries

* move watcher e2e tests to separate file

* don't watch libraries from a queue

* use event emitters

* shorten e2e test timeout

* refactor chokidar code to filesystem provider

* expose chokidar parameters to config file

* fix storage mock

* set default config for library watching

* add fs provider mocks

* cleanup

* add more unit tests for watcher

* chore: fix format + sql

* add more tests

* move unwatch feature back to library service

* add file event unit tests

* chore: formatting

* add documentation

* fix e2e tests

* chore: fix e2e tests

* fix library updating

* test cleanup

* fix typo

* cleanup

* fixing as per pr comments

* reduce library watch config file

* update storage config and mocks

* move negative event tests to unit tests

* fix library watcher e2e

* make watch configuration global

* remove the feature flag

* refactor watcher teardown

* fix microservices init

* centralize asset scan job queue

* improve docs

* add more tests

* chore: open api

* initialize app service

* fix docs

* fix library watch feature flag

* Update docs/docs/features/libraries.md

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

* fix: import right app service

* don't be truthy

* fix test speling

* stricter library update tests

* move fs watcher mock to external file

* subscribe to config changes

* docker does not need polling

* make library watch() private

* feat: add configuration ui

---------

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-31 08:15:54 +00:00
Markus
87c38d1832 feat(server, web): Added TranscodePolicy "Bitrate higher than max bitrate or not in accepted format" (#6479)
* chore: rebase

* chore: open api

* Add Database-Migration for setting targetCodec as acceptedCodec if it was set by admin

* Add TranscodePolicy setting, to only transcode files with a bitrate higher than set max bitrate

* Rename enum value of TranscodePolicy

* calculate max_bitrate according to "k" and "m" suffix for comparison

* remove migration

* minor changes

* UnitTest for Bitrate Policy

* Fix UnitTest

* Add missing output options

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2024-01-30 20:25:07 -05:00
Ben McCann
9c7dee8551 chore: migrate CLI to ESM and vitest (#6777)
* chore: migrate CLI to ESM and vitest

* fix lint

* update github workflow

* format
2024-01-30 18:23:33 -05:00
martin
1bfef200a5 fix(server): avoid leaking people data on shared links (#6779)
* fix: avoid leaking people data on shared links

* test: add e2e test
2024-01-30 15:34:00 -06:00
t4keda
76f8d030ce added a configuration option to select the dri node in transcoding (#6376)
* added a configuration option to select the dri node in transcoding

* chore: open api

* refactor: get hawrdware device

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-29 20:40:02 -05:00
Mert
ae7f174948 refactor: rename clip -> smart search (#6713) 2024-01-29 09:51:22 -05:00
Mert
e5a70329c9 fix(server): HEVC videos not playing on Apple devices (#6712)
* add hvc1 tag

* formatting
2024-01-28 21:17:20 -05:00
Mert
0770ad15e5 fix(server): do not process faces of deleted assets (#6710)
* return if asset is not found

* add unit test

* formatting
2024-01-29 01:17:54 +00:00
Mert
28806d03e3 fix(server): fix person pagination when deleting (#6707)
fix person delete pagination
2024-01-28 17:02:01 -05:00
martin
fa0913120d feat(web,server): search people (#5703)
* feat: search peoples

* fix: responsive design

* use existing count

* generate sql file

* fix: tests

* remove visible people

* fix: merge, hide...

* use component

* fix: linter

* chore: regenerate api

* fix: change name when searching for a face

* save search

* remove duplicate

* use enums for query parameters

* fix: increase to 20 for the local search

* use constants

* simplify

* fix: number of people more visible

* fix: merge

* fix: search

* fix: loading spinner position

* pr feedback
2024-01-27 19:54:31 -05:00
Daniel Dietzler
e9333837bf chore(server): use fs to read package.json (#6692)
use fs readsync instead of js import for reading package json
2024-01-27 17:39:33 -05:00
Zack Pollard
25cad79657 refactor: move asset stacks to their own entity (#6353)
* feat: auto-stack burst photos

* feat: move stacks to asset stack entity

* chore: pin node version with volta in server

* chore: update e2e cases

* chore: cleanup

* feat: migrate existing stacks

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-27 18:52:14 +00:00
Markus
8aef92affc feat(server, web): accepted codecs (#6460)
* chore: rebase

* chore: open api

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-26 12:02:56 -05:00
Jason Rasmussen
96b7885583 refactor(server): trash endpoints (#6652)
* refactor(server): trash endpoints

* chore: open api

* chore: fix wrong rename
2024-01-26 11:48:37 -05:00
Jason Rasmussen
7ea55c7236 refactor(server): download endpoints (#6653)
* refactor(server): download controller

* chore: open api

* chore: fix mobile references
2024-01-26 08:19:13 -06:00
Jason Rasmussen
b306cf564e refactor(server): move asset detail endpoint to new controller (#6636)
* refactor(server): move asset by id to new controller

* chore: open api

* refactor: more consolidation

* refactor: asset service
2024-01-25 12:52:21 -05:00
Jason Rasmussen
7fc4abba72 feat(server): sql access checks (#6635) 2024-01-25 10:14:38 -05:00
Mert
bd87eb309c feat(server): optimize partial facial recognition (#6634)
* optimize partial facial recognition

* add tests

* use map

* bulk insert faces
2024-01-25 01:27:39 -05:00
Jason Rasmussen
a00768c9e5 chore(server): remove old device id endpoint (#6578)
* chore: remove old endpoint

* chore: open api

* chore: remove old tests
2024-01-22 20:54:53 -06:00
Aram Akhavan
a972dd4060 fix(server): extraction of Samsung Motionphoto videos (#6337)
* Fix extraction of samsung motionphoto videos

* Refactor binary tag extraction to the repository to consolidate exiftool usage

* format

* fix linting and swap argument orders

* Fix tag name and conditional order

* Add unit test

* Update server test assets submodule

* Remove old motion photo video assets when a new one is extracted

* delete first, then write

* Include motion photo asset uuid's in the filename

If the filenames are not uniquified, then we can't delete old/corrupt ones

* Fix formatting and fix/add tests

* chore: only use new uuid

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-22 13:04:45 -05:00
Jason Rasmussen
7b314f9435 chore(server): sort open api params (#6484)
* chore: sort spec

* chore: open api

* chore(mobile): sort auditDeletes params

---------

Co-authored-by: shenlong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
2024-01-22 11:49:51 -05:00
Jason Rasmussen
bd2dbb4944 fix(web): always use websocket transport (#6564) 2024-01-22 11:37:00 -05:00
Daniel Dietzler
607fd39130 fix(server): only calculate quota usage for internal assets (#6556)
only calculate usage for internal assets
2024-01-21 15:48:29 -06:00
Mert
311261bd4e fix(server): disable sharp file caching (#6542)
don't cache files
2024-01-20 23:10:14 -05:00
Mert
c8b33c00ec fix(server): use crf-based two pass for vp9 if max bitrate is disabled (#6535)
use crf-based two pass for vp9 if max bitrate is disabled
2024-01-20 15:05:08 -05:00
martin
17eaeb695e feat: smart merge (#6508)
* pr feedback

* fix: tests

* update assets statistics

* pr feedback

* pr feedback

* fix: linter

* pr feedback

* fix: don't limit the smart merge

* pr feedback

* fix: server code

* remove slider

* fix: tests

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-19 17:52:26 +00:00
Alex
07b874edda fix(web): revert smart merge (#6504)
* revert smart merge

* fix test

* fix test

* Remove Slider file
2024-01-19 11:34:20 -06:00
haosu
660b2e908d feat(format): hif format (#6477) 2024-01-18 11:18:56 -06:00
Mert
68f52818ae feat(server): separate face clustering job (#5598)
* separate facial clustering job

* update api

* fixed some tests

* invert clustering

* hdbscan

* update api

* remove commented code

* wip dbscan

* cleanup

removed cluster endpoint

remove commented code

* fixes

updated tests

minor fixes and formatting

fixed queuing

refinements

* scale search range based on library size

* defer non-core faces

* optimizations

removed unused query option

* assign faces individually for correctness

fixed unit tests

remove unused method

* don't select face embedding

update sql

linting

fixed ml typing

* updated job mock

* paginate people query

* select face embeddings because typeorm

* fix setting face detection concurrency

* update sql

formatting

linting

* simplify logic

remove unused imports

* more specific delete signature

* more accurate typing for face stubs

* add migration

formatting

* chore: better typing

* don't select embedding by default

remove unused import

* updated sql

* use normal try/catch

* stricter concurrency typing and enforcement

* update api

* update job concurrency panel to show disabled queues

formatting

* check jobId in queueAll

fix tests

* remove outdated comment

* better facial recognition icon

* wording

wording

formatting

* fixed tests

* fix

* formatting & sql

* try to fix sql check

* more detailed description

* update sql

* formatting

* wording

* update `minFaces` description

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-18 00:08:48 -05:00
Mert
9a2fa21b28 fix(server): scale transcoded videos if dimensions are odd (#6461)
scale if odd resolution
2024-01-17 22:16:44 -05:00
Steven Carter
d4146e3e6d feat(server): provide the ability to search archived photos (#6332)
* Feat: provide the ability to search archived photos

Adds a query parameter (`searchArchived`) to the search URL parameters
to allow the results to contain archived photos.

* chore: rename includeArchived => withArchived

* chore: open api

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-18 02:08:00 +00:00
martin
f0b328fb6b feat(server, web): smart merge (#5796)
* pr feedback

* fix: tests

* update assets statistics

* pr feedback

* pr feedback

* fix: linter

* pr feedback

* fix: don't limit the smart merge

* pr feedback

* fix: server code

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-18 01:52:11 +00:00
Alex
c55503496f fix(server): set log level of immich-admin process in boostrap function (#6458)
* fix(server): set log level of immich-admin process in boostrap function

* Remove log
2024-01-17 16:47:46 -05:00
Jason Rasmussen
6f291006e4 fix(server): handle 5 digit years (#6457) 2024-01-17 16:08:38 -05:00
renovate[bot]
0350058689 fix(deps): update server (#6415)
* fix(deps): update server

* chore: fix tests

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-17 18:24:51 +00:00
Alex
78de4f1312 feat(mobile): quota (#6409)
* feat(mobile): quota

* openapi

* user entity update

* Render quota

* refresh usage upon opening the app bar

* stop backup when quota exceed
2024-01-16 20:08:31 -06:00
Jason Rasmussen
abce82e235 fix(server): enable/disable password login on truenas (#6433) 2024-01-16 16:40:09 -05:00
Sushain Cherivirala
7fc1954e2a fix(server): add filename search (#6394)
Fixes https://github.com/immich-app/immich/issues/5982.

There are basically three options:

1. Search `originalFileName` by dropping a file extension from the query
(if present). Lower fidelity but very easy - just a standard index &
equality.
2. Search `originalPath` by adding an index on `reverse(originalPath)`
and using `starts_with(reverse(query) + "/", reverse(originalPath)`. A
weird index & query but high fidelity.
3. Add a new generated column called `originalFileNameWithExtension` or
something. More storage, kinda jank.

TBH, I think (1) is good enough and easy to make better in the future.
For example, if I search "DSC_4242.jpg", I don't really think it matters
if "DSC_4242.mov" also shows up.

edit: There's a fourth approach that we discussed a bit in Discord and
decided we could switch to it in the future: using a GIN. The minor
issue is that Postgres doesn't tokenize paths in a useful (they're a
single token and it won't match against partial components). We can
solve that by tokenizing it ourselves. For example:

```
immich=# with vecs as (select to_tsvector('simple', array_to_string(string_to_array('upload/library/sushain/2015/2015-08-09/IMG_275.JPG', '/'), ' ')) as vec)  select * from vecs where vec @@ phraseto_tsquery('simple', array_to_string(string_to_array('library/sushain', '/'), ' '));
                                      vec
-------------------------------------------------------------------------------
 '-08':6 '-09':7 '2015':4,5 'img_275.jpg':8 'library':2 'sushain':3 'upload':1
(1 row)
```

The query is also tokenized with the 'split-by-slash-join-with-space'
strategy. This strategy results in `IMG_275.JPG`, `2015`, `sushain` and
`library/sushain` matching. But, `08` and `IMG_275` do not match. The
former is because the token is `-08` and the latter because the
`img_275.jpg` token is matched against exactly.
2024-01-15 14:40:28 -06:00
Tom Vincent
984feafb90 fix(server): extract image description (#6344) 2024-01-15 11:19:41 -05:00
Alex
d096caccac chore(web): quota enhancement (#6371)
* chore(web): quota enhancement

* show quota in user table

* update quota for single user ioption

* Add a note how to set unlimited storage

* fixed deletion doesn't update quota

* refactor relation

* fixed test

* re-refactor

* update sql

* fix e2e test

* Update server/src/domain/user/user.service.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* revert e2e test

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-15 09:04:29 -06:00
cfitzw
deb1f970a8 feat(server, web): quotas (#4471)
* feat: quotas

* chore: open api

* chore: update status box and upload error message

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-12 19:43:36 -05:00
Jason Rasmussen
f4edb6c4bd feat(server): track metadata extracted at (#6352) 2024-01-12 19:39:45 -05:00
Jason Rasmussen
2439c5ab57 refactor: open api (#6334) 2024-01-12 07:36:27 -05:00
waclaw66
902977f165 fix(server): exif gps decoding (#6138) 2024-01-10 07:36:54 -05:00
Jason Rasmussen
bf1dd36fa9 refactor(server): split api and jobs into separate e2e suites (#6307)
* refactor: domain and infra modules

* refactor(server): e2e tests
2024-01-09 23:04:16 -05:00
Jason Rasmussen
12dc7c48c9 refactor(server): domain and infra modules (#6301) 2024-01-09 17:07:01 -05:00
Daniel Dietzler
7cc0904273 feat(server): disable onboarding when config file is set (#6256) 2024-01-08 09:22:26 -06:00
Zack Pollard
5a66314ead test: small improvements to database init tests (#6232) 2024-01-07 01:53:09 +00:00
Michael Manganiello
e262298090 fix(server): Split database queries based on PostgreSQL bound params limit (#6034)
* fix(server): Split database queries based on PostgreSQL bound params limit

PostgreSQL uses a 16-bit integer to indicate the number of bound
parameters.

This means that the maximum number of parameters for any query is 65535.
Any query that tries to bind more than that (e.g. searching by a list of
IDs) requires splitting the query into multiple chunks.

This change includes refactoring every Repository that runs queries
using a list of ids, and either flattening or merging results.

Fixes #5788, #5997.

Also, potentially a fix for #4648 (at least based on
[this comment](https://github.com/immich-app/immich/issues/4648#issuecomment-1826134027)).

References:

* https://github.com/typeorm/typeorm/issues/7565
* [PostgreSQL message format - Bind](https://www.postgresql.org/docs/15/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-BIND)

* misc: Create Chunked decorator to simplify implementation

* feat: Add ChunkedArray/ChunkedSet decorators
2024-01-06 20:36:12 -05:00
maxer137
6835d4519a feat(server): add postgres major version check (#6213)
* feat(server): Throw error when PostgreSQL version is not within the supported versions
The pgvecto.rs extension, though not distributed, can be built for PostgreSQL 12 and 13.
An installation of PostgreSQL 12 with the pgvecto.rs extensions installed will not be caught by immich.
This causes immich to attempt to run the database migrations without having a proper environment.
With assertPostgresql the server will throw an error if the PostgreSQL version is not within the supported range.

* Replaced assertion with lesser than comparison
As requested by @zackpollard

* Changed the comparison to use the minPostgresVersion variable.
If we define one we might as well use it. makes changing the versioning later easier

* Added two new tests, modified two existing tests

`should return if minimum supported PostgreSQL and vectors version are installed`:
Check if init returns properly and that getPostgresVersion is called twice

`should thrown an error if PostgreSQL version is below minimum supported version`:
Checks if the init function correctly returns an error

`should suggest image with postgres ${major} if database is ${major}`:
Modified to set MockResolvedValue instead of MockResolvedValueOnce. With the new check we get the PostgreSQL version twice. So it needs to be set during the entire test.

`should not suggest image if postgres version is not in 14, 15 or 16`:
Modified the bounds to [14, 18]. Because values below 14 now will not get called.
Also Modified to call `getPostgresVersion.MockResolvedValueOnce` for twice, because it gets called twice.

* Fixed two mistakes in the jest functions from previous commit #2abcb60

`should thrown an error if PostgreSQL version is below minimum supported version`:
The regex function I wrote mistakingly used the negate function which check that the error *did not* contain the phrase "PostgreSQL". Which is the opposite

`should not suggest image if postgres version is not in 14, 15 or 16`:
confused bounds for a normal javascript array. Changed the test to only check for values above 16. As values below 14 will get thrown out by test `should return if minimum supported PostgreSQL and vectors version are installed`

I apologise for the mistakes in my previous commit.

* Format fix

---------

Co-authored-by: max <wak@vanling.net>
2024-01-06 19:24:09 -05:00