[BUG] Internal server error 500 when clicking "Explore" #757

Closed
opened 2026-02-04 22:20:16 +03:00 by OVERLORD · 17 comments
Owner

Originally created by @zyg117 on GitHub (Mar 21, 2023).

The bug

When clicking "Explore" on the web, the server generates an internal error 500.

immich-server shows this log:

[Nest] 1 - 03/21/2023, 8:36:19 AM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not Found
ObjectNotFound: Request failed with HTTP code 404 | Server said: Not Found
at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:338:21)
at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:199:98
at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

image

The OS that Immich Server is running on

Unraid 6.11.5

Version of Immich Server

v1.51.1

Version of Immich Mobile App

v1.50.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

standard, like the official

Your .env content

standard, like the official

Reproduction steps

Updated the containers to v1.51.1
Clicked "Explore" in the web interface.

Additional information

No response

Originally created by @zyg117 on GitHub (Mar 21, 2023). ### The bug When clicking "Explore" on the web, the server generates an internal error 500. immich-server shows this log: [Nest] 1 - 03/21/2023, 8:36:19 AM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not Found ObjectNotFound: Request failed with HTTP code 404 | Server said: Not Found at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:338:21) at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:199:98 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) at processTicksAndRejections (node:internal/process/task_queues:96:5) ![image](https://user-images.githubusercontent.com/36789713/226559399-56ab6d3c-ec14-4eda-b644-aa0d96ed9a9c.png) ### The OS that Immich Server is running on Unraid 6.11.5 ### Version of Immich Server v1.51.1 ### Version of Immich Mobile App v1.50.0 ### Platform with the issue - [X] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML standard, like the official ``` ### Your .env content ```Shell standard, like the official ``` ### Reproduction steps ```bash Updated the containers to v1.51.1 Clicked "Explore" in the web interface. ``` ### Additional information No response
Author
Owner

@JosiahBull commented on GitHub (Mar 21, 2023):

May be a duplicate, see #2028

@JosiahBull commented on GitHub (Mar 21, 2023): May be a duplicate, see #2028
Author
Owner

@zyg117 commented on GitHub (Mar 21, 2023):

May be a duplicate, see #2028

I have checked it already and it's different from that question.

@zyg117 commented on GitHub (Mar 21, 2023): > May be a duplicate, see #2028 I have checked it already and it's different from that question.
Author
Owner

@alextran1502 commented on GitHub (Mar 21, 2023):

@zyg117 Can you add LOG_LEVEL=debug in your Immich environment variable file, then restart the containers and try to access the explore page again then help us capture a screenshot of the log

@alextran1502 commented on GitHub (Mar 21, 2023): @zyg117 Can you add LOG_LEVEL=debug in your Immich environment variable file, then restart the containers and try to access the explore page again then help us capture a screenshot of the log
Author
Owner

@alextran1502 commented on GitHub (Mar 21, 2023):

@florianchevallier can you restart the containers, it maybe the machine learning is not start up properly

@alextran1502 commented on GitHub (Mar 21, 2023): @florianchevallier can you restart the containers, it maybe the machine learning is not start up properly
Author
Owner

@alextran1502 commented on GitHub (Mar 21, 2023):

@kai23 I think your container is not up-to-date

@alextran1502 commented on GitHub (Mar 21, 2023): @kai23 I think your container is not up-to-date
Author
Owner

@jrasm91 commented on GitHub (Mar 21, 2023):

@florianchevallier you should open a separate ticket (or post on discord for faster replies) as you are running into an unrelated issue with your setup.

@jrasm91 commented on GitHub (Mar 21, 2023): @florianchevallier you should open a separate ticket (or post on discord for faster replies) as you are running into an unrelated issue with your setup.
Author
Owner

@kai23 commented on GitHub (Mar 21, 2023):

On my side, it was indeed because I was still using the old repository for the docker images, after using the new ones (from ghcr.io), the 404 went away

@kai23 commented on GitHub (Mar 21, 2023): On my side, it was indeed because I was still using the old repository for the docker images, after using the new ones (from ghcr.io), the 404 went away
Author
Owner

@x-real-ip commented on GitHub (Mar 21, 2023):

I have this same issue when using the search bar.

immich-server log:

[Nest] 1  - 03/21/2023, 8:32:56 PM   ERROR [ExceptionsHandler] Request failed with status code 500
Error: Request failed with status code 500
    at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

immich-machine-learning log:

[2023-03-21 20:32:56,276] ERROR in app: Exception on /sentence-transformer/encode-text [POST]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/src/app/src/main.py", line 53, in clip_encode_text
    model = _get_model(clip_text_model)
  File "/usr/src/app/src/main.py", line 24, in _get_model
    _model_cache[key] = SentenceTransformer(model)
  File "/opt/venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 87, in __init__
    snapshot_download(model_name_or_path,
  File "/opt/venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 476, in snapshot_download
    os.makedirs(nested_dirname, exist_ok=True)
  File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.cache'
10.42.1.245 - - [21/Mar/2023 20:32:56] "POST /sentence-transformer/encode-text HTTP/1.1" 500 -

@x-real-ip commented on GitHub (Mar 21, 2023): I have this same issue when using the search bar. immich-server log: ``` [Nest] 1 - 03/21/2023, 8:32:56 PM ERROR [ExceptionsHandler] Request failed with status code 500 Error: Request failed with status code 500 at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:539:35) at endReadableNT (node:internal/streams/readable:1345:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) ``` immich-machine-learning log: ``` [2023-03-21 20:32:56,276] ERROR in app: Exception on /sentence-transformer/encode-text [POST] Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/usr/src/app/src/main.py", line 53, in clip_encode_text model = _get_model(clip_text_model) File "/usr/src/app/src/main.py", line 24, in _get_model _model_cache[key] = SentenceTransformer(model) File "/opt/venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 87, in __init__ snapshot_download(model_name_or_path, File "/opt/venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 476, in snapshot_download os.makedirs(nested_dirname, exist_ok=True) File "/usr/local/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/local/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/local/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/local/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/.cache' 10.42.1.245 - - [21/Mar/2023 20:32:56] "POST /sentence-transformer/encode-text HTTP/1.1" 500 - ```
Author
Owner

@twitsforbrains commented on GitHub (Mar 22, 2023):

@theautomation I had the same error. As you can see it says permission denied on /.cache. I had passed in a uid into this container just cause I'm paranoid. Removing the uid and letting it be root fixes the problem. The library sentence-transformers used by immich appears to try to create this folder. Unfortunately even when I tried creating this folder in the container with the right permissions I was getting a different error, a 405 error when trying to access https://huggingface.co/api/models/sentence-transformers/clip-ViT-B-32. For simplicity I just removed the user / uid / gid from my docker compose for this container. Hope that was your issue and hope this helps

@twitsforbrains commented on GitHub (Mar 22, 2023): @theautomation I had the same error. As you can see it says permission denied on /.cache. I had passed in a uid into this container just cause I'm paranoid. Removing the uid and letting it be root fixes the problem. The library [sentence-transformers](https://github.com/UKPLab/sentence-transformers/blob/3e1929fddef16df94f8bc6e3b10598a98f46e62d/sentence_transformers/SentenceTransformer.py#L61) used by immich appears to try to create this folder. Unfortunately even when I tried creating this folder in the container with the right permissions I was getting a different error, a 405 error when trying to access https://huggingface.co/api/models/sentence-transformers/clip-ViT-B-32. For simplicity I just removed the user / uid / gid from my docker compose for this container. Hope that was your issue and hope this helps
Author
Owner

@zyg117 commented on GitHub (Mar 22, 2023):

@zyg117 Can you add LOG_LEVEL=debug in your Immich environment variable file, then restart the containers and try to access the explore page again then help us capture a screenshot of the log

Where should I go to set up? Is it this configuration file :
image
image
image

If it's here, I've already added it and restarted as you instructed, but when I accessed that page again, I found that the log output didn't change. Did I set something wrong?

Here is the error log output:

image

And the full log output:
image
image
image
image

@zyg117 commented on GitHub (Mar 22, 2023): > @zyg117 Can you add LOG_LEVEL=debug in your Immich environment variable file, then restart the containers and try to access the explore page again then help us capture a screenshot of the log > Where should I go to set up? Is it this configuration file : ![image](https://user-images.githubusercontent.com/36789713/226814140-e43b4192-c00a-4b3c-a9c6-7594d2d0836f.png) ![image](https://user-images.githubusercontent.com/36789713/226814171-649e4609-926f-4411-b57b-923ca901f082.png) ![image](https://user-images.githubusercontent.com/36789713/226814218-a64aeed5-359c-4aa7-9ee2-54f86e63785b.png) If it's here, I've already added it and restarted as you instructed, but when I accessed that page again, I found that the log output didn't change. Did I set something wrong? Here is the error log output: ![image](https://user-images.githubusercontent.com/36789713/226814734-ef0aeac3-912a-41fa-9084-eceedd82949c.png) And the full log output: ![image](https://user-images.githubusercontent.com/36789713/226814873-9cbce53b-c86e-4b10-b840-ca5c0a60dbf5.png) ![image](https://user-images.githubusercontent.com/36789713/226814901-465d85bb-a242-4430-8d1d-65be1dfe3afc.png) ![image](https://user-images.githubusercontent.com/36789713/226814925-05226fa8-287e-4992-ae44-55f66c14cc02.png) ![image](https://user-images.githubusercontent.com/36789713/226814950-988399b0-f64d-4176-9093-a112550530e9.png)
Author
Owner

@w00tlarr commented on GitHub (Mar 22, 2023):

Hi - I got these errors but resolved it. Here are the steps for me:

  1. Make sure your docker-compose is pulling all images from ghcr.io repo (i.e.: ghcr.io/immich-app/immich-server:release) instead from altran1502.

  2. Have to start containers in order - after docker-compose -d up (tho I use Portainer), I manually shutdown all containers and restart one by one in this order, immich-db, redis, typesense, then wait up to 5 mins before immich-machine-learning, immich-server, immich-microservice. The connection between typesense and immich-server is pretty tight (per release notes), so make sure typesense is up before the server is. [I did comment out the logging: driver: none section of typesense docker-compose to monitor in the docker logs.] This part is a lot of trials and errors for me to get all containers to run healthily - if someone can suggest a docker-compose method to check for dependencies + health checks (not just container starts), please suggest one. Thanks in advance.

  3. Once up, go to the Immich Web UI > Administration > Encode Clip > All (button). Mine ran overnight (~over 100K photos/videos). Trying to search before this is done will give the 500 errors (404 in immich-server logs). At the end, you'll need to have patience for this for large libraries.

  4. Note: I am getting these errors still though. But my search works now.
    Immich-Microservice logs:
    [Nest] 1 - 03/22/2023, 8:58:30 AM ERROR [SmartInfoService] Unable run clip encoding pipeline: 795e0760-3aa3-483c-a52b-70dbbe5e1e02
    QueryFailedError: numeric field overflow
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
    at runMicrotasks ()
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SmartInfoRepository.upsert (/usr/src/app/dist/apps/microservices/libs/infra/src/db/repository/smart-info.repository.js:25:9)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/apps/microservices/libs/domain/src/smart-info/smart-info.service.js:97:13)

    at async ClipEncodingProcessor.onEncodeClip (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors.js:112:9)
    Immich-Postgres
    2023-03-22 12:58:30.591 UTC [472] ERROR: numeric field overflow
    2023-03-22 12:58:30.591 UTC [472] DETAIL: A field with precision 20, scale 19 must round to an absolute value less than 10^1.
    2023-03-22 12:58:30.591 UTC [472] STATEMENT: INSERT INTO "smart_info"("assetId", "tags", "objects", "clipEmbedding") VALUES ($1, DEFAULT, DEFAULT, $2) ON CONFLICT ( "assetId" ) DO UPDATE SET "assetId" = EXCLUDED."assetId", "clipEmbedding" = EXCLUDED."clipEmbedding"

Lastly THANK YOU! for such a wonderful feature. Loving it so far!
Just growing pains. ;-)

@w00tlarr commented on GitHub (Mar 22, 2023): Hi - I got these errors but resolved it. Here are the steps for me: 1) Make sure your docker-compose is pulling all images from ghcr.io repo (i.e.: ghcr.io/immich-app/immich-server:release) instead from altran1502. 2) Have to start containers in order - after docker-compose -d up (tho I use Portainer), I manually shutdown all containers and restart one by one in this order, immich-db, redis, typesense, then wait up to 5 mins before immich-machine-learning, immich-server, immich-microservice. The connection between typesense and immich-server is pretty tight (per release notes), so make sure typesense is up before the server is. [I did comment out the _logging: driver: none_ section of typesense docker-compose to monitor in the docker logs.] This part is a lot of trials and errors for me to get all containers to run healthily - if someone can suggest a docker-compose method to check for dependencies + health checks (not just container starts), please suggest one. Thanks in advance. 3) Once up, go to the Immich Web UI > Administration > Encode Clip > All (button). Mine ran overnight (~over 100K photos/videos). **Trying to search before this is done will give the 500 errors (404 in immich-server logs)**. At the end, you'll need to have patience for this for large libraries. 4) Note: I am getting these errors still though. But my search works now. Immich-Microservice logs: [Nest] 1 - 03/22/2023, 8:58:30 AM ERROR [SmartInfoService] Unable run clip encoding pipeline: 795e0760-3aa3-483c-a52b-70dbbe5e1e02 QueryFailedError: numeric field overflow at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33) at async SmartInfoRepository.upsert (/usr/src/app/dist/apps/microservices/libs/infra/src/db/repository/smart-info.repository.js:25:9) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/apps/microservices/libs/domain/src/smart-info/smart-info.service.js:97:13) at async ClipEncodingProcessor.onEncodeClip (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors.js:112:9) -- Immich-Postgres 2023-03-22 12:58:30.591 UTC [472] ERROR: numeric field overflow 2023-03-22 12:58:30.591 UTC [472] DETAIL: A field with precision 20, scale 19 must round to an absolute value less than 10^1. 2023-03-22 12:58:30.591 UTC [472] STATEMENT: INSERT INTO "smart_info"("assetId", "tags", "objects", "clipEmbedding") VALUES ($1, DEFAULT, DEFAULT, $2) ON CONFLICT ( "assetId" ) DO UPDATE SET "assetId" = EXCLUDED."assetId", "clipEmbedding" = EXCLUDED."clipEmbedding" Lastly THANK YOU! for such a wonderful feature. Loving it so far! Just growing pains. ;-)
Author
Owner

@jrasm91 commented on GitHub (Mar 22, 2023):

Thanks!

We're working to make the start order/logic between typesense and the other containers more flexible.

The overflow has shown up a few times. If you could grab the thumbnail for that asset, I would be interested to know if it is a valid image. If you could find and upload it, we can troubleshoot that issue. Ultimately it means that one won't show up in clip searches, but everything else should continue to work as expected.

@jrasm91 commented on GitHub (Mar 22, 2023): Thanks! We're working to make the start order/logic between typesense and the other containers more flexible. The overflow has shown up a few times. If you could grab the thumbnail for that asset, I would be interested to know if it is a valid image. If you could find and upload it, we can troubleshoot that issue. Ultimately it means that one won't show up in clip searches, but everything else should continue to work as expected.
Author
Owner

@x-real-ip commented on GitHub (Mar 23, 2023):

@theautomation I had the same error. As you can see it says permission denied on /.cache. I had passed in a uid into this container just cause I'm paranoid. Removing the uid and letting it be root fixes the problem. The library sentence-transformers used by immich appears to try to create this folder. Unfortunately even when I tried creating this folder in the container with the right permissions I was getting a different error, a 405 error when trying to access https://huggingface.co/api/models/sentence-transformers/clip-ViT-B-32. For simplicity I just removed the user / uid / gid from my docker compose for this container. Hope that was your issue and hope this helps

Thanks for this tip :). I have removed the UID en GID and yes the search is working then, however this is a workaround in the wrong direction I would like to keep using the UID and GID because of security reasons but most of all the user an group of those ID's owns the NFS share I have mounted into the container. So searching is working with this workaround but now I am seeing errors when I start an "ENCODE CLIP" job.

[2023-03-23 20:04:41,843] ERROR in app: Exception on /image-classifier/tag-image [POST]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/src/app/src/main.py", line 43, in image_classification
    return run_engine(model, assetPath), 200
  File "/usr/src/app/src/main.py", line 59, in run_engine
    predictions = engine(path)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/image_classification.py", line 100, in __call__
    return super().__call__(images, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1084, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1090, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/image_classification.py", line 103, in preprocess
    image = load_image(image)
  File "/opt/venv/lib/python3.10/site-packages/transformers/image_utils.py", line 232, in load_image
    raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and upload/5c86787b-cae9-4382-b7ca-cc0075d77b96/thumb/CLI/b8e4adf4-76ee-41ac-8379-e921c6182229.jpeg is not a valid path

Now I don't know if the cause of this error is due to UID GID being removed or some other reason.
btw I am running this in Kubernetes This is my deployment yaml for the machine learning container/pod

@x-real-ip commented on GitHub (Mar 23, 2023): > @theautomation I had the same error. As you can see it says permission denied on /.cache. I had passed in a uid into this container just cause I'm paranoid. Removing the uid and letting it be root fixes the problem. The library [sentence-transformers](https://github.com/UKPLab/sentence-transformers/blob/3e1929fddef16df94f8bc6e3b10598a98f46e62d/sentence_transformers/SentenceTransformer.py#L61) used by immich appears to try to create this folder. Unfortunately even when I tried creating this folder in the container with the right permissions I was getting a different error, a 405 error when trying to access https://huggingface.co/api/models/sentence-transformers/clip-ViT-B-32. For simplicity I just removed the user / uid / gid from my docker compose for this container. Hope that was your issue and hope this helps Thanks for this tip :). I have removed the UID en GID and yes the search is working then, however this is a workaround in the wrong direction I would like to keep using the UID and GID because of security reasons but most of all the user an group of those ID's owns the NFS share I have mounted into the container. So searching is working with this workaround but now I am seeing errors when I start an "ENCODE CLIP" job. ``` [2023-03-23 20:04:41,843] ERROR in app: Exception on /image-classifier/tag-image [POST] Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/usr/src/app/src/main.py", line 43, in image_classification return run_engine(model, assetPath), 200 File "/usr/src/app/src/main.py", line 59, in run_engine predictions = engine(path) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/image_classification.py", line 100, in __call__ return super().__call__(images, **kwargs) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1084, in __call__ return self.run_single(inputs, preprocess_params, forward_params, postprocess_params) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1090, in run_single model_inputs = self.preprocess(inputs, **preprocess_params) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/image_classification.py", line 103, in preprocess image = load_image(image) File "/opt/venv/lib/python3.10/site-packages/transformers/image_utils.py", line 232, in load_image raise ValueError( ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and upload/5c86787b-cae9-4382-b7ca-cc0075d77b96/thumb/CLI/b8e4adf4-76ee-41ac-8379-e921c6182229.jpeg is not a valid path ``` Now I don't know if the cause of this error is due to UID GID being removed or some other reason. btw I am running this in Kubernetes [This is my deployment yaml for the machine learning container/pod](https://github.com/theautomation/immich/blob/main/deploy/k8s/manifest.yaml#L691-L808)
Author
Owner

@jagjordi commented on GitHub (Mar 24, 2023):

I am also seeing this error when running encode clip or tag objects:

172.25.0.7 - - [24/Mar/2023:15:33:03 +0000] "POST /image-classifier/tag-image HTTP/1.1" 500 265 "-" "axios/0.26.1"
[2023-03-24 15:33:03,567] ERROR in app: Exception on /object-detection/detect-object [POST]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/src/app/src/main.py", line 37, in object_detection
    return run_engine(model, assetPath), 200
  File "/usr/src/app/src/main.py", line 59, in run_engine
    predictions = engine(path)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/object_detection.py", line 95, in __call__
    return super().__call__(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1084, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1090, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/object_detection.py", line 98, in preprocess
    image = load_image(image)
  File "/opt/venv/lib/python3.10/site-packages/transformers/image_utils.py", line 232, in load_image
    raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and upload/a8e843dc-2b16-4175-b247-8ee5539f08ab/thumb/CLI/da09f667-6bd8-41a3-90a8-093b2306b5bc.jpeg is not a valid path
@jagjordi commented on GitHub (Mar 24, 2023): I am also seeing this error when running encode clip or tag objects: ``` 172.25.0.7 - - [24/Mar/2023:15:33:03 +0000] "POST /image-classifier/tag-image HTTP/1.1" 500 265 "-" "axios/0.26.1" [2023-03-24 15:33:03,567] ERROR in app: Exception on /object-detection/detect-object [POST] Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() File "/opt/venv/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/usr/src/app/src/main.py", line 37, in object_detection return run_engine(model, assetPath), 200 File "/usr/src/app/src/main.py", line 59, in run_engine predictions = engine(path) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/object_detection.py", line 95, in __call__ return super().__call__(*args, **kwargs) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1084, in __call__ return self.run_single(inputs, preprocess_params, forward_params, postprocess_params) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1090, in run_single model_inputs = self.preprocess(inputs, **preprocess_params) File "/opt/venv/lib/python3.10/site-packages/transformers/pipelines/object_detection.py", line 98, in preprocess image = load_image(image) File "/opt/venv/lib/python3.10/site-packages/transformers/image_utils.py", line 232, in load_image raise ValueError( ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and upload/a8e843dc-2b16-4175-b247-8ee5539f08ab/thumb/CLI/da09f667-6bd8-41a3-90a8-093b2306b5bc.jpeg is not a valid path ```
Author
Owner

@jrasm91 commented on GitHub (Mar 24, 2023):

@jagjordi - can you verify that the UPLOAD_LOCATION volume is mounted to the machine learning container?

@jrasm91 commented on GitHub (Mar 24, 2023): @jagjordi - can you verify that the UPLOAD_LOCATION volume is mounted to the machine learning container?
Author
Owner

@jagjordi commented on GitHub (Mar 25, 2023):

@jrasm91 It looks like it is mounted:

root@Tower:~# docker exec -it immich_machine_learning bash
root@3fb4fa0cb6be:/usr/src/app# ls
Dockerfile  README.md  gunicorn.conf.py  src  upload
root@3fb4fa0cb6be:/usr/src/app# cd upload/
root@3fb4fa0cb6be:/usr/src/app/upload# ls
1dd38e30-8f47-4129-bcd0-36e6537dfa54  5001418f-1e48-4a12-81c9-10ff90defb78
21500b77-b342-4707-acf0-b6f1daa55577  9d015ed5-e62c-4dac-b6f6-3f5f25c73e00
root@3fb4fa0cb6be:/usr/src/app/upload# cd 9d015ed5-e62c-4dac-b6f6-3f5f25c73e00/
root@3fb4fa0cb6be:/usr/src/app/upload/9d015ed5-e62c-4dac-b6f6-3f5f25c73e00# ls
1980  2007  2010  2013  2016  2019  2022           original
2003  2008  2011  2014  2017  2020  2023           profile
2004  2009  2012  2015  2018  2021  encoded-video  thumb
root@3fb4fa0cb6be:/usr/src/app/upload/9d015ed5-e62c-4dac-b6f6-3f5f25c73e00# 

docker_compose.yml (ML section)

  immich-machine-learning:                                                                                                                                                                                                      
    container_name: immich_machine_learning                                                                                                                                                                                     
    image: ghcr.io/immich-app/immich-machine-learning:release                                                                                                                                                                   
    volumes:                                                                                                                                                                                                                    
      - ${UPLOAD_LOCATION}:/usr/src/app/upload                                                                                                                                                                                  
      - model-cache:/cache
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    restart: always

It seems to be correctly mounted into /usr/src/app/upload

@jagjordi commented on GitHub (Mar 25, 2023): @jrasm91 It looks like it is mounted: ``` root@Tower:~# docker exec -it immich_machine_learning bash root@3fb4fa0cb6be:/usr/src/app# ls Dockerfile README.md gunicorn.conf.py src upload root@3fb4fa0cb6be:/usr/src/app# cd upload/ root@3fb4fa0cb6be:/usr/src/app/upload# ls 1dd38e30-8f47-4129-bcd0-36e6537dfa54 5001418f-1e48-4a12-81c9-10ff90defb78 21500b77-b342-4707-acf0-b6f1daa55577 9d015ed5-e62c-4dac-b6f6-3f5f25c73e00 root@3fb4fa0cb6be:/usr/src/app/upload# cd 9d015ed5-e62c-4dac-b6f6-3f5f25c73e00/ root@3fb4fa0cb6be:/usr/src/app/upload/9d015ed5-e62c-4dac-b6f6-3f5f25c73e00# ls 1980 2007 2010 2013 2016 2019 2022 original 2003 2008 2011 2014 2017 2020 2023 profile 2004 2009 2012 2015 2018 2021 encoded-video thumb root@3fb4fa0cb6be:/usr/src/app/upload/9d015ed5-e62c-4dac-b6f6-3f5f25c73e00# ``` `docker_compose.yml` (ML section) ``` immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:release volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - model-cache:/cache env_file: - .env environment: - NODE_ENV=production restart: always ``` It seems to be correctly mounted into `/usr/src/app/upload`
Author
Owner

@alextran1502 commented on GitHub (Apr 23, 2023):

This would happen if the system has non-avx CPU and try Typesense search, the workaround is to disable typesense

@alextran1502 commented on GitHub (Apr 23, 2023): This would happen if the system has non-avx CPU and try Typesense search, the workaround is to disable typesense
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#757