[BUG] Cannot download model #1488

Closed
opened 2026-02-05 02:03:39 +03:00 by OVERLORD · 1 comment
Owner

Originally created by @piciuok on GitHub (Oct 20, 2023).

The bug

While starting whole stack, immich_machine_learning service run into issues while trying download some models.

The OS that Immich Server is running on

Linux / QNAP

Version of Immich Server

v.1.82.1

Version of Immich Mobile App

v.1.82.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /share/CACHEDEV2_DATA/Fotografie:/mnt/media/Mateusz/Fotografie:ro
      - /share/CACHEDEV2_DATA/Wideo/NagraniaFonem:/mnt/media/Mateusz/Wideo:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    devices:
      - /dev/dri:/dev/dri  # If using Intel QuickSync or VAAPI
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /share/CACHEDEV2_DATA/Fotografie:/mnt/media/Mateusz/Fotografie:ro
      - /share/CACHEDEV2_DATA/Wideo/NagraniaFonem:/mnt/media/Mateusz/Wideo:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    environment:
      - NODE_OPTIONS=--max-old-space-size=10240
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /share/CACHEDEV3_DATA/DockerData/immich/model-cache:/cache
    env_file:
      - stack.env
    restart: always


  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - stack.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - /share/CACHEDEV3_DATA/DockerData/immich/typesense:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /share/CACHEDEV3_DATA/DockerData/immich/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always
    
networks:
  default:
    name: npm_private_net
    external: true

Your .env content

UPLOAD_LOCATION=/share/CACHEDEV3_DATA/DockerData/immich/upload
IMMICH_VERSION=release
TYPESENSE_API_KEY=dummy
DB_PASSWORD=dummy
DB_HOSTNAME=immich_postgres
DB_USERNAME=dummy
DB_DATABASE_NAME=dummy
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Run whole stack

Additional information

                             take a while.                                      
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572
76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 0th
attempt
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572
76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 1th
attempt
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572
76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 2th
attempt
  textual.onnx 0.0% • 0/100 bytes • ? • -:--:--
                                               
Exception in ASGI application
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 75, in predict
    model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 101, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 63, in load
    self.download()
  File "/usr/src/app/models/base.py", line 58, in download
    self._download()
  File "/usr/src/app/models/clip.py", line 43, in _download
    self._download_model(*models[0])
  File "/usr/src/app/models/clip.py", line 101, in _download_model
    download_model(
  File "/opt/venv/lib/python3.11/site-packages/clip_server/model/pretrained_models.py", line 239, in download_model
    raise RuntimeError(
RuntimeError: Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual.onnx within retry limit 3
[10/20/23 20:03:52] INFO     Downloading clip model 'ViT-B-32::openai'.This may 
                             take a while.                                      
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572
76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 0th
attempt```
Originally created by @piciuok on GitHub (Oct 20, 2023). ### The bug While starting whole stack, immich_machine_learning service run into issues while trying download some models. ### The OS that Immich Server is running on Linux / QNAP ### Version of Immich Server v.1.82.1 ### Version of Immich Mobile App v.1.82.1 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ["start.sh", "immich"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /share/CACHEDEV2_DATA/Fotografie:/mnt/media/Mateusz/Fotografie:ro - /share/CACHEDEV2_DATA/Wideo/NagraniaFonem:/mnt/media/Mateusz/Wideo:ro - /etc/localtime:/etc/localtime:ro env_file: - stack.env depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} devices: - /dev/dri:/dev/dri # If using Intel QuickSync or VAAPI command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /share/CACHEDEV2_DATA/Fotografie:/mnt/media/Mateusz/Fotografie:ro - /share/CACHEDEV2_DATA/Wideo/NagraniaFonem:/mnt/media/Mateusz/Wideo:ro - /etc/localtime:/etc/localtime:ro env_file: - stack.env environment: - NODE_OPTIONS=--max-old-space-size=10240 depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - /share/CACHEDEV3_DATA/DockerData/immich/model-cache:/cache env_file: - stack.env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - stack.env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - /share/CACHEDEV3_DATA/DockerData/immich/typesense:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - stack.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - /share/CACHEDEV3_DATA/DockerData/immich/pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 depends_on: - immich-server - immich-web restart: always networks: default: name: npm_private_net external: true ``` ### Your .env content ```Shell UPLOAD_LOCATION=/share/CACHEDEV3_DATA/DockerData/immich/upload IMMICH_VERSION=release TYPESENSE_API_KEY=dummy DB_PASSWORD=dummy DB_HOSTNAME=immich_postgres DB_USERNAME=dummy DB_DATABASE_NAME=dummy REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. Run whole stack ``` ### Additional information ```10/20/23 20:03:36] INFO Downloading clip model 'ViT-B-32::openai'.This may take a while. Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572 76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 0th attempt Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572 76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 1th attempt Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572 76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 2th attempt textual.onnx 0.0% • 0/100 bytes • ? • -:--:-- Exception in ASGI application Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__ await super().__call__(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__ await self.middleware_stack(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__ raise e File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__ await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__ await route.handle(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 75, in predict model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 101, in load await loop.run_in_executor(app.state.thread_pool, _load) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 94, in _load model.load() File "/usr/src/app/models/base.py", line 63, in load self.download() File "/usr/src/app/models/base.py", line 58, in download self._download() File "/usr/src/app/models/clip.py", line 43, in _download self._download_model(*models[0]) File "/usr/src/app/models/clip.py", line 101, in _download_model download_model( File "/opt/venv/lib/python3.11/site-packages/clip_server/model/pretrained_models.py", line 239, in download_model raise RuntimeError( RuntimeError: Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual.onnx within retry limit 3 [10/20/23 20:03:52] INFO Downloading clip model 'ViT-B-32::openai'.This may take a while. Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572 76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 0th attempt```
Author
Owner

@alextran1502 commented on GitHub (Oct 20, 2023):

Please see this comment https://github.com/immich-app/immich/issues/4117#issuecomment-1772790612

@alextran1502 commented on GitHub (Oct 20, 2023): Please see this comment https://github.com/immich-app/immich/issues/4117#issuecomment-1772790612
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1488