immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

No more HWA-Support for gfx900 in Immich Machine Learning 2.4.0 (ROCM) and above #8098

New Issue
Open
opened 2026-02-05 13:31:53 +03:00 by OVERLORD · 7 comments
OVERLORD commented 2026-02-05 13:31:53 +03:00
Owner
Copy Link

Originally created by @babushanaelsa on GitHub (Dec 20, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

I'm hosting Immich on a mini PC with a AMD Radeon RX Vega 8 (Renoir) iGPU. This GPU ist not officially supported by ROCM and never has been, but it worked until v2.4.0.

As it turns out, the newer images are missing the File /opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat (for my model, don't know what else could be missing).

As a workaround I've extracted this file from an older version of Immich machine learning, put it in my data location and mapped it into the container so it works again, even with the newest v2.4.1.

Is it possible to get this file and maybe the missing configs for other models as well back into the image?

The OS that Immich Server is running on

LMDE 7

Version of Immich Server

2.4.1

Version of Immich Mobile App

2.4.1

Platform with the issue

  • Server
  • Web
  • Mobile

Device make and model

No response

Your docker-compose.yml content

immich-machine-learning:
    container_name: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rocm
    labels:
      - com.centurylinklabs.watchtower.monitor-only=true
    networks:
      - default
    user: 1000:105 #video
    group_add:
      - video
      - render
    devices:
      - /dev/dri:/dev/dri
      - /dev/kfd:/dev/kfd
    volumes:
      - ${UPLOAD_LOCATION}/model-cache:/cache
      - /var/lib/docker/volumes/immich/TensileLibrary_lazy_gfx900.dat:/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    environment:
      - HSA_OVERRIDE_GFX_VERSION=9.0.0
      - HSA_USE_SVM=0
      - MACHINE_LEARNING_MODEL_TTL=120
      - NODE_ENV=producton
    security_opt:
      - seccomp=unconfined
    depends_on:
      immich-server:
       condition: service_started
    restart: always

Your .env content

no need for this issue

Reproduction steps

  1. use newest version of ML with my machine
  2. Upload pictures from mobile
  3. See log of ML container which says "no file for gfx900 found in /opt/rocm......"
    ...

Relevant log output

 Setting execution providers to                     

                             ['ROCMExecutionProvider', 'CPUExecutionProvider'], 

                             in descending order of preference                  

rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx900

 List of available TensileLibrary Files : 

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1201.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1200.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1151.dat"

"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat"

Additional information

No response

Originally created by @babushanaelsa on GitHub (Dec 20, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug I'm hosting Immich on a mini PC with a AMD Radeon RX Vega 8 (Renoir) iGPU. This GPU ist not officially supported by ROCM and never has been, but it worked until v2.4.0. As it turns out, the newer images are missing the File /opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat (for my model, don't know what else could be missing). As a workaround I've extracted this file from an older version of Immich machine learning, put it in my data location and mapped it into the container so it works again, even with the newest v2.4.1. Is it possible to get this file and maybe the missing configs for other models as well back into the image? ### The OS that Immich Server is running on LMDE 7 ### Version of Immich Server 2.4.1 ### Version of Immich Mobile App 2.4.1 ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Device make and model _No response_ ### Your docker-compose.yml content ```YAML immich-machine-learning: container_name: immich-machine-learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rocm labels: - com.centurylinklabs.watchtower.monitor-only=true networks: - default user: 1000:105 #video group_add: - video - render devices: - /dev/dri:/dev/dri - /dev/kfd:/dev/kfd volumes: - ${UPLOAD_LOCATION}/model-cache:/cache - /var/lib/docker/volumes/immich/TensileLibrary_lazy_gfx900.dat:/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro env_file: - stack.env environment: - HSA_OVERRIDE_GFX_VERSION=9.0.0 - HSA_USE_SVM=0 - MACHINE_LEARNING_MODEL_TTL=120 - NODE_ENV=producton security_opt: - seccomp=unconfined depends_on: immich-server: condition: service_started restart: always ``` ### Your .env content ```Shell no need for this issue ``` ### Reproduction steps 1. use newest version of ML with my machine 2. Upload pictures from mobile 3. See log of ML container which says "no file for gfx900 found in /opt/rocm......" ... ### Relevant log output ```shell Setting execution providers to ['ROCMExecutionProvider', 'CPUExecutionProvider'], in descending order of preference rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx900 List of available TensileLibrary Files : "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1201.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1200.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1151.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat" ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 13:31:56 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 20, 2025):

Is this in practice a dupe of #22874?

@bo0tzz commented on GitHub (Dec 20, 2025): Is this in practice a dupe of #22874?
OVERLORD commented 2026-02-05 13:31:57 +03:00
Author
Owner
Copy Link

@babushanaelsa commented on GitHub (Dec 20, 2025):

Is this in practice a dupe of #22874?

I'm not quite sure, I added my error logs above. I think they differ a little bit.

@babushanaelsa commented on GitHub (Dec 20, 2025): > Is this in practice a dupe of [#22874](https://github.com/immich-app/immich/issues/22874)? I'm not quite sure, I added my error logs above. I think they differ a little bit.
OVERLORD commented 2026-02-05 13:31:58 +03:00
Author
Owner
Copy Link

@babushanaelsa commented on GitHub (Dec 20, 2025):

just to mention it: I tried setting HSA_OVERRIDE_GFX_VERSION to 9.0.1 and 9.0.8 but didn't work

@babushanaelsa commented on GitHub (Dec 20, 2025): just to mention it: I tried setting HSA_OVERRIDE_GFX_VERSION to 9.0.1 and 9.0.8 but didn't work
OVERLORD commented 2026-02-05 13:31:59 +03:00
Author
Owner
Copy Link

@hiteshkardam commented on GitHub (Dec 25, 2025):

I closed my discussion post reporting the same issue https://github.com/immich-app/immich/discussions/24685

The issue appears to be caused by the commit that updated the ROCm Ubuntu image from 6.3.4 to 6.4.3. The newer image no longer ships with prebuilt rocBLAS Tensile libraries for gfx900 01fddd58c6

To work around this, I built rocBLAS + Tensile from source for gfx900 using the deprecated rocm-6.4.3 branch of the rocBLAS repository https://github.com/ROCm/rocBLAS.git
docker buildx build --progress=plain --load -f Dockerfile-rocblas-gfx900 -t immich-machine-learning:V2.4.1-rocm-rocblas-gfx900 .

The build was done inside the rocm/dev-ubuntu-22.04:6.4.3-complete image with lazy library loading enabled (see:
https://rocm.docs.amd.com/projects/Tensile/en/latest/src/conceptual/solution-selection-catalogs.html).

I then replaced the resulting libraries into the Immich ML v2.4.1-rocm release image and exported it to my immich host using:
docker save immich-machine-learning:V2.4.1-rocm-rocblas-gfx900 | gzip > immich-v2.4.1-rocblas-gfx900.tar.gz

On the Immich host, I loaded the image with:
gunzip -c immich-v2.4.1-rocblas-gfx900.tar.gz | docker load

Finally, I updated the immich-machine-learning container to use the following image:
immich-machine-learning:V2.4.1-rocm-rocblas-gfx900

This setup is working fine, although OCR is still crashing, which I'm still investigating.

Following is my Dockerfile-rocblas-gfx900 I created with some help from AI, if you want to give it a try

# Fetch ROCM Ubuntu Image
ARG ROCM_IMAGE=rocm/dev-ubuntu-22.04:6.4.3-complete
FROM ${ROCM_IMAGE} AS rocblas_build

# Build rocBLAS 6.4.3
ARG ROCBLAS_REF=rocm-6.4.3
ARG ROCBLAS_ARCH=gfx900

RUN apt-get update && apt-get install -y --no-install-recommends \
    git ca-certificates build-essential cmake ninja-build \
    wget tar xz-utils unzip pkg-config \
    python3 python3-pip python3-venv python3-yaml python3-joblib python3-psutil \
    libmsgpack-dev \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /src
RUN git clone --depth 1 --branch "${ROCBLAS_REF}" https://github.com/ROCm/rocBLAS.git
WORKDIR /src/rocBLAS

# Build rocBLAS + Tensile for gfx900 only (library only, no clients)
RUN python3 ./rmake.py --install_invoked -c -a "${ROCBLAS_ARCH}" \
    --build_dir=/src/rocBLAS/build --src_path=/src/rocBLAS \
    --no-merge-architectures \
    --lazy-library-loading \
    --cmake-darg BUILD_CLIENTS_TESTS=OFF \
    --cmake-darg BUILD_CLIENTS_BENCHMARKS=OFF \
    --cmake-darg BUILD_CLIENTS_SAMPLES=OFF

# Locate the install root
RUN set -eux; \
    INSTALL_ROOT="/src/rocBLAS/build/release/rocblas-install"; \
    test -d "${INSTALL_ROOT}"; \
    ls -l "${INSTALL_ROOT}/lib/librocblas.so"*; \
    test -d "${INSTALL_ROOT}/lib/rocblas/library"; \
    (ls -1 "${INSTALL_ROOT}/lib/rocblas/library" | grep -E 'gfx900' >/dev/null); \
    mkdir -p /out/lib /out/rocblas; \
    cp -a "${INSTALL_ROOT}/lib/librocblas.so"* /out/lib/; \
    cp -a "${INSTALL_ROOT}/lib/rocblas" /out/rocblas/

# Use the published Immich ML ROCm image as the final base
FROM ghcr.io/immich-app/immich-machine-learning:v2.4.1-rocm AS final

# Check ROCm exists where we expect
RUN set -eux; \
    test -d /opt/rocm; \
    ls -ld /opt/rocm/lib || true

# Replace rocBLAS + Tensile libs
RUN rm -rf /opt/rocm/lib/rocblas /opt/rocm/lib/librocblas.so* || true
COPY --from=rocblas_build /out/lib/ /opt/rocm/lib/
COPY --from=rocblas_build /out/rocblas/rocblas /opt/rocm/lib/rocblas

# Tensile libraries path for rocBLAS
ENV ROCBLAS_TENSILE_LIBPATH=/opt/rocm/lib/rocblas/library
@hiteshkardam commented on GitHub (Dec 25, 2025): I closed my discussion post reporting the same issue https://github.com/immich-app/immich/discussions/24685 The issue appears to be caused by the commit that updated the ROCm Ubuntu image from 6.3.4 to 6.4.3. The newer image no longer ships with prebuilt rocBLAS Tensile libraries for gfx900 https://github.com/immich-app/immich/commit/01fddd58c6a3f4ad7f82a921a2a0a41a629334e6 To work around this, I built rocBLAS + Tensile from source for gfx900 using the deprecated rocm-6.4.3 branch of the rocBLAS repository https://github.com/ROCm/rocBLAS.git docker buildx build --progress=plain --load -f Dockerfile-rocblas-gfx900 -t immich-machine-learning:V2.4.1-rocm-rocblas-gfx900 . The build was done inside the rocm/dev-ubuntu-22.04:6.4.3-complete image with lazy library loading enabled (see: https://rocm.docs.amd.com/projects/Tensile/en/latest/src/conceptual/solution-selection-catalogs.html). I then replaced the resulting libraries into the Immich ML v2.4.1-rocm release image and exported it to my immich host using: docker save immich-machine-learning:V2.4.1-rocm-rocblas-gfx900 | gzip > immich-v2.4.1-rocblas-gfx900.tar.gz On the Immich host, I loaded the image with: gunzip -c immich-v2.4.1-rocblas-gfx900.tar.gz | docker load Finally, I updated the immich-machine-learning container to use the following image: immich-machine-learning:V2.4.1-rocm-rocblas-gfx900 This setup is working fine, although OCR is still crashing, which I'm still investigating. Following is my Dockerfile-rocblas-gfx900 I created with some help from AI, if you want to give it a try ``` # Fetch ROCM Ubuntu Image ARG ROCM_IMAGE=rocm/dev-ubuntu-22.04:6.4.3-complete FROM ${ROCM_IMAGE} AS rocblas_build # Build rocBLAS 6.4.3 ARG ROCBLAS_REF=rocm-6.4.3 ARG ROCBLAS_ARCH=gfx900 RUN apt-get update && apt-get install -y --no-install-recommends \ git ca-certificates build-essential cmake ninja-build \ wget tar xz-utils unzip pkg-config \ python3 python3-pip python3-venv python3-yaml python3-joblib python3-psutil \ libmsgpack-dev \ && rm -rf /var/lib/apt/lists/* WORKDIR /src RUN git clone --depth 1 --branch "${ROCBLAS_REF}" https://github.com/ROCm/rocBLAS.git WORKDIR /src/rocBLAS # Build rocBLAS + Tensile for gfx900 only (library only, no clients) RUN python3 ./rmake.py --install_invoked -c -a "${ROCBLAS_ARCH}" \ --build_dir=/src/rocBLAS/build --src_path=/src/rocBLAS \ --no-merge-architectures \ --lazy-library-loading \ --cmake-darg BUILD_CLIENTS_TESTS=OFF \ --cmake-darg BUILD_CLIENTS_BENCHMARKS=OFF \ --cmake-darg BUILD_CLIENTS_SAMPLES=OFF # Locate the install root RUN set -eux; \ INSTALL_ROOT="/src/rocBLAS/build/release/rocblas-install"; \ test -d "${INSTALL_ROOT}"; \ ls -l "${INSTALL_ROOT}/lib/librocblas.so"*; \ test -d "${INSTALL_ROOT}/lib/rocblas/library"; \ (ls -1 "${INSTALL_ROOT}/lib/rocblas/library" | grep -E 'gfx900' >/dev/null); \ mkdir -p /out/lib /out/rocblas; \ cp -a "${INSTALL_ROOT}/lib/librocblas.so"* /out/lib/; \ cp -a "${INSTALL_ROOT}/lib/rocblas" /out/rocblas/ # Use the published Immich ML ROCm image as the final base FROM ghcr.io/immich-app/immich-machine-learning:v2.4.1-rocm AS final # Check ROCm exists where we expect RUN set -eux; \ test -d /opt/rocm; \ ls -ld /opt/rocm/lib || true # Replace rocBLAS + Tensile libs RUN rm -rf /opt/rocm/lib/rocblas /opt/rocm/lib/librocblas.so* || true COPY --from=rocblas_build /out/lib/ /opt/rocm/lib/ COPY --from=rocblas_build /out/rocblas/rocblas /opt/rocm/lib/rocblas # Tensile libraries path for rocBLAS ENV ROCBLAS_TENSILE_LIBPATH=/opt/rocm/lib/rocblas/library ```
OVERLORD commented 2026-02-05 13:31:59 +03:00
Author
Owner
Copy Link

@karatektus commented on GitHub (Jan 29, 2026):

just ran into this. is this something thats going to be fixed?

@karatektus commented on GitHub (Jan 29, 2026): just ran into this. is this something thats going to be fixed?
OVERLORD commented 2026-02-05 13:32:00 +03:00
Author
Owner
Copy Link

@marcio-af-oliveira commented on GitHub (Jan 29, 2026):

Also got this issue!

@marcio-af-oliveira commented on GitHub (Jan 29, 2026): Also got this issue!
OVERLORD commented 2026-02-05 13:32:01 +03:00
Author
Owner
Copy Link

@marcio-af-oliveira commented on GitHub (Feb 3, 2026):

@hiteshkardam tested your image and it also worked on my end and got the same crashes on OCR.
I tried disabling the multithread by using MACHINE_LEARNING_WORKERS=1 like @moooyo said in another issue open #22874 but didn't work either

@marcio-af-oliveira commented on GitHub (Feb 3, 2026): @hiteshkardam tested your image and it also worked on my end and got the same crashes on OCR. I tried disabling the multithread by using MACHINE_LEARNING_WORKERS=1 like @moooyo said in another issue open [#22874](https://github.com/immich-app/immich/issues/22874) but didn't work either
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#8098
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?