immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 13 Packages Projects Releases 10 Wiki Activity

Support hardware-accelerated decoding and tone-mapping #1407

New Issue
Closed
opened 2026-02-05 01:41:41 +03:00 by OVERLORD · 10 comments
OVERLORD commented 2026-02-05 01:41:41 +03:00
Owner
Copy Link

Originally created by @mertalev on GitHub (Oct 3, 2023).

This is a tracking issue for adding hardware decoding and tone-mapping support for transcoding.

What

Hardware-accelerated decoding loads videos to an acceleration device to decode with the device's built-in support for certain codecs and formats. This differs from software decoding, where videos are instead loaded and decoded by a program.

Hardware-accelerated tone-mapping is similarly performed within the acceleration device, but takes place after decoding.

Why

Hardware decoding is good for a number of reasons.

  1. It's faster
    • Accelerated decoding is naturally faster by virtue of its dedicated hardware optimization
    • By keeping data in the acceleration device, it avoids starvation from the CPU not serving decoded data quickly enough
    • Decoded video can be directly used by the acceleration device without needing to do a relatively expensive CPU->GPU transfer
    • It avoids contention in cases where the CPU is concurrently doing other intensive work
  2. It reduces CPU load
    • Since the CPU doesn't decode the video, the incurred load of decoding is very minimal
    • Particularly on lower-end devices, the relative performance of the acceleration device compared to the CPU can be drastic, meaning that using the CPU with software decoding requires exerting it heavily in order to keep up with the device's encoding speed

Concerns

  1. Source videos come in many different forms, and it's tricky to know in advance whether the device can decode a given video (at least in JavaScript)
  2. Different APIs may expose different tone-mapping options and modes, so supporting the current settings in each API may require more effort
  3. Hardware tone-mapping is essentially a pre-requisite for hardware decoding
    • Since hardware decoding loads videos to the device, software tone-mapping would require a GPU->CPU transfer after decoding followed by another CPU->GPU transfer after tone-mapping, the overhead of which defeats the point of acceleration
  4. While less relevant in recent years, hardware decoding can in some cases have lower quality than software decoding

Tasks

  • Ensure it uses software decoding if the user's hardware can't decode or tone-map a video
  • Ensure that in cases of incompatibility, it still uses accelerated encoding rather than falling back entirely to software
  • Ensure that current tone-mapping options are available for each API where possible
  • Support Quick Sync
  • Support NVENC
  • Support VAAPI
Originally created by @mertalev on GitHub (Oct 3, 2023). This is a tracking issue for adding hardware decoding and tone-mapping support for transcoding. ### What Hardware-accelerated decoding loads videos to an acceleration device to decode with the device's built-in support for certain codecs and formats. This differs from software decoding, where videos are instead loaded and decoded by a program. Hardware-accelerated tone-mapping is similarly performed within the acceleration device, but takes place after decoding. ### Why Hardware decoding is good for a number of reasons. 1. It's faster - Accelerated decoding is naturally faster by virtue of its dedicated hardware optimization - By keeping data in the acceleration device, it avoids starvation from the CPU not serving decoded data quickly enough - Decoded video can be directly used by the acceleration device without needing to do a relatively expensive CPU->GPU transfer - It avoids contention in cases where the CPU is concurrently doing other intensive work 2. It reduces CPU load - Since the CPU doesn't decode the video, the incurred load of decoding is very minimal - Particularly on lower-end devices, the relative performance of the acceleration device compared to the CPU can be drastic, meaning that using the CPU with software decoding requires exerting it heavily in order to keep up with the device's encoding speed ### Concerns 1. Source videos come in many different forms, and it's tricky to know in advance whether the device can decode a given video (at least in JavaScript) 2. Different APIs may expose different tone-mapping options and modes, so supporting the current settings in each API may require more effort 4. Hardware tone-mapping is essentially a pre-requisite for hardware decoding - Since hardware decoding loads videos to the device, software tone-mapping would require a GPU->CPU transfer after decoding followed by another CPU->GPU transfer after tone-mapping, the overhead of which defeats the point of acceleration 5. While less relevant in recent years, hardware decoding can in some cases have lower quality than software decoding ### Tasks - [x] Ensure it uses software decoding if the user's hardware can't decode or tone-map a video - [ ] Ensure that in cases of incompatibility, it still uses accelerated encoding rather than falling back entirely to software - [x] Ensure that current tone-mapping options are available for each API where possible - [x] Support Quick Sync - [x] Support NVENC - [x] Support VAAPI
OVERLORD added the 🗄️server label 2026-02-05 01:41:41 +03:00
OVERLORD commented 2026-02-05 01:41:48 +03:00
Author
Owner
Copy Link

@cyfdecyf commented on GitHub (Oct 5, 2023):

Just add my two cents about tone-mapping using Intel Quick Sync.

Calling ffmpeg with -vf "vpp_qsv=tonemap=1" enables hardware accelerated tone-mapping with QSV, but it's only available when oneVPL is enabled when compiling FFmpeg. (When run FFmpeg ./configure, just replace --enable-libmfx with --enable-libvpl, Intel oneVPL library is needed of course.) According to oneVPL dispatching behavior, I'm not sure whether this would work with Intel processor before Tiger Lake.

@cyfdecyf commented on GitHub (Oct 5, 2023): Just add my two cents about tone-mapping using Intel Quick Sync. Calling ffmpeg with `-vf "vpp_qsv=tonemap=1"` enables hardware accelerated tone-mapping with QSV, but it's only available when [oneVPL is enabled](https://github.com/FFmpeg/FFmpeg/blob/9ae4863cc57c38fa2f41f3b8c732fede4a2b78c6/libavfilter/vf_vpp_qsv.c#L393) when compiling FFmpeg. (When run FFmpeg `./configure`, just replace `--enable-libmfx` with `--enable-libvpl`, Intel oneVPL library is needed of course.) According to oneVPL [dispatching behavior](https://github.com/oneapi-src/oneVPL#onevpl-dispatcher-behavior-when-targeting-intel-gpus), I'm not sure whether this would work with Intel processor before Tiger Lake.
OVERLORD commented 2026-02-05 01:41:51 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Oct 5, 2023):

The version of FFmpeg we use is built with oneVPL, so that shouldn't be an issue, but it does seem like this wouldn't work if it dispatches to Media SDK. Jellyfin docs mention that the main advantage of QSV's tonemapping is lower power consumption, but otherwise OpenCL has wider hardware compatibility and is more customizable. Maybe that's the direction to go in that case.

@mertalev commented on GitHub (Oct 5, 2023): The version of FFmpeg we use is built with oneVPL, so that shouldn't be an issue, but it does seem like this wouldn't work if it dispatches to Media SDK. [Jellyfin docs](https://jellyfin.org/docs/general/administration/hardware-acceleration/intel/#tone-mapping-methods) mention that the main advantage of QSV's tonemapping is lower power consumption, but otherwise OpenCL has wider hardware compatibility and is more customizable. Maybe that's the direction to go in that case.
OVERLORD commented 2026-02-05 01:41:56 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Oct 15, 2023):

Does it will apply for generate the thumbnails it could be good for big library of photos ?

@yodatak commented on GitHub (Oct 15, 2023): Does it will apply for generate the thumbnails it could be good for big library of photos ?
OVERLORD commented 2026-02-05 01:42:00 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Oct 15, 2023):

No, it wouldn't have an effect on images. But for live/motion photos, the video portion of these would benefit.

@mertalev commented on GitHub (Oct 15, 2023): No, it wouldn't have an effect on images. But for live/motion photos, the video portion of these would benefit.
OVERLORD commented 2026-02-05 01:42:05 +03:00
Author
Owner
Copy Link

@rishid commented on GitHub (Feb 1, 2024):

I was curious about why immich even with hardware transcoding enabled was basically maxing out my 16 cpu cores even with only doing 1 transcode job. It also only is using 15% of my GPU render capability.

I went ahead and played with some of the ffmpeg options. Most of this is known but just adding my findings here:

Here is a sample immich ffmpeg call when using Intel QSV

ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i upload/upload/4ef.../...c2f.MOV -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -v verbose -vf zscale=t=linear:npl=100,tonemap=hable:desat=0,zscale=p=bt709:t=bt709:m=bt709:range=pc,format=nv12,hwupload=extra_hw_frames=64,scale_qsv=1080:-1 -preset 7 -global_quality 23 upload/encoded-video/4ef.../...c7d.mp4

As stated in original post, we aren't using hardware decoding, by enabling this I see about a 5% reduction in CPU load.
I get a 5% improvement also by setting the preset to fast.

I am not super familiar with ffmpeg but the remainder of extra cpu load is coming from the filters. Is there a reason we need to do tone-mapping and all the zscale options? If I trim it down to the following, I get about 75% reduction in CPU load.

/usr/lib/jellyfin-ffmpeg/ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -c:v hevc_qsv -i /config/test.MOV -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -v verbose -vf format=nv12,hwupload=extra_hw_frames=64,scale_qsv=1080:-1 -preset fast -global_quality 23 /config/test_OUT.mp4
@rishid commented on GitHub (Feb 1, 2024): I was curious about why immich even with hardware transcoding enabled was basically maxing out my 16 cpu cores even with only doing 1 transcode job. It also only is using 15% of my GPU render capability. I went ahead and played with some of the ffmpeg options. Most of this is known but just adding my findings here: Here is a sample immich ffmpeg call when using Intel QSV ``` ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i upload/upload/4ef.../...c2f.MOV -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -v verbose -vf zscale=t=linear:npl=100,tonemap=hable:desat=0,zscale=p=bt709:t=bt709:m=bt709:range=pc,format=nv12,hwupload=extra_hw_frames=64,scale_qsv=1080:-1 -preset 7 -global_quality 23 upload/encoded-video/4ef.../...c7d.mp4 ``` As stated in original post, we aren't using hardware decoding, by enabling this I see about a 5% reduction in CPU load. I get a 5% improvement also by setting the `preset` to `fast`. I am not super familiar with ffmpeg but the remainder of extra cpu load is coming from the filters. Is there a reason we need to do tone-mapping and all the zscale options? If I trim it down to the following, I get about 75% reduction in CPU load. ``` /usr/lib/jellyfin-ffmpeg/ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -c:v hevc_qsv -i /config/test.MOV -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -v verbose -vf format=nv12,hwupload=extra_hw_frames=64,scale_qsv=1080:-1 -preset fast -global_quality 23 /config/test_OUT.mp4 ```
OVERLORD commented 2026-02-05 01:42:07 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Feb 1, 2024):

@rishid Thumbnail generation is still using CPU. If you don't have machine learning setup to use GPU, it will also uses CPU

@alextran1502 commented on GitHub (Feb 1, 2024): @rishid Thumbnail generation is still using CPU. If you don't have machine learning setup to use GPU, it will also uses CPU
OVERLORD commented 2026-02-05 01:42:12 +03:00
Author
Owner
Copy Link

@rishid commented on GitHub (Feb 1, 2024):

Sure understood but specifically the single parent ffmpeg process, which is doing the video transcoding for encoded-videos, is the showing cpu usage of ~800% on my machine.

@rishid commented on GitHub (Feb 1, 2024): Sure understood but specifically the single parent ffmpeg process, which is doing the video transcoding for encoded-videos, is the showing cpu usage of ~800% on my machine.
OVERLORD commented 2026-02-05 01:42:16 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Feb 1, 2024):

Unsure, perhaps passing through configuration is not right?

@alextran1502 commented on GitHub (Feb 1, 2024): Unsure, perhaps passing through configuration is not right?
OVERLORD commented 2026-02-05 01:42:19 +03:00
Author
Owner
Copy Link

@rishid commented on GitHub (Feb 1, 2024):

I completely forgot there are a lot of config knobs for Video Transcoder settings available in Immich - I think all my observations can be controlled already.

@rishid commented on GitHub (Feb 1, 2024): I completely forgot there are a lot of config knobs for Video Transcoder settings available in Immich - I think all my observations can be controlled already.
OVERLORD commented 2026-02-05 01:42:22 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 22, 2024):

For Quick Sync, I got VPP tone-mapping working, but OpenCL doesn't work (something about not being able to allocate memory to the OpenCL device) and Vulkan is almost thrice as slow because it doesn't support zero-copy like it does for CUDA. VPP doesn't have the tone-mapping settings we use for other backends, but it is also the fastest option and tailored specifically for Intel devices. I can use that for QSV and let VAAPI use OpenCL (once I figure out how to get it to work).

@mertalev commented on GitHub (May 22, 2024): For Quick Sync, I got VPP tone-mapping working, but OpenCL doesn't work (something about not being able to allocate memory to the OpenCL device) and Vulkan is almost thrice as slow because it doesn't support zero-copy like it does for CUDA. VPP doesn't have the tone-mapping settings we use for other backends, but it is also the fastest option and tailored specifically for Intel devices. I can use that for QSV and let VAAPI use OpenCL (once I figure out how to get it to work).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1407
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?