mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-07-16 05:43:54 +03:00
Use keyframes as scrubbing thumbnails for Trickplay, providing a ~110x speed boost for Trickplay image generation #5596
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @n00mkrad on GitHub (Apr 10, 2024).
Currently, trickplay uses the ffmpeg fps filter to extract scrubbing thumbnails.
This is slow as it needs to decode the entire video.
Simply extracting all keyframes instead is roughly 110 times faster (244 vs 2.2 FPS, tested on Win11, Ryzen 7950X3D Eco Mode)
Current ffmpeg command for 10s interval:
My proposal, 110x faster:
Note the
-skip_frame nokeybefore the input, which tells the decoder to only decode keyframes, and-vsync passthroughbefore the output, which avoids duplicated frames in the output as ffmpeg would otherwise duplicate frames to match the input frame rate.I'm currently looking at the code and trying to get it to work, though I'm not familiar with the codebase and how the timestamps are stored, if at all.
@jellyfin-bot commented on GitHub (Apr 10, 2024):
Hi, it seems like your issue report has the following item(s) that need to be addressed:
This is an automated message, currently under testing. Please file an issue here if you encounter any problems.
@gnattu commented on GitHub (Apr 10, 2024):
You will need much more work for this to align to our current behaviors.
The key frame interval in videos is not a constant value, and we do support the arbitrary trickplay thumbnail interval settings. When this interval is smaller than the key frame interval then you need to align the key frame to the closet timing. If the key frame interval is larger than the user-specified interval you may need to duplicate this frame to match the required thumbnail count. In either case, this will generate slightly different thumbnails to our current behavior.
I’m personally fine with this as long as the generated thumbnails have correct timing
@nyanmisaka commented on GitHub (Apr 10, 2024):
Only software decoders and hardware accelerators may benefit from this. QSV and CUVID cannot, as these decoders do not support skip_frame option.
7e59c4f908/libavcodec/h264_slice.c (L2128)@n00mkrad commented on GitHub (Apr 10, 2024):
I honestly fail how this is relevant as the speedup from doing software decoding on just keyframes is still much much faster than HW decoding all frames.
Ultimately it could just be optional.
@n00mkrad commented on GitHub (Apr 10, 2024):
Well, how does it work currently?
My proposal would certainly require timestamps to be stored, but I'm a bit confused what the current method is - Is the interval simply loaded from the settings? Because that'd mean that all Trickplay data would become worthless as soon as the option is changed.
@nyanmisaka commented on GitHub (Apr 10, 2024):
Since decoding keyframes is cheap, P and B frames will be ignored and thus save a lot of CPU cycles. Just like few people care about hardware decoding MJPEG. Also, the FPS filters don't need to wait for several frames before emitting a frame to the encoder.
@gnattu commented on GitHub (Apr 11, 2024):
Sorry each file doesn't have stored timestamp and probably hard to change it to unless you want to rewrite how trickplay works in a whole.
It will create a database entry upon generation, so it is not loaded from the settings when reading images. It calculates the defined interval, puts multiple small images into a larger one, and then generates an HLS for those image sequences. Each of these larger images is called a tile, and a tile includes 100 thumbnails by default. The player would expect the interval or frame time between these 100 images to be equal, and for each tile, the server will calculate a tile duration using the thumbnail interval multiplied by the number of thumbnails in that tile.
Now you will know why it is a big problem that the B-frame interval is not a constant and you will need more works. You can save timestamp for the b-frame images and do you calculation to provide as close as possible alternate frame for the required timestamp(calculated using interval * image number) to make it work.
@n00mkrad commented on GitHub (Apr 11, 2024):
I just tested something - We can just leave the fps filter in the command, that way it reliably outputs the correct timestamps.
Only downside is the potentially lower accuracy as the thumbnail might be a duped/dropped frame, though this should not be an issue unless intervals of <10s are used.
@gnattu commented on GitHub (Apr 11, 2024):
This is what's my main concern and this is a supported use case.
This should be an opt-in feature and have a configuration panel. Also we may need to make this option to be mutually exclusive with certain hardware decoders.
@n00mkrad commented on GitHub (Apr 11, 2024):
I tried what I mentioned in the previous comment (adding
-skip_frame nokeywithout removing the fps filter) and it works reliably. Blu-ray has a max 1s keyframe interval, streaming services around 2-4s, so I think accuracy is good enough.But yes it should be an option considering it does have minor downsides.