mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-07-16 05:43:54 +03:00
[Issue]: Slow performances when scanning large amount of files inside a directory #3395
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ServeurpersoCom on GitHub (Nov 25, 2021).
Please describe your bug
On my Debian 10.11 Server Jellyfin 10.7.7 installed from APT repo I get a slow web interface when there is any library scanning in progress.
There is the same problem on two servers I5 8GB RAM and on a Intel Core i9-9900K up to 5GHz and 64GB DDR4 and RTX2080
Library on multiples Seagate Exos 16TB drives and about 4000 movies and 150 TVshow
When I add 1 movie the scanning take very long, I don't understand why Jellyfin run entire library scan for one movie added or renamed.
There is no way to make GUI more responsive during scan ? This is not a CPU power problem but I don't understand why...
Jellyfin Version
10.7.7
if other:
No response
Environment
Jellyfin logs
FFmpeg logs
Please attach any browser or client logs here
No JS error
Please attach any screenshots here
No response
Code of Conduct
@ServeurpersoCom commented on GitHub (Nov 25, 2021):
I found a problem in my Apache reverse proxy configuration : the Websocket URL is not /socket but /jellyfin/socket... Showed after some time in js console, now I check if the problem do not repeat...
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
To reproduce the problem it takes a large amount of files in a "movies" directory, Jellyfin rescans all files even for a single change which this slows down the response of the GUI for all the duration of the scan
@cvium commented on GitHub (Nov 26, 2021):
How are the drives connected to the server?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
USB 3 UASP 6 GB here for 3 drive attached (and 1 slower not for Jellyfin library)
(root|~) lsusb
Bus 001 Device 002: ID 8087:8000 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 006: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA
Bus 003 Device 005: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA
Bus 003 Device 004: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA
Bus 003 Device 003: ID 174c:5136 ASMedia Technology Inc. ASM1053 SATA 3Gb/s bridge
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 8086:0186 Intel Corp. WiMAX Connection 2400m
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
(root|~) lsusb -t
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
|__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M
|__ Port 3: Dev 5, If 0, Class=Mass Storage, Driver=uas, 5000M
|__ Port 4: Dev 6, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/9p, 480M
|__ Port 7: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
I get the maximum Seagate Exos hard drive read write bandwidth but I don't know if the USB bridge make a bottleneck in term of "command by seconds".
I will have to switch to on a dedicated motherboard with SATA ... because all my tests are done on USB for now and scan is slow even on powerfull i9
@cvium commented on GitHub (Nov 26, 2021):
And Jellyfin has direct access to the drives? There's no NFS, Samba etc. inbetween?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
Yes 100% local mounts, No network or share
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
Here is my server fstab
UUID=89152d5d-007f-41f1-b379-4b1af49cca1b /mnt/d0 ext4 nofail,auto 0 2
UUID=e6ea84d7-cc25-4fd2-bfe4-fa730167669f /mnt/d1 ext4 nofail,auto 0 2
UUID=6bbd9a08-7295-4892-a39b-fd898a3fc824 /mnt/d2 ext4 nofail,auto 0 2
UUID=11cebeb7-1daa-4028-b911-33c693652ce5 /mnt/d3 ext4 nofail,auto 0 2
Long scan is when Jellyfin have direct access to /mnt/d1/Movies with 4000 movies
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
I can run
find *.mkv -exec ffprobe {} ;
faster than Jellyfin scanning, I think there is a performance issue
@cvium commented on GitHub (Nov 26, 2021):
It does more than just ffprobe though. Which metadata providers do you have enabled?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
All default selected, French and France for language.
Chapter extraction enabled (but tested without extraction on another server same performance problem)
I am available for any test to perform, there is no scan process log ?
For sure it does more than just ffprobe, but why there is no optimization like this on the library scanner :
@cvium commented on GitHub (Nov 26, 2021):
How are your drives mounted? What's the fstab command or whatever you use?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
Here is my server fstab
UUID=89152d5d-007f-41f1-b379-4b1af49cca1b /mnt/d0 ext4 nofail,auto 0 2
UUID=e6ea84d7-cc25-4fd2-bfe4-fa730167669f /mnt/d1 ext4 nofail,auto 0 2
UUID=6bbd9a08-7295-4892-a39b-fd898a3fc824 /mnt/d2 ext4 nofail,auto 0 2
UUID=11cebeb7-1daa-4028-b911-33c693652ce5 /mnt/d3 ext4 nofail,auto 0 2
Drive are ext4 without lazy init inode table in progress;
They are mounted on startup without boot lock if not present
@cvium commented on GitHub (Nov 26, 2021):
What is the output of
stat <filename>? Pick any of your movie files.And are you certain that ffprobe is running on every scan?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
And are you certain that ffprobe is running on every scan?
-> No sorry, I write this as an example of directory modification tracking if I want to code a similar process
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
The best for me is if the file size is the same from a scan to another scan any processing is not required... If the location or file date change it's not required redo the deep analysis of the file thus, we could point the library to another identical film directory but to another location without Jellyfin restarting all the analysis
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
There is only 3 cases where the deep analysis of a given file must be triggered :
@cvium commented on GitHub (Nov 26, 2021):
I don't follow anymore. Is the scanner calling ffprobe on all existing files? Or are you just guessing?
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
I don't know if the scanner call ffprobe on all existing file.
I just ask why Jellyfin is slow to scan, slower than a "find *.mkv -exec ffprobe {} ;"
@cvium commented on GitHub (Nov 26, 2021):
It's hard to say. As mentioned it does a lot more than just ffprobe. Maybe if you enable debug logging it'll be clearer: https://jellyfin.org/docs/general/administration/troubleshooting.html#debug-logging
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
Thanks I enable debug soon and i check how it do and how i can help.
Sorry for my approximate English...
The good question is why it do a lot of processing on every untouched movie only for one file added? it's look useless and make Jellyfin bad for serious movie collection...
This is a problem because it take more than 1 hour on powerful server to add or rename only 1 movie! and during this hour the GUI slow down a lot:(
A lower priority for this task (like Linux nice command) and a more intelligent algorithm is required for the scanning process of Jellyfin.
Do you understand my issue? I don't know if it's a bug or feature request but I'm really annoyed with its poor performance because i make many tiny modifications on movie names and the server is always slow ...
@cvium commented on GitHub (Nov 26, 2021):
I understand your issue but it's not an issue that I have been able to replicate.
@ServeurpersoCom commented on GitHub (Nov 26, 2021):
I send you an access to my Jellyfin and I deleted the message here, but I don't know if github send of the notification mail is instantaneous
@nigiriemoji commented on GitHub (Nov 28, 2021):
I get same issue on Raspberry Pi 4 installed via dietpi. No idea how to troubleshoot whenever scanning, too slow to much things
@Boilerplate4u commented on GitHub (Nov 30, 2021):
I thought I'd just throw in my two cents' worth but this sounds like a classic io starvation. Check and analyse iops for the scanning thread/pid and use ionice/ioprio_set to lower the priority and/or change the scheduling class.
EDIT:
I made a quick search in the repository and couldn't find any setpriority, ionice, ioprio_set or similar. Just a suggestion but I would implement this for all typ of background jobs/procs/threads, also for plugins.
@ServeurpersoCom commented on GitHub (Nov 30, 2021):
The major bug is that Jellyfin spends too much time checking media file that has not been modified since the previous scan... A simple timestamp and/or size check on the media and voila, it will work as well as Amby or Plex. This is a major performance bug which makes it appalling to use with 1000+ movie collection files... But what is the point of installing a media server for 20 medias in 2 directory? I think it's a few lines of code to add to the scanner, anyone know the scanner source name / path?
@nigiriemoji commented on GitHub (Dec 2, 2021):
So it's how it scans that is the issue? I got like 6-7TB of 1080p movies/tvs and plex scans that entire thing in maybe an hour for initial scan but every scan after is a minute or so. I'm on a Raspberry Pi 4 1GB and plex is perfect but jellyfin scan just slows down my device immensely
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
I found how to reproduce the problem on any server without a large movie collection :
list.txt
script.sh.txt
sample_640x360.mkv.txt
(remove .txt to script, sample video and chmod +x the script)
This create a fake large 4100 movies collection from a fake lightweight video.
Jellyfin scan and discover the library, wait...
Just rename one file, launch htop and cry !
@cvium commented on GitHub (Dec 2, 2021):
So you have 4100 movies in the same folder?
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
Yes, but it's not a problem for any modern operating system. Access to individual file is instant. Browse with any bash script of program is instant.
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
Remember Jellyfin abstract all directories to a giant list for each library...
Now I create a script that split all movies into year or decades to check refresh performance...
@cvium commented on GitHub (Dec 2, 2021):
The best setup is really 1 folder per unique movie :)
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
Seriously? This is really crap to deal with! With FileZilla, I get the total bytes essential for me when I add or modify releases. I have a local copy of all movies on Windows and a copy on the Jellyfin Linux server
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
If it's the way to get real refresh performance I can do an abstraction layer with script and symlink...
Real movie directory not in Jellyfin library, and fake one with directories and symlinks.
Abstraction directory for Jellyfin is easy to generate by script... but all symlinks and directories must be deleted and created for each refresh...
Do you think Jellyfin handle correctly symlink for a movie release change and refresh ?
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
I made this script, it work on the real server now.
I renamed my big directory with real MKV files and added this script inside it.
I created the empty target directory "Films" for Jellyfin.
Jellyfin refresh is blasting fast now. Harddrives noise during refresh is not comparable. I wait the first refresh for the 1 file modification performance test.
Tested on my testing server with fake movies :
Symlinks timestamps are preserved compared to the pointed files. without timestamps preserved refresh time is about 20 second for 4000 movies. With timestamps preserved Jellyfin refresh time is less than one second for 1 files modification on 4000 movies.
EDIT : I updated the script for Hardlinks instead of Symlinks because Jellyfin can't get the file for download (and streaming URL for an external native player do not work with symlinks)
@ServeurpersoCom commented on GitHub (Dec 2, 2021):
Jellyfin always need an optimization for large numbers of files inside directory but my workaround script successfully tested.
Jellyfin can refresh 4100 movies each inside sub-directories if few seconds when I add or rename some movies. No more slow GUI or high CPU usage.
For an automatic launch (by a systemd timer unit every 1 minute) the script can be improved with a check for changes inside the source directory before symlinks regeneration.
@Norrox commented on GitHub (May 21, 2022):
@crobibero is the fix merged yet? Still have this slow scan in 10.8
@crobibero commented on GitHub (May 21, 2022):
Yes, this has been a part of the past few betas.
@JamsRepos commented on GitHub (Jun 5, 2022):
How drastic of a time difference should it make? A scheduled scan takes around 10 minutes for me still and halts IO speeds massively on 10.8 beta3.
@HawksRepos commented on GitHub (Jun 5, 2022):
I’m having huge performance issues on every library scan, whether it’s a single file being added or a whole library scan, the IO wait of my server goes up into the 80’s/90’s and kills the whole server. At the same time I can scan another media server program with the same library and the scan is done within seconds without even noticing a performance hit.
@Norrox commented on GitHub (Jun 7, 2022):
My problem was that Jellyfin was writing metadata to my array (slow) my solution was to make sure my share only used my m.2 cache drive
@toby11 commented on GitHub (Sep 29, 2022):
I have this issue as well scanning is very slow and io delays are caused whilst scanning, even when just adding one new file.
Has anyone confirmed this has actually been fixed? As the original commenter mentioned surely it's easy to only process files that have been modified/added.
The comment earlier about best way to have a movie collection is folder per movie surely cannot be true. Who organises their films like that?? It seems like the logic really needs to be looked at in the scanning if that is the case.
@owiofwm2i commented on GitHub (Jul 2, 2023):
I have this problem too, I have 10500 movies, every time jellyfin is too sloooooooooow
@JamsRepos commented on GitHub (Jul 2, 2023):
Don't think there's just priority on fixing this at the minute, I've gone ahead and moved back to Plex for the foreeable future until this can be fixed.
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
Make 1 sub directory for each movie and it can scan more than 20k movies in about 3 minutes (not at first scan or chapter extraction !)
@JamsRepos commented on GitHub (Jul 2, 2023):
Even then, when trying to index the homepage with a large library size, the page is still like crawling through cobble.
@cvium commented on GitHub (Jul 2, 2023):
I have spent hours trying to replicate it. I am done trying. When someone can provide the steps to reproduce it it will get fixed.
@JamsRepos commented on GitHub (Jul 2, 2023):
I've never had an issue directly with the speed of a scan, only the speed of pages trying to load. My issue directly was the IO speeds rinsing the server, when doing a large scan. I wouldn't know how to re-produce my niche element.
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
yes jellyfin has a serious thread management problem. the web interface should not be slowed down during the scan but it does. Both when going to the user list on the dashboard Or when scanning, the web interface becomes slow, even on monster servers with several CPUs overclocked to 5GHz and an astronomical amount of RAM and a RAID of world' faster PCIexpress 5 SSD, yet CPU consumption remains below a few percent. But they are two other issues (not solved)
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
I think it's a I/O bottleneck when it access to library SQLite database file. disabling the write barrier on the SSD where SQLite reside solve the problem but it's dangerous for the whole system and/or database file. they disabled WAL mode for library it's the problem. But this not explain the thread management for dashboard because it must do not make I/O access to the library database lol
@cvium commented on GitHub (Jul 2, 2023):
Even the dashboard must access the db to get sessions, users etc.
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
Yes but WAL disabled only for library, but dashboard show too mutch useless information about users, movie logo, movie posters, and not essential like IP, title + year in text format (event with movies logos art enabled !!!!)
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
also when there more than 10 users the dashboard look crap... with a CSS overflow bug... etc.... not a serious server...
@cvium commented on GitHub (Jul 2, 2023):
WAL is enabled again in 10.9 unstable.
@ServeurpersoCom commented on GitHub (Jul 2, 2023):
Thanks a lot for this information. I can't wait the 10.9 stable (I run on 10.8.10). I run only stable on "production" :)
@brenty411 commented on GitHub (Jul 20, 2023):
On 10.8.10 here, and yes the library scan performance hits are still massive. System becomes unusable when scanning Movies library with 4000+ movies. Each movie has it's own folder (via hardlinks) so this isn't a single/multiple directories problem. I'll do a backup and give 10.9 unstable a try, since this might as well be unstable as well.
Database is hosted on local flash storage, movies are via a large raid array (I/O not saturated during scans). Fingers crossed WAL change fixes this.
@sy19890515 commented on GitHub (Jul 28, 2023):
it only tool an hour to scan 6-7TB of 1080p movies/tvs? I have about 14TB of 2160p movies/tvs, I think if we only consider the number of files, mine are much lesser than yours, but mine takes 2 days to finishe the first scan, and it misses quite a lot of files so I have to manually scan it again for a second and third time... the second time took about a half day (I started it before I went to bed and it finished past noon), the third time took about 4 hours...
emby's first scan took a whole day but after that everytime was pretty fast, all the manual scans after the first scan only take about 30 sec (unless I suddenly add a large folder into the library that is).