[FEATURE] Ability to control workload in scanning process

[kenken1985]

Is your feature request related to a problem? Please describe.
My computer consistently crash within a few hours when the scan process is going on. While the CPU and RAM load stays around 50%, the disk operation is consistently at 100%, causing my system to crash may be due to overheating.

Describe the solution you'd like
I hope it has an option to control the workload of the scanning process, for example 1 album per 5 minutes.

Additional context
I use trackfs (https://github.com/andresch/trackfs) to use Cuesheet for my Navidrome instance, which inccrease the workload too.

[NeptuneHub]

Thanks for the feedback. AudioMuse-AI don't have an option to control the workload during the analysis. It try to do as fast as he can.
If you deploy in Kubernetes (but maybe also docker-compose should allow) you should try to limit the use of CPU maybe to don't use everything.

On kubernetes for example, if you have a 4core cpu, you can set to use only 3. Also be sure to have 8gb of RAM to also avoid crash: it doesn't use it everytime, but in some operation it can use.

About disk operation seems strange: the analysis of a song takes an avarage of 10 seconds, so you already have 10 second from the download of one song and the other because it download one song at time. So the disk shouldn't be the issue.

[kenken1985]

From the process monitor, it seems it is either rq_worker.py or rq_worker_high_priority.py that is having a high disk operation. And it is mostly write operation that is happening instead, so I don't think it is anything about downloading song. I mostly has this problem starting from >1600 album, with my total at 3900. Maybe the Redis database has problem scaling to large number of songs?

[NeptuneHub]

I’m checking if there is any optimisation that I can do (let’s say that this is a constant topic for me) but till now I didn’t find nothing of problematic.
My song library is of 80k+ song and on old i5-6500 4core CPU, 8GB of ram for AudioMuse-AI and all running on NVME SSD I never had this issue.
I had some random crash just in ARM with Raspberry PI 5 and NVME hat, but the Raspberry PI itself was never stable itself with that hat.

On which CPU/RAM are you using it? Is jellyfin ? Which version of jellyfin ? Audiomuse-AI and the Media Serve run on an NVME SSD ? (I never tested AudioMuse-AI on an HDD for example).

[kenken1985]

My laptop is i7-8550U quad core with 14GB RAM. The server is Navidrome which is on an external SSD (and Audiomuse also) , but the music library itself is on external HDD. Maybe the HDD is the issue, but somehow even when I reboot and restart Audiomuse again, the number of finished albums did not seems to be cumulative. Every time it starts it start from at around 1800 (fluctuate), no matter how many albums it finished previously. For now I guess I can only wait for feature update since there is no way I can proceed further.

[NeptuneHub]

I have the song on external usb3 hdd too, so it will nto be that the issue.
The Analysis do a frist "fast" check (for fast I mean a couple of minutes depending of how big his your collection) of what song are already analyzed and meanwhile it analyze the not scanned song.

The update of the status, being a batch, is not timely update, but I can confirm you that it doesn't re-analyze the same songs two time.

I can think about a way to keep it slower to don't use big resource but the best way for now could be just limit the resource assigned to the container. I mean read and write from the hdd shouldn't be an issue.

Just for my knowledge, you use navidrome (which version?) with last version of AudioMuse-AI (v0.7.4-beta).
Do you use docker compose, Kubernetes or what?
Did you set any limit to resources?

Because you have a configuration faster than mine. If you're not running other big stuff in parallel, you shoudln't have issue.

[kenken1985]

I am running Navidrome version 0.58.0 (9dbe0c18). Audiomuse is 0.7.4 beta. Both are using docker compose and I did not set resource limitation, as far as I can tell the CPU and RAM usage stay at aroung 50% so it should be an issue.

I understand the update is not real time. But when before my laptop crash it has gone up to ~2300 album completed, but when I reboot it go back to 1800 displayed as skipped, it is normal behavior? Even if it is crashed I am expecting some data be saved.

I have other server running but shouldn't be resources heavy, unless when Kopia is doing daily backup.

As an alternative I may try to use my main laptop with a Nvidia GPU to see if things can run thru, are there anything to prevent me from migrating the redis and postgres data from a window to a linux machine?

Hopefully I can at least identify if it is the problem on Navidrome side or on Audiomuse side.

[NeptuneHub]

I don’t see any blocker in changing machines. Potentially you can just deploy only the worker on the other machine pointing to the same Redis and PostgreSQL (and off course same media server).

[AlternateIf]

may very well be related to the usb drive. i had problems with my mini pc and connected usb drives too. (not related to audiomuse though)
maybe you can stress test the drive/bus with some operations to check

regarding the skipped message: i think albums that have already been processed will be skipped. that means on your previous run before the crash occured 1800 albums have been analyzed already

[NeptuneHub]

My song library is also on external USB3 drive.

To me this crash are very strange and I’m really overthinking what could be the reason.

Download one song each 10-20 seconds from the media server shouldn’t be an issue.
The analysis of the song in the local disk.. I mean… you’re reading a file.
The write in the database? The same 1 song each 10-20second written there?

I literally run 10+ worker on 4 different node that downloads song from 1 music server/1 external usb disk. And nothing crashed.

I’m not saying that the code is perfect and improvement on the resource usage is still in my mind but on intel I never had this crashes. Only on a Raspberry PI sometimes, happen. But in this case for me is the experimental support of NvME SSD the issue on raspberry more than audiomuse-so itself.

Anyway if you have some log, some metrics to analyze, they can help me to check.

[AlternateIf]

don't think it has anything to do with audiomuse at all. i had the same problem doing backups onto said drive. all drives inside my Terramaster DAS (via usb 3.2) work fine but the only other external thing connected (wd 8tb with external PSU) keeps having problems similar to those mentioned here. (e.g. it also happens when i run audio normalization within jellyfin that also uses this drive because songs are on it)

problaby a bus, driver, or enclosure firmware problem since i even disabled autosuspend for usb

i'll probably just move the drive to the DAS if i can't figure it out

[NeptuneHub]

So I'm going to closing this issue because I'm not able to reproduce, and at the moment I even don't know what to do.

Anyway improve performance is still on my roadmap and I'll keep an eye open to storage read just in case there is something that can be done better. thanks.