Performance improvement?

[karajan1001]

Hello.

We are users of pathspec in some other project. I have a performance question.

For a long list of rules (dozens) matches large amount of files (hundreds of thousands) the match_file takes a long time. Is there any method to improve its performance?
For example, using a big regex instead of multiple small ones.

[cpburnz]

Can you provide an example of how you're specifically performing the matches? About how long is a long time? Is it on the order of minutes, hours, or days? This will help me look into the performance issue.

[karajan1001]

For example, It would take 20μs for each file. And 2 seconds for 100k file. And if we use big regex and use if expression to skip the normalization in the UNIX system. It could be 100ms (maybe several hundred for Windows users). This could give great help to user experience in the interactive tools relied on path specification.

[excitoon]

I checked pathspec against gitignorefile on this branch https://github.com/excitoon/3/tree/pathspec . On big project (16188 directories, 204718 files) it is still faster:

real	0m43.853s

vs

real	0m25.885s

I'll check if I can fix it.

[excitoon]

I made it to:

real	0m28.939s

so far. Thing is, gitignorefile's results are more precise, and if I could afford wrong results, it would be much more fast.