Support cublas computation without requiring CUDA installed

[AsakusaRinne]

related issue: #345

[martindevans]

llama.cpp copies cudart, cublas and cublasLt64 into the release package.

See here: https://github.com/ggerganov/llama.cpp/blob/master/.github/workflows/build.yml#L497

[AsakusaRinne]

What about splitting the packages into three kinds in the future?

• LLamaSharp.Core: only the main package of LLamaSharp, no native library.
• LLamaSharp: main package and all native libraries.
• LLamaSharp.Cuda: main package, native libraries and cublas dependencies, which does not require users to have cuda installed.

[SignalRT]

@AsakusaRinne We should think also on LLaVa (or more generic any multimodal model). Right now this is not managed by the llama.cpp libraries but with an additional library. Until now there is no GPU so it's simple to go with only one package or integrating that DLL on the native libraries, but I suppose that in the near future this will change.

[AsakusaRinne]

@SignalRT I noticed that LLaVA requires another model named CLIP and llama.cpp doesn't provide C APIs for LLaVA but only an example. Do you have any idea about how to support it?

[SignalRT]

@AsakusaRinne Yes, I have a version working, but not yet integrated cleanly on the currrent Executor architecture.

[martindevans]

but not yet integrated cleanly on the currrent Executor architecture.

Just a note on that, the current executors will probably be replaced at some point when we swap over to batch decoding. Just a heads up so you don't spend a tonne of effort on that!

what about splitting the packages into three kinds in the future?

If we're going to split it I'd say there should be an option for CPU-only inference to be installed without any CUDA backend. Just because the CUDA binaries are huge.

[Onkitova]

Hello again.
Was tinkering a little bit with https://github.com/sandrohanea/whisper.net which is also a c# binding, but for another ggerganov project -- whisper.cpp -- not llama.cpp.
The thing is: it also utilizes cuda through cublas. At first, it looked like it won't work without CUDA Toolkit installed as well (our current state), but then I tried a method from issue discussion and... it accidentally worked.

What I want to say is that we can do the same thing and possibly not bloat LlamaSharp with another ~400mb nuget of nvidia libraries. Instead, we possibly can:

1. tweak loading libraries code a little bit to allow loading these necessary nvidia libraries directly from application folder

2. redistribute those libraries as an archive, like ggerganov does with every new release
   

3. add a little how-to notice about cuda utilization to README.md

[AsakusaRinne]

tweak loading libraries code a little bit to allow loading these necessary nvidia libraries directly from application folder

I'm not sure if it will work to specify a path to load them. Generally if these files are not added to PATH environment variable, we should put them the same folder with the library that depends on them. I'll have a try of it :)

redistribute those libraries as an archive, like ggerganov does with every new release and add a little how-to notice about cuda utilization to README.md

It's a good idea. As you mentioned, yes, the sizes of these files are very large, which I ignored before.

[Onkitova]

I'm not sure if it will work to specify a path to load them.

It might work, because -- as I mentioned above -- this is how it works for original llama.cpp and whisper.cpp as well (and it works the same way even in c# binding). Possibly we don't even need specifying path, just not mess with how ggerganov library (bin-win-cublas-cu12.2.0-x64.zip version for example) handles looking for nvidia libraries nearby on launch.

Another example:

• llama.cpp original repo, "bin-win-cublas-cu12.2.0-x64.zip" version
• CUDA Toolkit is not installed, system PATH variables (regarding CUDA) are not presented at all

Don't get me wrong: not rushing you in any way or direction -- just want to point out the findings.

[martindevans]

I've started #361 with an update for new llama.cpp binaries. I'm not sure I'll get time to investigate packaging up the cudart binaries myself, but if anyone wants to work on a PR to modify the Compile Binaries action (this one) to package whatever is necessary I'll be happy to modify my PR to use the new binaries.