CH07: Fine-Tuning is Faster on my CPU than GPU (RTX 4060 Ti)

[wongzc]

Thanks for the amazing course — I’ve really learned a lot from it!

I'm currently working through CH07 to fine-tune the pre-trained model on instruction data.

Using the exact code from Section 7.6 (no changes), device="cuda" on my 4060 Ti GPU, and it takes about 25 minutes just to reach Ep1 Step 000025.
Then, I switched to device="cpu", and it finishes the entire fine-tuning in ~24 minutes.

GPU is detected (torch.cuda.is_available() == True) and shows ~97% utilization during training, but it still runs significantly slower than CPU.

Just wondering — is this expected? Or could something be misconfigured on my end?
Appreciate any advice, and thanks again for the great learning experience!

[d-kleine]

For me, this sounds not okay. I just have tested it on my setup (3080 TI) and I reached the point Ep 1 (Step 000025) within 10 seconds. Would it be possible for you to share the notebook including your code and outputs from the CUDA run?

Do you have the 8GB or 16GB version of the 4060 TI?

I suspect that the model is not running on the GPU, even though it shows utilization during training.

[wongzc]

I am using the 8GB version of the 4060 Ti.
As for the code, I’m running it directly from Section 7.6 of the training without any changes, aside from installed torch with cuda

I am suspecting is it because of the GPU memory overflow?
GPU utilization hovers around 97%, and VRAM usage hits ~7.7 / 8.0 GB, with some spill into shared memory.

[d-kleine]

I am suspecting is it because of the GPU memory overflow?

Yeah, might be–8 GB is not much memory. But I think once it would overflow, the model model would collapse, break the execution and return an error then.

I suspect that for some reason that only some of the computations will be performed on the GPU, but not all of them (e.g., one or more tensors run on CPU).

If you can, please double-check the code if there any changes (so called "diffs") to the provided solution code, for example with VS Code, (if you are familiar with using that). Or download another copy of the notebook without editing it and run it, too see whether the whether the training runs on GPU then (faster) or not.

[wongzc]

If you can, please double-check the code if there any changes (so called "diffs") to the provided solution code, for example with VS Code, (if you are familiar with using that). Or download another copy of the notebook without editing it and run it, too see whether the whether the training runs on GPU then (faster) or not.

Thanks Kleine, I tried git clone the code into my local, pip install from requirements.txt ( uninstalled torch, and reinstalled with torch-cuda from pytorch site) and rerun the fine-tuning
but the progress is still very slow ( taking about 10 mins to reach Ep 1 (Step 000010)

this is what I took from task manager when the fine tuning is running

[d-kleine]

I see–unfortunately, I don't have any idea why the training runs so slow on your GPU. I am very sorry that I cannot help you on this issue🙁

@rasbt is currently recovering from an injury, he will take a look into this as soon as he's feeling better.

[wongzc]

No worries at all! really appreciate you for taking time to test and respond
Wish Sebastian a speedy recovery!