Reproducing pytorch's nn.MultiheadAttention

[Hawar-Dzaee]

This notebook investigates and explains PyTorch's out-of-the-box nn.MultiheadAttention, and its arguments in order to reproduce @rasbt's MultiheadAttention class from Chapter 3.

https://github.com/Hawar-Dzaee/Foundation-LLM-from-scratch/blob/main/pytorch_notebooks/reproducing_pytorch_mha%20copy.ipynb

[rasbt]

Looks awesome, thanks for sharing this!