Question about unused instance variable in CausalAttention class

[cihankarabulut]

In the CausalAttention class implementation on pg. 81, I noticed that self.d_out = d_out is assigned in the init method, but this instance variable doesn't seem to be used anywhere else in the class. The output dimension is already passed directly to the linear layers (nn.Linear(d_in, d_out, bias=qkv_bias)), so I'm curious:

1.Is there a specific reason for storing d_out as an instance variable?
2.Is this intended for future extensions of the class?
3. Or is this simply a remnant from a previous version that could be safely removed?

I'm trying to understand best practices for transformer implementations and when it makes sense to store parameters as instance variables versus just using them directly.

[casinca]

That's funny because I remember seeing this one when reading the book but never PR for consistency with the notebook.

I can't talk for Sebastian but pretty sure, it's your point 2 and 3.
This specific class isn't really used, it's just an intermediate step to better grasp the actual GPT2 MultiHeadAttention (MHA) class that comes later on, in p.86.

You'll notice that for MHA, self.d_out is actually used for reshaping the context tensor to a 3D shape. But in the context of CausalAttention, it's probably just a remnant or code reused from the MHA class.

[cihankarabulut]

Thanks for the explanation! That makes perfect sense. I haven't gotten to the MultiHeadAttention class yet in my reading, so I didn't realize that self.d_out would be used there.