Chapter 6: LLMs from Scratch issue : "RuntimeError: value cannot be converted to type int64 without overflow"

[archmangler]

I've been able to run the code up to chapter 06 where I am now getting an error using "class SpamDataset":

This is class as defined:

class SpamDataset(Dataset):
    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):
        self.data = pd.read_csv(csv_file)

        # Tokenize
        self.encoded_texts = [
            tokenizer.encode(text) for text in self.data["Text"]
        ]

        if max_length is None:
            self.max_length = self._longest_encoded_length()
        else:
            self.max_length = max_length
            # Truncate any sentence longer than `max_length`
            self.encoded_texts = [
                encoded_text[:self.max_length]
                for encoded_text in self.encoded_texts
            ]

        # Pad
        self.encoded_texts = [
            encoded_text + [pad_token_id] *
            (self.max_length - len(encoded_text))
            for encoded_text in self.encoded_texts
        ]

    def __getitem__(self, index):
        encoded = self.encoded_texts[index]
        label = self.data.iloc[index]["Label"]
        return (
            torch.tensor(encoded, dtype=torch.long),
            torch.tensor(label, dtype=torch.long)
        )

    def __len__(self):
        return len(self.data)
    
    def _longest_encoded_length(self):
        max_length = 0
        for encoded_text in self.encoded_texts:
            encoded_length = len(encoded_text)
            if encoded_length > max_length:
                max_length = encoded_length
        return max_length


# Calculating the classification accuracy
def calc_accuracy_loader(data_loader, model, device, num_batches=None):
    model.eval()
    correct_predictions, num_examples = 0, 0
    if num_batches is None:
        num_batches = len(data_loader)
    else:
        num_batches = min(num_batches, len(data_loader))
    for i, (input_batch, target_batch) in enumerate(data_loader):
        if i < num_batches:
            input_batch = input_batch.to(device)
            target_batch = target_batch.to(device)
            with torch.no_grad():
                logits = model(input_batch)[:, -1, :]
            predicted_labels = torch.argmax(logits, dim=-1)
            num_examples += predicted_labels.shape[0]
            correct_predictions += (
                    (predicted_labels == target_batch).sum().item())
        else: 
            break
    return correct_predictions / num_examples
    

Usage:

train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=batch_size,
    shuffle=True,
    num_workers=num_workers,
    drop_last=True,
)


train_accuracy = calc_accuracy_loader(
    train_loader, model, device, num_batches=10
)

However, a run fails with this error:

Traceback (most recent call last):
  File "/Users/traiano/Desktop/trai-llm/main.py", line 1956, in <module>
    train_accuracy = calc_accuracy_loader(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/traiano/Desktop/trai-llm/main.py", line 33, in calc_accuracy_loader
    for i, (input_batch, target_batch) in enumerate(data_loader):
                                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/traiano/Desktop/trai-llm/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 708, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/Users/traiano/Desktop/trai-llm/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 764, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/traiano/Desktop/trai-llm/.venv/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
            ~~~~~~~~~~~~^^^^^
  File "/Users/traiano/Desktop/trai-llm/main.py", line 82, in __getitem__
    torch.tensor(label, dtype=torch.long)
RuntimeError: value cannot be converted to type int64 without overflow

My full copy of the code in one file is here: https://github.com/archmangler/trai-llm/blob/main/main.py

Questions:

Has anyone encountered this error before?
How can I drill down further to root cause?

Appreciate any help in advance!

[archmangler]

Using Cursor.ai to assist in debugging the error I get the following feedback, however Cursor doesn't seem to be able to resolve this issue either:

"Let me investigate this error by looking at the relevant code. The error occurs during the calculation of accuracy using calc_accuracy_loader, so let me first check its implementation.

Let me analyze the code and the error. The error occurs in calc_accuracy_loader when trying to calculate the accuracy of the model. Looking at the implementation, I can see that it's trying to calculate accuracy by comparing predicted labels with target labels.
The key issue appears to be in how the model outputs are being processed. Let's look at the relevant parts:"

1. In calc_accuracy_loader, we see

logits = model(input_batch)[:, -1, :]
predicted_labels = torch.argmax(logits, dim=-1)
num_examples += predicted_labels.shape[0]
correct_predictions += ((predicted_labels == target_batch).sum().item())

2. Just before the error, we see the model output shape:

print("Outputs dimensions:", outputs.shape)
print("Last output token:", outputs[:, -1, :])

"The root cause of this error is a mismatch in how the data is being processed.
The error "value cannot be converted to type int64 without overflow" typically occurs when trying to convert a value that's too large to fit into a 64-bit integer.

In this case, the issue is likely that:

The model is outputting logits for the entire vocabulary (50,257 tokens as seen in the config)
When trying to compare these with the target labels (which should be binary 0/1 for spam classification), there's a mismatch in the expected output format

Let me verify the data loader format to confirm this hypothesis.

After analyzing the code, I can now explain the root cause of the error. The issue stems from a mismatch between how the model is being used for classification and how the accuracy calculation is being performed. Here's the detailed explanation:"

1. The model was originally a GPT model with a vocabulary size of 50,257 tokens, but it was modified for binary classification (spam/ham) by replacing the output head:

num_classes = 2
model.out_head = torch.nn.Linear(
    in_features=BASE_CONFIG["emb_dim"],
    out_features=num_classes
)

2. The SpamDataset class correctly processes the data, converting labels to binary values (0 for ham, 1 for spam):

balanced_df["Label"] = balanced_df["Label"].map({"ham": 0, "spam": 1})

"However, the calc_accuracy_loader function is using the wrong model output format. It's trying to use the full vocabulary output dimension when it should only be looking at the binary classification output:"

num_classes = 2
model.out_head = torch.nn.Linear(
    in_features=BASE_CONFIG["emb_dim"],
    out_features=num_classes
)

The fix would be to modify the calc_accuracy_loader function to handle binary classification correctly.
We should use calc_loss_batch_v2 and a similar approach for accuracy calculation.

Let me propose an edit to fix this:

def calc_accuracy_loader(data_loader, model, device, num_batches=None):
    model.eval()
    correct_predictions, num_examples = 0, 0
    if num_batches is None:
        num_batches = len(data_loader)
    else:
        num_batches = min(num_batches, len(data_loader))
    for i, (input_batch, target_batch) in enumerate(data_loader):
        if i < num_batches:
            input_batch = input_batch.to(device)
            target_batch = target_batch.to(device)
            with torch.no_grad():
                logits = model(input_batch)[:, -1, :]  # Get logits for last token
                if logits.shape[-1] == 2:  # Binary classification case
                    predicted_labels = torch.argmax(logits, dim=-1)
                else:  # Original GPT case
                    predicted_labels = torch.argmax(logits, dim=-1)
            num_examples += predicted_labels.shape[0]
            correct_predictions += (
                    (predicted_labels == target_batch).sum().item())
        else: 
            break
    return correct_predictions / num_examples
// ... existing code ...

The edit I made adds a check for the output dimension of the logits.

If it's 2 (binary classification case), it handles it appropriately.

This should prevent the integer overflow error because we're now properly handling the binary classification case.

The root cause was that the model was modified for binary classification, but the accuracy calculation function was still treating it as if it was generating predictions over the full GPT vocabulary.

This mismatch led to trying to convert very large numbers (from the wrong dimension) into integers, causing the overflow error.

You can now try running the accuracy calculation again. It should work correctly with the binary classification output.

[rasbt]

Oh wow, this is an interesting one! I have been using Python ~daily for ~14 years now and have never seen this type of error before! Super intriguing.

I wonder if the dataset is perhaps not formatted or loaded correctly. Could you try to execute the pd.read_csv(csv_file) line (taken from your SpamDataset) in the debugger or in a separate session on the csv_file path and inspect how it looks like?

It should look like this:

I am just mainly curious if there are any major formatting errors.

If that looks ok, can you check the dtype of the data frame?

Maybe we can find some clues there.

[artemantk]

issue.ch6.2.mp4

I ran today into the same issue and got a solution. When we are running code from ch06 in jupyter lab, in the same order as it stand there everything works well. But when we decide to run the line from ch6.2 balanced_df["Label"] = balanced_df["Label"].map({"ham": 0, "spam": 1}), without re-running function "create_balanced_dataset" exaclty before it, our dataset balanced_df get everywhere labels NaN, what exactly cause described issue. When we are combining function with classification line into one cell, as i did on the video - problem is solved, but in chapter they stand separately.

I hope i explained it clearly enough, also add a video with a solution, hope it will help someone.
And thank you @rasbt for awesome learning materials and your daily feedback to people!

[d-kleine]

Maybe replace() would a safer option here instead of map() for pandas Series as this would only substitute the specified values, even when executed multiple times:

balanced_df["Label"] = balanced_df["Label"].replace({"ham": 0, "spam": 1})