support multiple base models

[rootfs]

Tested with modernbert-base and large, deberta-v3, and minilm

[Copilot]

Pull Request Overview

This PR expands the codebase to support multiple base models for PII classification and multitask BERT fine-tuning. Previously limited to MiniLM, the system now supports various BERT variants including ModernBERT, DeBERTa v3, RoBERTa, ELECTRA, and others.

• Adds configurable model support with command-line interface for model selection
• Implements GPU device detection and optimized batch sizing per model
• Introduces comprehensive test framework for model accuracy evaluation

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pii_model_fine_tuning/requirements.txt	New dependencies for multiple model support including transformers, torch, and tiktoken
pii_model_fine_tuning/pii_bert_finetuning.py	Core refactoring to support multiple models with device detection and tokenizer fallbacks
multitask_bert_fine_tuning/multitask_bert_training.py	Extended multitask training with model configurations and progress tracking
multitask_bert_fine_tuning/multitask_accuracy_test.py	New accuracy testing framework for validating model performance across tasks

Comments suppressed due to low confidence (3)

pii_model_fine_tuning/requirements.txt:1

• The version 2.6.0+cu124 appears to be non-existent. PyTorch 2.6.0 has not been released as of my knowledge cutoff. Consider using a stable version like torch>=2.4.0+cu124 or torch>=2.3.0+cu124.

torch>=2.6.0+cu124

pii_model_fine_tuning/requirements.txt:13

• Protobuf version 6.0.0 does not exist. The latest stable versions are in the 4.x or 5.x series. Consider using protobuf>=4.21.0 or protobuf>=5.26.0.

protobuf>=6.0.0

[Copilot]

The condition if batch_size == 0 will never be true because the argparse default is 0, but the parameter default is 16. This logic should check against the actual argparse default or restructure the condition.

Suggested change

	# Auto-optimize batch size based on model if using default
	if batch_size == 0: # Default batch size
	# Auto-optimize batch size based on model if batch_size is set to 0 (default)
	if batch_size == 0:

[Copilot]

The condition if optimized_batch_size != batch_size will always be true when batch_size is 0 (the argparse default), making the optimization message misleading. The logic should compare against the intended default value.

Suggested change

	if optimized_batch_size != batch_size:
	if batch_size != 0 and optimized_batch_size != batch_size: