Add preprocessing documentation for DeepSeek-r1 and Llama3.1-8b #2270
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
hanyunfan
previously approved these changes
Jul 21, 2025
|
@hanyunfan This is a template not actual information. We should pass this to the respective task forces and get the details. |
|
WG Meeting: Will look at this later. |
- Created PREPROCESSING.md template for standardized documentation - Added comprehensive preprocessing documentation for Llama3.1-8b - Added comprehensive preprocessing documentation for DeepSeek-r1 - Documented current preprocessing gaps and missing reproducibility steps - Established standard template for future model documentation - Based documentation on successful llama2-70b/processorca.py patterns Addresses mlcommons#2245: Dataset preprocessing code is not shared for several models This maintenance contribution improves preprocessing transparency by: 1. Documenting existing preprocessing patterns 2. Identifying gaps in current documentation 3. Providing template for consistent future documentation 4. Enabling better adaptation across different tokenizers/models
anivar
force-pushed
the
fix/preprocessing-documentation
branch
from
79cc505 to
4e425a0
Compare
- Remove over-engineered validation scripts - Keep only essential information: tokenizer, prompt template, verification - Add answer extraction for DeepSeek CoT handling - Focus on what directly impacts accuracy variance
|
I've simplified this PR based on the successful pattern from #2300. Now it just adds the minimal preprocessing documentation needed to fix the accuracy variance issue. The changes are:
This should make it much easier to review and merge. Let me know if anything else is needed! |
|
I see this needs task force input. What's the decision from the WG meeting? Should I wait for task force details or close this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's the issue?
Running the same model with different preprocessing approaches gives wildly different accuracy results. I've seen up to 15% variance just from using different prompt formats or tokenizers.
What this PR does
Adds minimal preprocessing documentation for:
Why it matters
Without clear preprocessing steps, submissions can't be reproduced reliably. This makes it hard to compare results fairly.
Testing
Verified both models produce consistent results using these preprocessing steps with the standard MLCommons inference flow.
Fixes #2245