Organize koios benchmarks #2469

poname · 2023-12-09T20:10:24Z

This PR organizes the koios benchmarks in VTR and addresses a few open issues.

The koios tests are divided into 4 groups according to size and design features: medium, large, proxy, and system-verilog (marked with *_medium, *_large, *_proxy, and *_sv).
Two frontends are supported: Parmys and Odin (marked with *_odin where possible).
The complex DSP features are disabled in tests marked as*_no_hb.

The medium and large categories are based on the original publication and #2419.
The large, proxy, and sv groups are added to weekly suite.
The medium group is added to nightly suite.
The proxy tests are from: koios_proxy_benchmarks.
The deepfreeze style tests require system-verilog parser and are tested separately.

The golden results for weekly suites are generated.
The target architecture for *_no_hb tests changed from k6_frac_N10_frac_chain_depop50_mem32K_40nm to k6FracN10LB_mem20K_complexDSP_customSB_22nm and golden results updated.
The koios ReadME file updated.
The developer guide on how to run the koios benchmarks updated.

Description

Koios benchmarks are tested by the following tasks in VTR:

Suite	Test Description	Target	Complex DSP Features	Config file	Frontend	Parser
Nightly	Medium designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_nightly_test4/koios_medium	Parmys
Nightly	Medium designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_nightly_test4/koios_medium_no_hb	Parmys
Nightly	Medium designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_nightly_test4_odin/koios_medium	Odin
Nightly	Medium designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_nightly_test4_odin/koios_medium_no_hb	Odin
Weekly	Large designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_large	Parmys
Weekly	Large designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_weekly/koios_large_no_hb	Parmys
Weekly	Large designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_large_odin	Odin
Weekly	Large designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_weekly/koios_large_no_hb_odin	Odin
Weekly	Proxy designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_proxy	Parmys
Weekly	Proxy designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_weekly/koios_proxy_no_hb	Parmys
Weekly	deepfreeze designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_sv	Parmys	System-Verilog
Weekly	deepfreeze designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_weekly/koios_sv_no_hb	Parmys	System-Verilog

The following are only for regression testing and less important for benchmarking:

Suite	Test Description	Target	Complex DSP Features	Config file	Frontend
Strong	A test circuit to check the architecture file	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_strong/koios_test	Parmys
Strong	Same test circuit without hard blocks	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_strong/koios_test_no_hb	Parmys
Strong	A test circuit to check the architecture file	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_strong_odin/koios_test	Odin
Strong	Same test circuit without hard blocks	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_strong_odin/koios_test_no_hb	Odin
Nightly	The `conv_layer.v` design	multiple	✓	vtr_reg_nightly_test4/koios_medium_multi_arch	Parmys
Nightly	The `conv_layer.v` design	multiple	✓	vtr_reg_nightly_test4_odin/koios_medium_multi_arch	Odin
Nightly	Other designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_nightly_test6/koios_other	Parmys
Nightly	Other designs	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml		vtr_reg_nightly_test6/koios_other_no_hb	Parmys
Nightly	The `bwave_like.fixed.small.v` design	multiple	✓	vtr_reg_nightly_test6/koios_other_multi_arch	Parmys
Weekly	The `dla_like.large.v` design	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_dla_large	Parmys
Weekly	The `bwave_like.float.large.v` design	k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml	✓	vtr_reg_weekly/koios_bwave_float_large	Parmys

Related Issue

#2416
#2401
#1782

These 2 issues can be closed since Yosys+Odin is removed from VTR and Parmys has no issues with the Koios benchmarks:
#2104
#2131

And this PR: #2148

Motivation and Context

How Has This Been Tested?

Types of changes

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation
I have updated the documentation accordingly
I have added tests to cover my changes
All new and existing tests passed

…est-koios

poname · 2023-12-20T17:28:45Z

@vaughnbetz This PR organizes the Koios benchmarks according to circuit size and design features.
And makes it easier to refer to specific settings in the future.
5 open issues and one old PR can be closed.
Missing golden results added, documents updated, and settings adjusted.
Please refer to the PR description for more details.

Ready to merge.

vaughnbetz · 2023-12-20T19:55:06Z

Does the QoR gathering procedure at https://docs.verilogtorouting.org/en/latest/README.developers/#example-koios-benchmarks-qor-measurement have to change? Or is it still the same regtests that need to be run and parsed?

Adding @aman26kbm and @amin1377 for their comments.

poname · 2023-12-20T20:06:12Z

@vaughnbetz a few adjustments were required, already updated. Please check here:
https://vtr-verilog-to-routing--2469.org.readthedocs.build/en/2469/README.developers/#example-koios-benchmarks-qor-measurement

README.developers.md

aman26kbm

Do we not want to add tasks that enable and disable hard_mem? That would help verify Parmys' memory inference features.

I am not sure, but I think I had those before.

vtr_flow/benchmarks/system_verilog/koios_deepfreeze/deepfreeze.style1.sv

aman26kbm · 2023-12-21T04:39:06Z

I added a few comments. Overall, this is a great effort. I appreciate all the work you've put in, @poname. Thank you very much!

poname · 2023-12-22T17:45:41Z

I added a few comments. Overall, this is a great effort. I appreciate all the work you've put in, @poname. Thank you very much!

You're welcome!

vaughnbetz

Looks good; thanks. Will merge once CI passes. The pull request labeler failed with some permission issue (strange); I will ignore that.

poname · 2023-12-22T22:20:03Z

Looks good; thanks. Will merge once CI passes. The pull request labeler failed with some permission issue (strange); I will ignore that.

Thanks!
These issues can be closed:
#2416
#2401
#1782

#2104
#2131

And this PR: #2148

poname added 30 commits December 3, 2023 12:26

discriminate sv tests

ee58da0

LOG setting and shallow git

9c0df2e

new koios tasks

6ca236b

typo

b312971

golden result for koios weekly

ee51c31

cleaned tasks

4fc5007

removed repetition

61aeaf7

golden updated

235ac36

koios odin tasks

909f29c

original koios proxy

fc905bd

parmys_reg_basic koios_medium

4ce5b9a

Merge branch 'verilog-to-routing:master' into weekly-test-koios

d0e97e7

vtr_reg_nightly_test4 koios_medium

dece812

vtr_reg_nightly_test4_odin koios_medium

253d714

Merge remote-tracking branch 'origin/weekly-test-koios' into weekly-t…

add1d07

…est-koios

vtr_reg_weekly koios_large

8cdc914

koios proxy

0845e4a

koios proxy

99387a5

new koios settings

55f60b8

golden updated

6940679

koios proxy golden

5db168b

proxy cleaned from others

87eca87

proxy readme

9179a21

remove proxy from koios folder

e43b769

move koios deepfreeze to system_verilog folder

c9ec315

weekly koios sv updated

6852b8c

koios proxy additional

fd0d28d

additional koios sv

050cd59

removed proxy from reg_nightly_test6

f037607

organized

9550a0d

poname added 8 commits December 13, 2023 15:47

vtr_reg_nightly_test4_odin/koios_medium_no_hb

377bca3

koios weekly system-verilog deepfreeze

a0dcf56

weekly koios large golden updated

990ae0d

koios readme refined

c76e020

dev readme updated

45997fb

koios readme updated

cf830b5

dev readme updated for koios

68f7255

Merge branch 'master' into weekly-test-koios

449183e

poname marked this pull request as ready for review December 20, 2023 17:29

poname changed the title ~~[WIP] organize koios benchmark~~ organize koios benchmarks Dec 20, 2023

poname changed the title ~~organize koios benchmarks~~ Organize koios benchmarks Dec 20, 2023

vaughnbetz requested review from amin1377 and aman26kbm December 20, 2023 19:54

vaughnbetz requested changes Dec 20, 2023

View reviewed changes

README.developers.md Show resolved Hide resolved

README.developers.md Show resolved Hide resolved

aman26kbm reviewed Dec 21, 2023

View reviewed changes

vtr_flow/benchmarks/system_verilog/koios_deepfreeze/deepfreeze.style1.sv Outdated Show resolved Hide resolved

aman26kbm reviewed Dec 21, 2023

View reviewed changes

vtr_flow/benchmarks/system_verilog/koios_deepfreeze/deepfreeze.style1.sv Outdated Show resolved Hide resolved

poname added 4 commits December 22, 2023 12:21

moved koios_deepfreeze to koios_sv

e7c454d

koios proxy, sv, and no_hb added

9b80759

koios sv readme added

fd1e884

8b7fc51

poname requested a review from vaughnbetz December 22, 2023 17:57

vaughnbetz approved these changes Dec 22, 2023

View reviewed changes

vaughnbetz merged commit 5de9ba4 into verilog-to-routing:master Dec 25, 2023

poname deleted the weekly-test-koios branch December 25, 2023 17:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Organize koios benchmarks #2469

Organize koios benchmarks #2469

Uh oh!

poname commented Dec 9, 2023 •

edited

Loading

Uh oh!

poname commented Dec 20, 2023

Uh oh!

vaughnbetz commented Dec 20, 2023

Uh oh!

poname commented Dec 20, 2023

Uh oh!

Uh oh!

Uh oh!

aman26kbm left a comment

Uh oh!

Uh oh!

Uh oh!

aman26kbm commented Dec 21, 2023

Uh oh!

poname commented Dec 22, 2023

Uh oh!

vaughnbetz left a comment

Uh oh!

poname commented Dec 22, 2023

Uh oh!

Uh oh!

Organize koios benchmarks #2469

Organize koios benchmarks #2469

Uh oh!

Conversation

poname commented Dec 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

poname commented Dec 20, 2023

Uh oh!

vaughnbetz commented Dec 20, 2023

Uh oh!

poname commented Dec 20, 2023

Uh oh!

Uh oh!

Uh oh!

aman26kbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aman26kbm commented Dec 21, 2023

Uh oh!

poname commented Dec 22, 2023

Uh oh!

vaughnbetz left a comment

Choose a reason for hiding this comment

Uh oh!

poname commented Dec 22, 2023

Uh oh!

Uh oh!

poname commented Dec 9, 2023 •

edited

Loading