Skip to content

[feat] Include description (descr) in detailed listing #3513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

jack-morrison
Copy link
Member

@jack-morrison jack-morrison commented Jul 2, 2025

To accommodate a description string, this includes a small adjustment to the details formatting for variant # and file path, putting each on a new line.

Example output:

user@myhost:~/reframe-examples/tutorial$ reframe -C config/baseline_environs.py -c deps/parameterized.py -L
[ReFrame Setup]
  version:           4.9.0-dev.1
  command:           '/usr/local/share/reframe/bin/reframe -C config/baseline_environs.py -c deps/parameterized.py -L'
  launched by:       user@myhost
  working directory: '/home/user/reframe-examples/tutorial'
  settings files:    '<builtin>', 'config/baseline_environs.py'
  selected system:   'tutorialsys'
  check search path: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  stage directory:   '/home/user/reframe-examples/tutorial/stage'
  output directory:  '/home/user/reframe-examples/tutorial/output'
  log files:         '/tmp/rfm-ajxxgyr9.log'
  results database:  [off] '/home/user/.reframe/reports/results.db'

[List of matched checks]
- TestA %z=0 /7a14ae93
  variant: 0
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestA %z=1 /8258ae7a
  variant: 1
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestA %z=2 /423a76e9
  variant: 2
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestA %z=3 /e62d23e8
  variant: 3
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestA %z=4 /b1aa0bc1
  variant: 4
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestA %z=5 /536115e0
  variant: 5
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test A
- TestB /cc291487
  variant: 0
  file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
  description: Test B
    ^TestA %z=9 /ca1c96ee
      variant: 9
      file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
      description: Test A
    ^TestA %z=8 /75b6718c
      variant: 8
      file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
      description: Test A
    ^TestA %z=7 /1d87616c
      variant: 7
      file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
      description: Test A
    ^TestA %z=6 /06c8e673
      variant: 6
      file: '/home/user/reframe-examples/tutorial/deps/parameterized.py'
      description: Test A
Found 11 check(s)

Closes #3458.

@jack-morrison
Copy link
Member Author

jack-morrison commented Jul 2, 2025

I'm having a bit of difficulty getting the new tests to run locally. It's not clear to me what's different about these test classes vs others in the project.

@vkarak, could you please give these a look and point out anything you see me doing incorrectly? Others seem to use the @rfm.simple_test decorator?

== warnings summary ==

unittests/resources/checks/descr_test.py:5
  /usr/local/share/reframe/unittests/resources/checks/descr_test.py:5: PytestCollectionWarning: cannot collect test class 'TestWithDescription' because it has a __init__ constructor (from: unittests/resources/checks/descr_test.py)
    @rfm.simple_test

unittests/resources/checks/descr_test.py:19
  /usr/local/share/reframe/unittests/resources/checks/descr_test.py:19: PytestCollectionWarning: cannot collect test class 'TestWithoutDescription' because it has a __init__ constructor (from: unittests/resources/checks/descr_test.py)
    @rfm.simple_test

unittests/resources/checks/descr_test.py:33
  /usr/local/share/reframe/unittests/resources/checks/descr_test.py:33: PytestCollectionWarning: cannot collect test class 'TestWithEmptyDescription' because it has a __init__ constructor (from: unittests/resources/checks/descr_test.py)
    @rfm.simple_test

@jack-morrison jack-morrison changed the title [feat] Print description with detailed listing [feat] Include description (descr) in detailed listing Jul 2, 2025
@vkarak vkarak requested review from vkarak and teojgo July 2, 2025 21:28
@vkarak vkarak added this to the ReFrame 4.9 milestone Jul 2, 2025
@jack-morrison jack-morrison moved this from Todo to In Progress in ReFrame Backlog Jul 16, 2025
@jack-morrison jack-morrison moved this from In Progress to Todo in ReFrame Backlog Jul 16, 2025
Copy link

codecov bot commented Jul 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.26%. Comparing base (b57f957) to head (4537a1f).
Report is 36 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3513      +/-   ##
===========================================
+ Coverage    91.16%   91.26%   +0.10%     
===========================================
  Files           62       62              
  Lines        13253    13364     +111     
===========================================
+ Hits         12082    12197     +115     
+ Misses        1171     1167       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jack-morrison jack-morrison marked this pull request as draft July 17, 2025 15:16
@jack-morrison jack-morrison marked this pull request as ready for review July 17, 2025 16:02
@jack-morrison
Copy link
Member Author

Now with passing tests and codecov 🙂

Copy link
Contributor

@vkarak vkarak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the unit tests are a bit over-engineered for such a trivial addition :-) This would lead us to maintain much more code. I would therefore suggest to simply extend the existing test_list_with_details by setting/unsetting etc. the descr field of some of the existing tests in frontend_checks.py.

@@ -11,6 +11,7 @@ class stream_test(rfm.RunOnlyRegressionTest):
valid_systems = ['*']
valid_prog_environs = ['*']
executable = 'stream.x'
descr = 'Run the STREAM memory bandwidth benchmark.'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not relevant for this PR.

Comment on lines +104 to +105
if hasattr(u.check, 'descr') and u.check.descr:
details_fields.append(f'description: {u.check.descr}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would always add the description even if it's not set. I would do the following:

  1. If the attribute is not present (pretty uncommon, as it would only happen if a user specifically does descr = required in a test), I would print <undefined>
  2. If it's empty, I would print <n/a>.

Also as a style comment, we always leave a blank line after if/for/while etc. blocks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

Show also the test description with --list-detailed option
2 participants