Evals does not evaluate all metrics if one fails

[vinod-seshadri-TA]

Title

AgentEvaluator only reports the first failing metric, subsequent metrics not evaluated

Describe the bug

When running tests that involve AgentEvaluator.evaluate_eval_set, if the first metric in the criteria fails, an AssertionError is raised immediately. This causes the test execution to halt, so any subsequent metrics are not evaluated. As a result, the test report does not provide a complete overview of all metric failures for the given agent evaluation.

To Reproduce

Steps to reproduce the behavior:

• Create a test case with multiple metrics in the criteria dict
• Ensure that the first metric will fail and the second metric would also fail if evaluated.
• Run the test with pytest.
• Observe that only the first metric failure is reported; the second is not evaluated or reported.

Expected behavior

All metrics in the criteria should be evaluated, and the test should report failures for all that do not meet the threshold, not just the first one.

Actual behavior

Test execution stops at the first failing metric due to an assertion, and subsequent metrics are not checked.

Potential Fix

Accumulate failures for each metric in a list, and after all metrics are evaluated, raise an AssertionError with details of all failed metrics.

Version observed

v1.0.0

[ankursharmas]

Tests run through pytest generally have an expectation to fail/stop at the first failure itself, so i would apply the same reasoning for this case as well.

I see two options for you here:

1. If you want to continue using pytest approach then you should break down the evals on a per metric basis. Keep the test small. That would help with general maintenance and debugging.
2. Consider using adk eval cli command for greater control, especially what you are looking for: https://google.github.io/adk-docs/evaluate/#3-adk-eval-run-evaluations-via-the-cli

Please re-open the issue, if you think otherwise. Happy to discuss more. :)

[vinod-seshadri-TA]

@ankursharmas I am not able to reopen the issue, but I have some concerns, hence kindly reopen

Regarding option 1

While running within pytest , there is no option to control which metrics to run neither in the evaluate function nor in the eval set json file,

async def evaluate(
      agent_module: str,
      eval_dataset_file_path_or_dir: str,
      num_runs: int = NUM_RUNS,
      agent_name: Optional[str] = None,
      initial_session_file: Optional[str] = None,
  ):

In the code I see that it is hardcoded,
https://github.com/google/adk-python/blob/c224626ae189d02e5c410959b3631f6bd4d4d5c1/src/google/adk/evaluation/agent_evaluator.py#L48C1-L48C18

so how do you propose to implement what you have mentioned.

[ankursharmas]

@vinod-seshadri-TA
Apologies for this mixup. For now I have closed the new issue and we continue the discussion here.

Regarding your last question:

You can create a test_config.json file like this:

tests/integration/fixture/home_automation_agent/test_config.json
or tests/integration/fixture/hello_world_agent/test_config.json

You can place it along with your test files and it will be picked up automatically.

I am going to keep the ticket open, until your issue is fully resolved.

[ankursharmas]

By the way, I am also addressing the original ask of executing all evals for all the metrics. Failures will be gathered along the way and will be reported towards the end.