Skip to content

Conversation

muhammad-othman
Copy link
Member

Description

This PR adds a new CBOR performance benchmark runner for the AWS SDK for .NET. To measure and compare the performance of the CBOR protocol against the existing protocols across real AWS service APIs.

Motivation and Context

DOTNET-7728

Testing

Ran the benchmarks and generated the benchmark artifacts.

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project
  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I have read the README document
  • I have added tests to cover my changes
  • All new and existing tests passed

License

  • I confirm that this pull request can be released under the Apache 2 license

@muhammad-othman muhammad-othman force-pushed the muhamoth/DOTNET-7728-cbor-perf branch from 3d6238d to cf80923 Compare September 4, 2025 19:34
@dscpinheiro dscpinheiro requested a review from Copilot September 4, 2025 19:51
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a comprehensive CBOR performance benchmarking framework for the AWS SDK for .NET. The framework enables performance comparison between the new CBOR protocol and existing protocols (JSON, Query) across various AWS service operations.

  • Adds a new benchmarking infrastructure with BenchmarkDotNet to measure serialization, deserialization, and total request times
  • Implements benchmark tests for CloudWatch ListMetrics, SecretsManager operations, and synthetic Echo service tests
  • Provides a dual-protocol setup allowing comparison between CBOR and legacy protocols using parallel repository configurations

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
README.md Documentation explaining the benchmark framework, setup requirements, and usage instructions
PrepareBenchmarkServices.ps1 PowerShell script to configure service models for CBOR vs old protocols in parallel repositories
Utils.cs Core utilities for benchmark execution, result storage, AWS resource management, and data serialization
Various benchmark files Specific benchmark implementations for CloudWatch, SecretsManager, and Echo service operations
BaseBenchmarks.cs Abstract base class providing common benchmark infrastructure and measurement logic
Project files Solution and project configuration supporting dual-protocol compilation via conditional compilation

{
request = new ListMetricsRequest
{
Namespace = "TestNamespace",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the result going to be empty with a custom namespace?

I know it depends on the account, but isn't there a value we can use that'll return some values (and actually exercise the marshallers)? From some testing aws cloudwatch list-metrics --namespace AWS/Usage returns results even in an account I don't use often.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to run another Benchmarks before this one that adds metrics to this namespace, I'm currently working on this benchmarks.

{
_client = new AmazonEchoClient(new AmazonEchoConfig
{
DisableRequestCompression = true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, does this matter for the benchmark? I thought the model had to explicitly enable it for the SDK to compress the request...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, just wanted to be consistent and add it to all the service clients.

@muhammad-othman muhammad-othman force-pushed the muhamoth/DOTNET-7728-cbor-perf branch from cf80923 to d949630 Compare September 4, 2025 20:58
@muhammad-othman muhammad-othman force-pushed the muhamoth/DOTNET-7728-cbor-perf branch from d949630 to 0569f6d Compare September 4, 2025 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants