-
Notifications
You must be signed in to change notification settings - Fork 867
Add CBOR performance benchmark runner with intial CloudWatch ListMetricsBenchmarks #4000
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: development
Are you sure you want to change the base?
Conversation
3d6238d
to
cf80923
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces a comprehensive CBOR performance benchmarking framework for the AWS SDK for .NET. The framework enables performance comparison between the new CBOR protocol and existing protocols (JSON, Query) across various AWS service operations.
- Adds a new benchmarking infrastructure with BenchmarkDotNet to measure serialization, deserialization, and total request times
- Implements benchmark tests for CloudWatch ListMetrics, SecretsManager operations, and synthetic Echo service tests
- Provides a dual-protocol setup allowing comparison between CBOR and legacy protocols using parallel repository configurations
Reviewed Changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
README.md | Documentation explaining the benchmark framework, setup requirements, and usage instructions |
PrepareBenchmarkServices.ps1 | PowerShell script to configure service models for CBOR vs old protocols in parallel repositories |
Utils.cs | Core utilities for benchmark execution, result storage, AWS resource management, and data serialization |
Various benchmark files | Specific benchmark implementations for CloudWatch, SecretsManager, and Echo service operations |
BaseBenchmarks.cs | Abstract base class providing common benchmark infrastructure and measurement logic |
Project files | Solution and project configuration supporting dual-protocol compilation via conditional compilation |
...chmarks/CborPerformanceBenchmarksRunner/SecretsManagerBenchmarks/DescribeSecretBenchmarks.cs
Outdated
Show resolved
Hide resolved
sdk/test/Performance/CborPerformanceBenchmarks/CborPerformanceBenchmarksRunner/Utils.cs
Show resolved
Hide resolved
sdk/test/Performance/CborPerformanceBenchmarks/CborPerformanceBenchmarksRunner/Utils.cs
Show resolved
Hide resolved
sdk/test/Performance/CborPerformanceBenchmarks/PrepareBenchmarkServices.ps1
Outdated
Show resolved
Hide resolved
{ | ||
request = new ListMetricsRequest | ||
{ | ||
Namespace = "TestNamespace", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't the result going to be empty with a custom namespace?
I know it depends on the account, but isn't there a value we can use that'll return some values (and actually exercise the marshallers)? From some testing aws cloudwatch list-metrics --namespace AWS/Usage
returns results even in an account I don't use often.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The plan is to run another Benchmarks before this one that adds metrics to this namespace, I'm currently working on this benchmarks.
...anceBenchmarks/CborPerformanceBenchmarksRunner/CloudWatchBenchmarks/ListMetricsBenchmarks.cs
Show resolved
Hide resolved
{ | ||
_client = new AmazonEchoClient(new AmazonEchoConfig | ||
{ | ||
DisableRequestCompression = true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious, does this matter for the benchmark? I thought the model had to explicitly enable it for the SDK to compress the request...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really, just wanted to be consistent and add it to all the service clients.
sdk/test/Performance/CborPerformanceBenchmarks/CborPerformanceBenchmarksRunner/Program.cs
Outdated
Show resolved
Hide resolved
...anceBenchmarks/CborPerformanceBenchmarksRunner/CloudWatchBenchmarks/ListMetricsBenchmarks.cs
Outdated
Show resolved
Hide resolved
cf80923
to
d949630
Compare
d949630
to
0569f6d
Compare
Description
This PR adds a new CBOR performance benchmark runner for the AWS SDK for .NET. To measure and compare the performance of the CBOR protocol against the existing protocols across real AWS service APIs.
Motivation and Context
DOTNET-7728
Testing
Ran the benchmarks and generated the benchmark artifacts.
Screenshots (if appropriate)
Types of changes
Checklist
License