Skip to content

Skiplist - date histogram sub aggregation performance change #19509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Skiplist - date histogram sub aggregation performance change #19509

wants to merge 1 commit into from

Conversation

asimmahmood1
Copy link
Contributor

Description

Follow up the changes made in #19438 (comment)

I think it was this change: 5d45233 (#19438) showed better results but I'm not seeing it again.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@asimmahmood1 asimmahmood1 self-assigned this Oct 2, 2025
@asimmahmood1
Copy link
Contributor Author

Checked vs this change

ec2-user@ip-172-31-61-197 ~]$ opensearch-benchmark compare -b $checkedin -c $candidate

/ __ ____ ___ ____ / / ____ / / / __ ) ____ / / ____ ___ ____ / /
/ / / / __ / _ / __ \ / _ / __ / ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ / __ `/ / ///
/ // / // / / / / // / / /_/ / / / // / / / / // / / / / / // / / / / / / / / // / / / ,<
_/ ./_// //____/_/_,/_/ _// // //_// //_// /// // //_,// //||
/_/

Comparing baseline
TestExecution ID: 40405b2d-2c15-4cf7-a483-5a0671adc672
TestExecution timestamp: 2025-09-30 21:07:54
TestProcedure: append-no-conflicts
ProvisionConfigInstance: external
User tags: asimmahm=3.3-nyc-dateskiplist-subagg-final

with contender
TestExecution ID: 92af10a8-392e-439d-83aa-ba0e2d3ff637
TestExecution timestamp: 2025-10-02 18:22:38
TestProcedure: append-no-conflicts
ProvisionConfigInstance: external
User tags: asimmahm=3.3-nyc-dateskiplist-subagg-final2

_______             __   _____

/ () ____ / / / /_____ ________
/ /_ / / __ / __ `/ / __ / / __ / / _
/ __/ / / / / / // / / / / // // / / / __/
// /// //_,// /__/_/_// ___/

Metric Task Baseline Contender %Diff Diff Unit
Cumulative indexing time of primary shards 0 0 0.00% 0 min
Min cumulative indexing time across primary shard 0 0 0.00% 0 min
Median cumulative indexing time across primary shard 0 0 0.00% 0 min
Max cumulative indexing time across primary shard 0 0 0.00% 0 min
Cumulative indexing throttle time of primary shards 0 0 0.00% 0 min
Min cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Median cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Max cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Cumulative merge time of primary shards 0 0 0.00% 0 min
Cumulative merge count of primary shards 0 0 0.00% 0
Min cumulative merge time across primary shard 0 0 0.00% 0 min
Median cumulative merge time across primary shard 0 0 0.00% 0 min
Max cumulative merge time across primary shard 0 0 0.00% 0 min
Cumulative merge throttle time of primary shards 0 0 0.00% 0 min
Min cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Median cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Max cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Cumulative refresh time of primary shards 0 0 0.00% 0 min
Cumulative refresh count of primary shards 2 2 0.00% 0
Min cumulative refresh time across primary shard 0 0 0.00% 0 min
Median cumulative refresh time across primary shard 0 0 0.00% 0 min
Max cumulative refresh time across primary shard 0 0 0.00% 0 min
Cumulative flush time of primary shards 0 0 0.00% 0 min
Cumulative flush count of primary shards 1 1 0.00% 0
Min cumulative flush time across primary shard 0 0 0.00% 0 min
Median cumulative flush time across primary shard 0 0 0.00% 0 min
Max cumulative flush time across primary shard 0 0 0.00% 0 min
Total Young Gen GC time 0.032 0.036 0.01% 0.004 s
Total Young Gen GC count 2 2 0.00% 0
Total Old Gen GC time 0 0 0.00% 0 s
Total Old Gen GC count 0 0 0.00% 0
Store size 4.36969 4.36969 0.00% 0 GB
Translog size 5.12227e-08 5.12227e-08 0.00% 0 GB
Heap used for segments 0 0 0.00% 0 MB
Heap used for doc values 0 0 0.00% 0 MB
Heap used for terms 0 0 0.00% 0 MB
Heap used for norms 0 0 0.00% 0 MB
Heap used for points 0 0 0.00% 0 MB
Heap used for stored fields 0 0 0.00% 0 MB
Segment count 10 10 0.00% 0
Min Throughput date_histogram_calendar_interval 1.50108 1.50289 0.12% 0.0018 ops/s
Mean Throughput date_histogram_calendar_interval 1.50176 1.50469 0.19% 0.00293 ops/s
Median Throughput date_histogram_calendar_interval 1.50162 1.50427 0.18% 0.00266 ops/s
Max Throughput date_histogram_calendar_interval 1.50309 1.5083 0.35% 0.00521 ops/s
50th percentile latency date_histogram_calendar_interval 227.48 154.661 -32.01% 🟢 -72.8193 ms
90th percentile latency date_histogram_calendar_interval 248.178 157.889 -36.38% 🟢 -90.2887 ms
99th percentile latency date_histogram_calendar_interval 262.979 161.17 -38.71% 🟢 -101.81 ms
100th percentile latency date_histogram_calendar_interval 263.36 176.769 -32.88% 🟢 -86.5908 ms
50th percentile service time date_histogram_calendar_interval 226.161 153.012 -32.34% 🟢 -73.1485 ms
90th percentile service time date_histogram_calendar_interval 246.997 156.593 -36.60% 🟢 -90.4046 ms
99th percentile service time date_histogram_calendar_interval 261.85 159.494 -39.09% 🟢 -102.357 ms
100th percentile service time date_histogram_calendar_interval 262.355 174.922 -33.33% 🟢 -87.4332 ms
error rate date_histogram_calendar_interval 0 0 0.00% 0 %
Min Throughput date_histogram_calendar_interval_with_filter 1.50943 1.50968 0.02% 0.00025 ops/s
Mean Throughput date_histogram_calendar_interval_with_filter 1.5156 1.516 0.03% 0.0004 ops/s
Median Throughput date_histogram_calendar_interval_with_filter 1.51419 1.51457 0.03% 0.00038 ops/s
Max Throughput date_histogram_calendar_interval_with_filter 1.52811 1.52884 0.05% 0.00073 ops/s
50th percentile latency date_histogram_calendar_interval_with_filter 9.87088 9.75382 -1.19% -0.11707 ms
90th percentile latency date_histogram_calendar_interval_with_filter 11.1966 11.3472 1.35% 0.15061 ms
99th percentile latency date_histogram_calendar_interval_with_filter 13.2912 14.5053 +9.13% 🔴 1.21407 ms
100th percentile latency date_histogram_calendar_interval_with_filter 13.4544 30.6908 +128.11% 🔴 17.2364 ms
50th percentile service time date_histogram_calendar_interval_with_filter 8.4715 8.32173 -1.77% -0.14977 ms
90th percentile service time date_histogram_calendar_interval_with_filter 9.47064 9.69457 2.36% 0.22394 ms
99th percentile service time date_histogram_calendar_interval_with_filter 11.7108 12.5508 +7.17% 🔴 0.84002 ms
100th percentile service time date_histogram_calendar_interval_with_filter 11.8357 29.4954 +149.21% 🔴 17.6597 ms
error rate date_histogram_calendar_interval_with_filter 0 0 0.00% 0 %
Min Throughput date_histogram_fixed_interval_with_metrics 0.236863 0.236365 -0.21% -0.0005 ops/s
Mean Throughput date_histogram_fixed_interval_with_metrics 0.236931 0.236734 -0.08% -0.0002 ops/s
Median Throughput date_histogram_fixed_interval_with_metrics 0.236915 0.236739 -0.07% -0.00018 ops/s
Max Throughput date_histogram_fixed_interval_with_metrics 0.237073 0.237013 -0.03% -6e-05 ops/s
50th percentile latency date_histogram_fixed_interval_with_metrics 357487 358005 0.14% 518.032 ms
90th percentile latency date_histogram_fixed_interval_with_metrics 497919 498335 0.08% 415.677 ms
99th percentile latency date_histogram_fixed_interval_with_metrics 529457 529861 0.08% 403.997 ms
100th percentile latency date_histogram_fixed_interval_with_metrics 532986 533422 0.08% 436.237 ms
50th percentile service time date_histogram_fixed_interval_with_metrics 4214.45 4212.22 -0.05% -2.22602 ms
90th percentile service time date_histogram_fixed_interval_with_metrics 4243.44 4233.65 -0.23% -9.79249 ms
99th percentile service time date_histogram_fixed_interval_with_metrics 4274.8 4286.27 0.27% 11.4718 ms
100th percentile service time date_histogram_fixed_interval_with_metrics 4293.39 4302.34 0.21% 8.94738 ms
error rate date_histogram_fixed_interval_with_metrics 0 0 0.00% 0 %

[INFO] SUCCESS (took 0 seconds)

@asimmahmood1
Copy link
Contributor Author

Baseline (without skiplist) vs Candidate

[ec2-user@ip-172-31-61-197 ~]$ opensearch-benchmark compare -b $baseline -c $candidate

/ __ ____ ___ ____ / / ____ / / / __ ) ____ / / ____ ___ ____ / /
/ / / / __ / _ / __ \ / _ / __ / ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ / __ `/ / ///
/ // / // / / / / // / / /_/ / / / // / / / / // / / / / / // / / / / / / / / // / / / ,<
_/ ./_// //____/_/_,/_/ _// // //_// //_// /// // //_,// //||
/_/

Comparing baseline
TestExecution ID: 65b238a9-325a-4393-b457-0588a57a58fa
TestExecution timestamp: 2025-09-26 19:46:26
TestProcedure: append-no-conflicts
ProvisionConfigInstance: external
User tags: asimmahm=3.3-nyc-dateskiplist-subagg-baseline

with contender
TestExecution ID: 92af10a8-392e-439d-83aa-ba0e2d3ff637
TestExecution timestamp: 2025-10-02 18:22:38
TestProcedure: append-no-conflicts
ProvisionConfigInstance: external
User tags: asimmahm=3.3-nyc-dateskiplist-subagg-final2

_______             __   _____

/ () ____ / / / /_____ ________
/ /_ / / __ / __ `/ / __ / / __ / / _
/ __/ / / / / / // / / / / // // / / / __/
// /// //_,// /__/_/_// ___/

Metric Task Baseline Contender %Diff Diff Unit
Cumulative indexing time of primary shards 0 0 0.00% 0 min
Min cumulative indexing time across primary shard 0 0 0.00% 0 min
Median cumulative indexing time across primary shard 0 0 0.00% 0 min
Max cumulative indexing time across primary shard 0 0 0.00% 0 min
Cumulative indexing throttle time of primary shards 0 0 0.00% 0 min
Min cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Median cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Max cumulative indexing throttle time across primary shard 0 0 0.00% 0 min
Cumulative merge time of primary shards 0 0 0.00% 0 min
Cumulative merge count of primary shards 0 0 0.00% 0
Min cumulative merge time across primary shard 0 0 0.00% 0 min
Median cumulative merge time across primary shard 0 0 0.00% 0 min
Max cumulative merge time across primary shard 0 0 0.00% 0 min
Cumulative merge throttle time of primary shards 0 0 0.00% 0 min
Min cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Median cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Max cumulative merge throttle time across primary shard 0 0 0.00% 0 min
Cumulative refresh time of primary shards 0 0 0.00% 0 min
Cumulative refresh count of primary shards 2 2 0.00% 0
Min cumulative refresh time across primary shard 0 0 0.00% 0 min
Median cumulative refresh time across primary shard 0 0 0.00% 0 min
Max cumulative refresh time across primary shard 0 0 0.00% 0 min
Cumulative flush time of primary shards 0 0 0.00% 0 min
Cumulative flush count of primary shards 1 1 0.00% 0
Min cumulative flush time across primary shard 0 0 0.00% 0 min
Median cumulative flush time across primary shard 0 0 0.00% 0 min
Max cumulative flush time across primary shard 0 0 0.00% 0 min
Total Young Gen GC time 0.033 0.036 0.01% 0.003 s
Total Young Gen GC count 2 2 0.00% 0
Total Old Gen GC time 0 0 0.00% 0 s
Total Old Gen GC count 0 0 0.00% 0
Store size 4.36969 4.36969 0.00% 0 GB
Translog size 5.12227e-08 5.12227e-08 0.00% 0 GB
Heap used for segments 0 0 0.00% 0 MB
Heap used for doc values 0 0 0.00% 0 MB
Heap used for terms 0 0 0.00% 0 MB
Heap used for norms 0 0 0.00% 0 MB
Heap used for points 0 0 0.00% 0 MB
Heap used for stored fields 0 0 0.00% 0 MB
Segment count 10 10 0.00% 0
Min Throughput date_histogram_calendar_interval 1.22898 1.50289 +22.29% 🔴 0.27391 ops/s
Mean Throughput date_histogram_calendar_interval 1.24148 1.50469 +21.20% 🔴 0.26321 ops/s
Median Throughput date_histogram_calendar_interval 1.24395 1.50427 +20.93% 🔴 0.26032 ops/s
Max Throughput date_histogram_calendar_interval 1.24555 1.5083 +21.09% 🔴 0.26275 ops/s
50th percentile latency date_histogram_calendar_interval 14122.3 154.661 -98.90% 🟢 -13967.7 ms
90th percentile latency date_histogram_calendar_interval 19465.8 157.889 -99.19% 🟢 -19308 ms
99th percentile latency date_histogram_calendar_interval 20690.9 161.17 -99.22% 🟢 -20529.7 ms
100th percentile latency date_histogram_calendar_interval 20831.6 176.769 -99.15% 🟢 -20654.8 ms
50th percentile service time date_histogram_calendar_interval 794.394 153.012 -80.74% 🟢 -641.381 ms
90th percentile service time date_histogram_calendar_interval 809.454 156.593 -80.65% 🟢 -652.862 ms
99th percentile service time date_histogram_calendar_interval 846.295 159.494 -81.15% 🟢 -686.801 ms
100th percentile service time date_histogram_calendar_interval 856.366 174.922 -79.57% 🟢 -681.444 ms
error rate date_histogram_calendar_interval 0 0 0.00% 0 %
Min Throughput date_histogram_calendar_interval_with_filter 1.50911 1.50968 0.04% 0.00057 ops/s
Mean Throughput date_histogram_calendar_interval_with_filter 1.51506 1.516 0.06% 0.00094 ops/s
Median Throughput date_histogram_calendar_interval_with_filter 1.51371 1.51457 0.06% 0.00086 ops/s
Max Throughput date_histogram_calendar_interval_with_filter 1.52712 1.52884 0.11% 0.00172 ops/s
50th percentile latency date_histogram_calendar_interval_with_filter 19.384 9.75382 -49.68% 🟢 -9.63021 ms
90th percentile latency date_histogram_calendar_interval_with_filter 20.1579 11.3472 -43.71% 🟢 -8.81071 ms
99th percentile latency date_histogram_calendar_interval_with_filter 23.0539 14.5053 -37.08% 🟢 -8.5486 ms
100th percentile latency date_histogram_calendar_interval_with_filter 23.2335 30.6908 +32.10% 🔴 7.45733 ms
50th percentile service time date_histogram_calendar_interval_with_filter 17.8957 8.32173 -53.50% 🟢 -9.57401 ms
90th percentile service time date_histogram_calendar_interval_with_filter 18.55 9.69457 -47.74% 🟢 -8.85547 ms
99th percentile service time date_histogram_calendar_interval_with_filter 21.1604 12.5508 -40.69% 🟢 -8.60958 ms
100th percentile service time date_histogram_calendar_interval_with_filter 21.3555 29.4954 +38.12% 🔴 8.1399 ms
error rate date_histogram_calendar_interval_with_filter 0 0 0.00% 0 %
Min Throughput date_histogram_fixed_interval_with_metrics 0.21029 0.236365 +12.40% 🔴 0.02607 ops/s
Mean Throughput date_histogram_fixed_interval_with_metrics 0.210669 0.236734 +12.37% 🔴 0.02607 ops/s
Median Throughput date_histogram_fixed_interval_with_metrics 0.210624 0.236739 +12.40% 🔴 0.02612 ops/s
Max Throughput date_histogram_fixed_interval_with_metrics 0.210908 0.237013 +12.38% 🔴 0.0261 ops/s
50th percentile latency date_histogram_fixed_interval_with_metrics 410523 358005 -12.79% 🟢 -52517.9 ms
90th percentile latency date_histogram_fixed_interval_with_metrics 571403 498335 -12.79% 🟢 -73068.8 ms
99th percentile latency date_histogram_fixed_interval_with_metrics 607548 529861 -12.79% 🟢 -77686.3 ms
100th percentile latency date_histogram_fixed_interval_with_metrics 611552 533422 -12.78% 🟢 -78129.9 ms
50th percentile service time date_histogram_fixed_interval_with_metrics 4731.7 4212.22 -10.98% 🟢 -519.474 ms
90th percentile service time date_histogram_fixed_interval_with_metrics 4763.05 4233.65 -11.11% 🟢 -529.407 ms
99th percentile service time date_histogram_fixed_interval_with_metrics 4794.49 4286.27 -10.60% 🟢 -508.225 ms
100th percentile service time date_histogram_fixed_interval_with_metrics 4813.14 4302.34 -10.61% 🟢 -510.803 ms
error rate date_histogram_fixed_interval_with_metrics 0 0 0.00% 0 %

[INFO] SUCCESS (took 0 seconds)

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 2, 2025

❌ Gradle check result for 49de99e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz will be requested when the pull request is marked ready for review anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross will be requested when the pull request is marked ready for review andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar will be requested when the pull request is marked ready for review Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE will be requested when the pull request is marked ready for review CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks will be requested when the pull request is marked ready for review cwperks is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis will be requested when the pull request is marked ready for review dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna will be requested when the pull request is marked ready for review gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 will be requested when the pull request is marked ready for review jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal will be requested when the pull request is marked ready for review kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 will be requested when the pull request is marked ready for review mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh will be requested when the pull request is marked ready for review msfroh is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 will be requested when the pull request is marked ready for review owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta will be requested when the pull request is marked ready for review reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 will be requested when the pull request is marked ready for review Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale will be requested when the pull request is marked ready for review sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli will be requested when the pull request is marked ready for review saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja will be requested when the pull request is marked ready for review shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami will be requested when the pull request is marked ready for review sohami is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@asimmahmood1 asimmahmood1

Labels
Search:Aggregations v3.3.0
Projects
Performance Roadmap
Status: Todo
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@asimmahmood1