Add benchmark_single_table_aws function #417

R-Palazzo · 2025-07-02T13:55:50Z

Resolve #414
CU-86b5ewpu4

what is the expected behavior for the additional_datasets_folder parameter in benchmark_single_table_aws()? Should it be an S3 bucket, or could it be a local dataset also?
Just checking, let me know if the parameter order makes sense, in benchmark_single_table() output_folder comes after timeout while it's the first parameter in benchmark_single_table_aws()

codecov · 2025-07-03T14:35:41Z

Codecov Report

Attention: Patch coverage is 75.88933% with 61 lines in your changes missing coverage. Please review.

Project coverage is 68.74%. Comparing base (cfbd119) to head (917d2e3).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sdgym/benchmark.py	72.34%	39 Missing ⚠️
sdgym/s3.py	63.41%	15 Missing ⚠️
sdgym/result_writer.py	90.14%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #417      +/-   ##
==========================================
+ Coverage   68.56%   68.74%   +0.17%     
==========================================
  Files          20       21       +1     
  Lines        1422     1622     +200     
==========================================
+ Hits          975     1115     +140     
- Misses        447      507      +60

Flag	Coverage Δ
integration	`55.11% <42.29%> (-3.33%)`	⬇️
unit	`57.02% <71.93%> (+2.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sdv-team · 2025-07-03T16:36:22Z

Task linked: CU-86b5ewpu4 SDGym - Add benchmark_single_table_aws function #414

amontanez24

It's ok that output_folder is in a different place in terms of order for this function than benchmark_single_table

sdgym/benchmark.py

amontanez24

I really like the changes! I left one small comment but I think this is great

sdgym/utils.py

amontanez24

One small comment. This looks great! Thank you for addressing the comments

sdgym/s3.py

amontanez24

Two small comments then

sdgym/benchmark.py

fealho

I left one final comment, it should be good to go after that

fealho · 2025-07-14T16:08:55Z

sdgym/s3.py

@@ -144,3 +150,52 @@ def write_csv(data, path, aws_key, aws_secret):
    """
    data_contents = data.to_csv(index=False).encode('utf-8')
    write_file(data_contents, path, aws_key, aws_secret)
+
+
+def _parse_s3_uri(s3_uri):


Isn't this method the same as parse_s3_path which already exists?

R-Palazzo changed the base branch from main to issue-410-save-synthesizers July 2, 2025 13:56

R-Palazzo changed the title ~~Issue 414 benchmark single table aws~~ Add benchmark_single_table_aws function Jul 3, 2025

R-Palazzo requested a review from amontanez24 July 3, 2025 16:39

R-Palazzo marked this pull request as ready for review July 3, 2025 16:43

R-Palazzo requested a review from a team as a code owner July 3, 2025 16:43

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from ccc4c05 to 647f17a Compare July 7, 2025 13:37

amontanez24 reviewed Jul 7, 2025

View reviewed changes

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from 647f17a to 8ebc6b7 Compare July 8, 2025 12:53

amontanez24 reviewed Jul 8, 2025

View reviewed changes

sdgym/utils.py Outdated Show resolved Hide resolved

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from d4e3d1d to 6f142aa Compare July 9, 2025 15:29

R-Palazzo requested review from amontanez24 and fealho July 9, 2025 16:18

amontanez24 reviewed Jul 9, 2025

View reviewed changes

sdgym/s3.py Show resolved Hide resolved

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from 6f142aa to 6364280 Compare July 10, 2025 14:35

amontanez24 reviewed Jul 10, 2025

View reviewed changes

sdgym/benchmark.py Outdated Show resolved Hide resolved

sdgym/benchmark.py Outdated Show resolved Hide resolved

fealho requested changes Jul 11, 2025

View reviewed changes

sdgym/benchmark.py Show resolved Hide resolved

sdgym/benchmark.py Outdated Show resolved Hide resolved

sdgym/benchmark.py Outdated Show resolved Hide resolved

sdgym/benchmark.py Show resolved Hide resolved

sdgym/benchmark.py Outdated Show resolved Hide resolved

R-Palazzo force-pushed the issue-410-save-synthesizers branch from eab6ecb to 662786d Compare July 14, 2025 07:27

Base automatically changed from issue-410-save-synthesizers to main July 14, 2025 07:53

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from caacb63 to 36b8dea Compare July 14, 2025 12:00

R-Palazzo requested review from fealho and amontanez24 July 14, 2025 15:28

amontanez24 approved these changes Jul 14, 2025

View reviewed changes

fealho reviewed Jul 14, 2025

View reviewed changes

R-Palazzo requested a review from fealho July 15, 2025 09:01

fealho approved these changes Jul 15, 2025

View reviewed changes

R-Palazzo added 3 commits July 15, 2025 18:47

def 414

6ba7eb0

pip install

7247d04

c1

1b77959

R-Palazzo added 27 commits July 15, 2025 18:47

update s3f dependency

d7a7413

move s3f dependency from pyproject to script

a5837ff

save results.csv

34ab993

fix tests

202ef6d

add unit test benchmark_single_table_aws

c207bb2

restructure

744348a

deprecate run_on_ec2

cb3a9e1

move method to s3.py + typo + default value

33e3032

fix saving result.csv

15a9cde

rebase

b16dc86

typo

62e53b3

fix tests

dc0bb50

make the increment work with s3

108fb49

fix _get_increment_run_id run_id

77d8645

fix unit test

f9d5893

address comments

a08921c

private attribute credentials

163d841

public method + region

5fc39dd

don't use private method

38d1066

remove uuid

2a4f569

update id when saving the jobs

c0797a8

remove extra '/'

7202d77

remove extra '/'

e303de5

install from released version

9801601

change _parse_s3_uri -> parse_s3_path

c491f41

point to release version

495052e

update LocalResultWriter

917d2e3

R-Palazzo force-pushed the issue-414-benchmark_single_table_aws branch from 7636acf to 917d2e3 Compare July 15, 2025 16:47

R-Palazzo merged commit f73dc02 into main Jul 17, 2025
55 checks passed

R-Palazzo deleted the issue-414-benchmark_single_table_aws branch July 17, 2025 07:41

Add benchmark_single_table_aws function #417

Add benchmark_single_table_aws function #417

Uh oh!

Conversation

R-Palazzo commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sdv-team commented Jul 3, 2025

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amontanez24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fealho left a comment

Choose a reason for hiding this comment

Uh oh!

fealho Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

R-Palazzo commented Jul 2, 2025 •

edited

Loading

codecov bot commented Jul 3, 2025 •

edited

Loading