Skip to content

Python: Add S3 Batch scenario with CloudFormation integration #7531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Python: Add S3 Batch scenario with CloudFormation integration #7531

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c20692e
Python: Add S3 Batch scenario
aBurmeseDev Jul 16, 2025
51f47be
chore(README): Add README for S3 Batch Python
aBurmeseDev Jul 16, 2025
697e304
test: Add unit tests for S3 Batch Scenario Python
aBurmeseDev Jul 16, 2025
7bc0ae7
Fix job activation and update tests
aBurmeseDev Jul 22, 2025
eef106f
chore(metadata): Add new metadata to s3-control-metadata
aBurmeseDev Jul 23, 2025
5d7447e
Add job status, priority updates, tagging
aBurmeseDev Aug 4, 2025
caabcaa
feat: add type hints and fix status message
aBurmeseDev Aug 5, 2025
2ed4d56
chore(metadata): add snippet tags and metadata for every operations
aBurmeseDev Aug 5, 2025
6a9de25
refactor: standardize user input handling with helper and update meta…
aBurmeseDev Aug 5, 2025
82a8560
refactor: restructure classes to follow Python examples standards
aBurmeseDev Aug 7, 2025
bdabee5
refactor: remove hard-coded region and use default config
aBurmeseDev Aug 9, 2025
01f94ac
refactor: use client injection pattern to follow Python examples stan…
aBurmeseDev Aug 13, 2025
f92369a
test: add service method stubbing pattern to follow Python Examples S…
aBurmeseDev Aug 13, 2025
3220e04
chore: fix snippet tags
aBurmeseDev Aug 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 102 additions & 11 deletions .doc_gen/metadata/s3-control_metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,15 @@ s3-control_Hello:
- description:
snippet_tags:
- s3control.java2.list_jobs.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.list_jobs
services:
s3-control: {ListJobs}

Expand All @@ -29,15 +38,24 @@ s3-control_CreateJob:
- s3control.java2.create_job.async.main
- description: Create a compliance retention job.
snippet_tags:
- s3control.java2.create_job.compliance.main
- s3control.java2.create_job.compliance.main
- description: Create a legal hold off job.
snippet_tags:
- s3control.java2.create_job.compliance.main
- s3control.java2.create_job.compliance.main
- description: Create a new governance retention job.
snippet_tags:
- s3.java2.create_governance_retemtion.main
- s3.java2.create_governance_retemtion.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.create_job
services:
s3-control: {CreateJob}
s3-control: {CreateJob}
s3-control_PutJobTagging:
languages:
Java:
Expand All @@ -49,8 +67,17 @@ s3-control_PutJobTagging:
- description:
snippet_tags:
- s3control.java2.job.put.tags.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.put_job_tagging
services:
s3-control: {PutJobTagging}
s3-control: {PutJobTagging}
s3-control_DescribeJob:
languages:
Java:
Expand All @@ -62,8 +89,17 @@ s3-control_DescribeJob:
- description:
snippet_tags:
- s3control.java2.describe_job.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.describe_job
services:
s3-control: {DescribeJob}
s3-control: {DescribeJob}
s3-control_DeleteJobTagging:
languages:
Java:
Expand All @@ -75,8 +111,17 @@ s3-control_DeleteJobTagging:
- description:
snippet_tags:
- s3control.java2.del_job_tagging.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.delete_job_tagging
services:
s3-control: {DeleteJobTagging}
s3-control: {DeleteJobTagging}
s3-control_GetJobTagging:
languages:
Java:
Expand All @@ -88,8 +133,17 @@ s3-control_GetJobTagging:
- description:
snippet_tags:
- s3control.java2.get_job_tagging.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.get_job_tagging
services:
s3-control: {GetJobTagging}
s3-control: {GetJobTagging}
s3-control_UpdateJobStatus:
languages:
Java:
Expand All @@ -101,8 +155,17 @@ s3-control_UpdateJobStatus:
- description:
snippet_tags:
- s3control.java2.cancel_job.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.update_job_status
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tag doesn't seem to exist in the code.

services:
s3-control: {UpdateJobStatus}
s3-control: {UpdateJobStatus}
s3-control_UpdateJobPriority:
languages:
Java:
Expand All @@ -114,8 +177,17 @@ s3-control_UpdateJobPriority:
- description:
snippet_tags:
- s3control.java2.update_job.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description:
snippet_tags:
- python.example_code.s3control.update_job_priority
services:
s3-control: {UpdateJobPriority}
s3-control: {UpdateJobPriority}
s3-control_Basics:
synopsis: learn core operations for &S3Control;.
category: Basics
Expand All @@ -132,5 +204,24 @@ s3-control_Basics:
- description: An action class that wraps operations.
snippet_tags:
- s3control.java2.job.actions.main
Python:
versions:
- sdk_version: 3
github: python/example_code/s3/scenarios/batch
sdkguide:
excerpts:
- description: Learn S3 Batch Basics Scenario.
snippet_tags:
- python.example_code.s3control.Batch.scenario
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tag doesn't seem to exist in the code.

services:
s3-control: {CreateJob, DeleteJobTagging, DescribeJob, GetJobTagging, ListJobs, PutJobTagging, UpdateJobPriority, UpdateJobStatus}
s3-control:
{
CreateJob,
DeleteJobTagging,
DescribeJob,
GetJobTagging,
ListJobs,
PutJobTagging,
UpdateJobPriority,
UpdateJobStatus,
}
56 changes: 56 additions & 0 deletions python/example_code/s3/scenarios/batch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Amazon S3 Batch for the SDK for Python (boto3)

## Overview

This example demonstrates how to use the AWS SDK for Python (boto3) to work with Amazon Simple Storage Service (Amazon S3) Batch Scenario. The scenario covers various operations such as creating an AWS Batch compute environment, creating a job queue, creating a job defination, and submitting a job, and so on.

Here are the top six service operations this scenario covers.

1. **Create an AWS Batch computer environment**: Creates an AWS Batch computer environment.

2. **Sets up a job queue**: Creates a job queue that will manage the submission of jobs.

3. **Creates a job definition**: Creates a job definition that specifies how the jobs should be executed.

4. **Registers a Job Definition**: Registers a job definition making it available for job submissions.

5. **Submits a Batch Job**: Submits a job.

6. **Checks the status of the job**: Checks the status of the job.

## ⚠ Important

- Running this code might result in charges to your AWS account. For more details, see [AWS Pricing](https://aws.amazon.com/pricing/) and [Free Tier](https://aws.amazon.com/free/).
- Running the tests might result in charges to your AWS account.
- We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
- This code is not tested in every AWS Region. For more information, see [AWS Regional Services](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services).

## Code examples

### Prerequisites

To run these examples, you need:

- Python 3.x installed.
- Run `python pip install -r requirements.txt`
- AWS credentials configured. For more information, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).

#### Running the workflow

To run this workflow, pull AWS tokens and run the command below:

```bash
python s3_batch.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be s3_batch_scenario.py now?

```

## Additional resources

- [Amazon S3 Developer Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is linking to an object-lock page, did you mean to link to something for S3 Batch?

- [Amazon S3 API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)
- [boto3 Amazon S3 reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html)

---

© Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: Apache-2.0
Loading
Loading