Skip to content

Commit 046feb0

Browse files
moomindanimoomindani
authored
Optimize GitHub actions workflows (#588)
* Optimize GitHub Actions workflows to prevent timeout issues ## Problem - Integration tests were timing out after 60 minutes due to long-running Python model tests - All tests were running in a single job, making debugging difficult ## Solution Split integration tests into two separate workflows: ### 1. Integration Tests (integration.yml) - Runs all functional tests EXCEPT Python model tests - Uses existing tox integration environment with --ignore=python_model/ - Maintains 60-minute timeout - Triggers on 'enable-functional-tests' label ### 2. Python Model Tests (python-models.yml) - Dedicated workflow for Python model tests only - Uses new tox python-models environment - Shorter 45-minute timeout (focused scope) - Separate S3 locations to avoid resource conflicts - Independent artifact collection ### 3. Tox Environment Updates (tox.ini) - integration: Excludes Python model tests - python-models: Only Python model tests - Maintains existing unit environment ## Benefits - ✅ Prevents 60-minute timeout issues - ✅ Parallel execution reduces overall CI time - ✅ Better isolation for debugging failures - ✅ Separate artifacts for each test type - ✅ Independent failure handling ## Testing After merge, PRs with 'enable-functional-tests' label will show: - 'Integration Tests' workflow (main functional tests) - 'Python Model Tests' workflow (Python model tests) Both workflows run in parallel with appropriate timeouts. * Clean up integration.yml by removing redundant jobs - Remove test-multiple-jobs (was only for debugging) - Remove python-models-pr job (now in dedicated python-models.yml) - Remove python-models-main job (now in dedicated python-models.yml) integration.yml now contains only: - functional-tests-pr: PR integration tests (excluding Python models) - functional-tests-main: Main branch integration tests (excluding Python models) python-models.yml contains: - python-model-tests-pr: Dedicated Python model tests for PRs This provides clean separation of concerns with no duplication.
1 parent 304b5b2 commit 046feb0

File tree

3 files changed

+108
-8
lines changed

3 files changed

+108
-8
lines changed

‎.github/workflows/integration.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535

3636
env:
3737
TOXENV: "integration"
38-
PYTEST_ADDOPTS: "-v --color=yes --csv integ_results.csv"
38+
PYTEST_ADDOPTS: "-v --color=yes --csv integ_results.csv --ignore=tests/functional/adapter/python_model/"
3939
DBT_AWS_ACCOUNT: ${{ secrets.DBT_AWS_ACCOUNT }}
4040
DBT_GLUE_ROLE_ARN: ${{ secrets.DBT_GLUE_ROLE_ARN }}
4141
DBT_GLUE_REGION: ${{ secrets.DBT_GLUE_REGION }}
@@ -87,8 +87,8 @@ jobs:
8787
- uses: actions/upload-artifact@v4
8888
if: always()
8989
with:
90-
name: unit_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
91-
path: unit_results.csv
90+
name: integ_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
91+
path: integ_results.csv
9292
overwrite: true
9393

9494
# workflow that is invoked when a push to main happens
@@ -106,7 +106,7 @@ jobs:
106106

107107
env:
108108
TOXENV: "integration"
109-
PYTEST_ADDOPTS: "-v --color=yes --csv integ_results.csv -s"
109+
PYTEST_ADDOPTS: "-v --color=yes --csv integ_results.csv -s --ignore=tests/functional/adapter/python_model/"
110110
DBT_AWS_ACCOUNT: ${{ secrets.DBT_AWS_ACCOUNT }}
111111
DBT_GLUE_ROLE_ARN: ${{ secrets.DBT_GLUE_ROLE_ARN }}
112112
DBT_GLUE_REGION: ${{ secrets.DBT_GLUE_REGION }}
@@ -156,6 +156,6 @@ jobs:
156156
- uses: actions/upload-artifact@v4
157157
if: always()
158158
with:
159-
name: unit_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
160-
path: unit_results.csv
159+
name: integ_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
160+
path: integ_results.csv
161161
overwrite: true

‎.github/workflows/python-models.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Python Model Tests
2+
3+
on:
4+
pull_request_target:
5+
types: [labeled]
6+
7+
permissions:
8+
id-token: write
9+
contents: read
10+
11+
concurrency:
12+
group: ${{ github.workflow }}-${{ github.event_name }}-${{ contains(github.event_name, 'pull_request') && github.event.pull_request.head.ref || github.sha }}
13+
cancel-in-progress: true
14+
15+
defaults:
16+
run:
17+
shell: bash
18+
19+
jobs:
20+
python-model-tests-pr:
21+
name: Python Model Tests - PR / python ${{ matrix.python-version }}
22+
if: contains(github.event.pull_request.labels.*.name, 'enable-functional-tests')
23+
24+
runs-on: ubuntu-latest
25+
timeout-minutes: 45
26+
27+
strategy:
28+
fail-fast: false
29+
matrix:
30+
python-version: ["3.13"]
31+
32+
env:
33+
TOXENV: "python-models"
34+
PYTEST_ADDOPTS: "-v --color=yes --csv python_model_results.csv -s"
35+
DBT_AWS_ACCOUNT: ${{ secrets.DBT_AWS_ACCOUNT }}
36+
DBT_GLUE_ROLE_ARN: ${{ secrets.DBT_GLUE_ROLE_ARN }}
37+
DBT_GLUE_REGION: ${{ secrets.DBT_GLUE_REGION }}
38+
39+
steps:
40+
- name: Check out the repository
41+
uses: actions/checkout@v3
42+
with:
43+
ref: ${{ github.event.pull_request.head.sha }}
44+
45+
- name: Set up Python ${{ matrix.python-version }}
46+
uses: actions/setup-python@v4
47+
with:
48+
python-version: ${{ matrix.python-version }}
49+
50+
- name: Install python dependencies
51+
run: |
52+
sudo apt-get update
53+
sudo apt-get install libsasl2-dev
54+
python -m pip install --user --upgrade pip
55+
python -m pip --version
56+
python -m pip install tox
57+
tox --version
58+
59+
- name: Generate session name
60+
id: session
61+
run: |
62+
repo="${GITHUB_REPOSITORY#${GITHUB_REPOSITORY_OWNER}/}"
63+
echo "name=${repo}-python-models-pr-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "${GITHUB_OUTPUT}"
64+
65+
- name: Configure AWS Credentials
66+
uses: aws-actions/configure-aws-credentials@v4
67+
with:
68+
role-session-name: ${{ steps.session.outputs.name }}
69+
role-to-assume: arn:aws:iam::${{ secrets.DBT_AWS_ACCOUNT }}:role/dbt-glue
70+
aws-region: ${{ secrets.DBT_GLUE_REGION }}
71+
mask-aws-account-id: true
72+
73+
- name: Run Python model tests
74+
run: |
75+
export DBT_S3_LOCATION=${{ secrets.DBT_S3_LOCATION }}/python-models-pr-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/${{ matrix.python-version }}
76+
tox
77+
78+
- name: Get current date
79+
if: always()
80+
id: date
81+
run: echo "date=$(date +'%Y-%m-%dT%H_%M_%S')" >> $GITHUB_OUTPUT
82+
83+
- uses: actions/upload-artifact@v4
84+
if: always()
85+
with:
86+
name: python_model_results_pr_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
87+
path: python_model_results.csv
88+
overwrite: true

‎tox.ini‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tox]
22
skipsdist = True
3-
envlist = unit, integration
3+
envlist = unit, integration, python-models
44

55
[testenv:{unit}]
66
allowlist_externals =
@@ -16,7 +16,19 @@ deps =
1616
[testenv:{integration}]
1717
allowlist_externals =
1818
/bin/bash
19-
commands = /bin/bash -c '{envpython} -m pytest -v {posargs} -s tests/functional/adapter/'
19+
commands = /bin/bash -c '{envpython} -m pytest -v {posargs} -s tests/functional/adapter/ --ignore=tests/functional/adapter/python_model/'
20+
passenv =
21+
DBT_*
22+
PYTEST_ADDOPTS
23+
AWS_*
24+
deps =
25+
-rdev-requirements.txt
26+
-e.
27+
28+
[testenv:{python-models}]
29+
allowlist_externals =
30+
/bin/bash
31+
commands = /bin/bash -c '{envpython} -m pytest -v {posargs} -s tests/functional/adapter/python_model/'
2032
passenv =
2133
DBT_*
2234
PYTEST_ADDOPTS

0 commit comments

Comments
 (0)