-
Notifications
You must be signed in to change notification settings - Fork 16
Automatic visualization publishing, package standardization and improved tooling #89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
dannon
wants to merge
5
commits into
main
Choose a base branch
from
publishing
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
f4cdba3
Add GitHub workflows for automatic package publishing
dannon 9a129ac
Drop Lerna
dannon 734836f
Enhance README with development setup and monorepo structure
dannon b3fd11b
Migrate from Yarn to pnpm workspace management
dannon b341a3a
Swap github workflows to pnpm
dannon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| name: Manual Package Publish | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| package: | ||
| description: 'Package to publish (e.g., heatmap, plotly)' | ||
| required: true | ||
| type: string | ||
| dry-run: | ||
| description: 'Dry run (do not actually publish)' | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
|
|
||
| jobs: | ||
| publish: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Validate package exists | ||
| run: | | ||
| if [ ! -d "packages/${{ github.event.inputs.package }}" ]; then | ||
| echo "Error: Package 'packages/${{ github.event.inputs.package }}' does not exist" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ ! -f "packages/${{ github.event.inputs.package }}/package.json" ]; then | ||
| echo "Error: Package 'packages/${{ github.event.inputs.package }}' has no package.json" | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '18' | ||
| registry-url: 'https://registry.npmjs.org' | ||
|
|
||
| - name: Setup pnpm | ||
| uses: pnpm/action-setup@v4 | ||
| with: | ||
| version: 10 | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| - name: Build package | ||
| working-directory: packages/${{ github.event.inputs.package }} | ||
| run: | | ||
| if grep -q '"build"' package.json; then | ||
| pnpm build | ||
| else | ||
| echo "No build script found, skipping build step" | ||
| fi | ||
|
|
||
| - name: Check package info | ||
| working-directory: packages/${{ github.event.inputs.package }} | ||
| run: | | ||
| package_name=$(jq -r '.name' package.json) | ||
| package_version=$(jq -r '.version' package.json) | ||
|
|
||
| echo "Package: $package_name" | ||
| echo "Version: $package_version" | ||
|
|
||
| # Check if already published | ||
| if npm view "$package_name@$package_version" version 2>/dev/null; then | ||
| echo "⚠️ Package $package_name@$package_version is already published" | ||
| echo "published=true" >> $GITHUB_ENV | ||
| else | ||
| echo "✅ Package $package_name@$package_version is not yet published" | ||
| echo "published=false" >> $GITHUB_ENV | ||
| fi | ||
|
|
||
| - name: Dry run | ||
| if: github.event.inputs.dry-run == 'true' | ||
| working-directory: packages/${{ github.event.inputs.package }} | ||
| run: | | ||
| echo "🧪 DRY RUN MODE - Would publish:" | ||
| npm pack --dry-run | ||
|
|
||
| - name: Publish to npm | ||
| if: github.event.inputs.dry-run == 'false' && env.published == 'false' | ||
| working-directory: packages/${{ github.event.inputs.package }} | ||
| env: | ||
| NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} | ||
| run: | | ||
| package_name=$(jq -r '.name' package.json) | ||
| package_version=$(jq -r '.version' package.json) | ||
|
|
||
| echo "🚀 Publishing $package_name@$package_version" | ||
| npm publish --access public | ||
|
|
||
| - name: Skip publishing | ||
| if: github.event.inputs.dry-run == 'false' && env.published == 'true' | ||
| run: | | ||
| echo "⏭️ Skipping publish - package already exists at this version" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| name: Publish Updated Packages | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| detect-changes: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| changed-packages: ${{ steps.detect-changes.outputs.packages }} | ||
| has-changes: ${{ steps.detect-changes.outputs.has-changes }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 2 | ||
|
|
||
| - name: Detect package version changes | ||
| id: detect-changes | ||
| run: | | ||
| # Get the list of changed package.json files | ||
| changed_files=$(git diff --name-only HEAD~1 HEAD | grep "packages/.*/package.json" || true) | ||
|
|
||
| if [ -z "$changed_files" ]; then | ||
| echo "has-changes=false" >> $GITHUB_OUTPUT | ||
| echo "No package.json files changed" | ||
| exit 0 | ||
| fi | ||
|
|
||
| changed_packages="" | ||
|
|
||
| for file in $changed_files; do | ||
| package_dir=$(dirname "$file") | ||
| package_name=$(basename "$package_dir") | ||
|
|
||
| # Check if version actually changed | ||
| old_version=$(git show HEAD~1:"$file" | jq -r '.version' 2>/dev/null || echo "") | ||
| new_version=$(cat "$file" | jq -r '.version' 2>/dev/null || echo "") | ||
|
|
||
| if [ "$old_version" != "$new_version" ] && [ -n "$new_version" ] && [ "$new_version" != "null" ]; then | ||
| echo "Package $package_name version changed: $old_version -> $new_version" | ||
| if [ -n "$changed_packages" ]; then | ||
| changed_packages="$changed_packages,$package_name" | ||
| else | ||
| changed_packages="$package_name" | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| if [ -n "$changed_packages" ]; then | ||
| echo "has-changes=true" >> $GITHUB_OUTPUT | ||
| echo "packages=$changed_packages" >> $GITHUB_OUTPUT | ||
| echo "Changed packages: $changed_packages" | ||
| else | ||
| echo "has-changes=false" >> $GITHUB_OUTPUT | ||
| echo "No version changes detected" | ||
| fi | ||
|
|
||
| publish: | ||
| needs: detect-changes | ||
| if: needs.detect-changes.outputs.has-changes == 'true' | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| package: ${{ fromJson(format('["{0}"]', join(split(needs.detect-changes.outputs.changed-packages, ','), '","'))) }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '18' | ||
| registry-url: 'https://registry.npmjs.org' | ||
|
|
||
| - name: Setup pnpm | ||
| uses: pnpm/action-setup@v4 | ||
| with: | ||
| version: 10 | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| - name: Build package | ||
| working-directory: packages/${{ matrix.package }} | ||
| run: | | ||
| # Try different build commands based on what's available | ||
| if grep -q '"build"' package.json; then | ||
| pnpm build | ||
| else | ||
| echo "No build script found, skipping build step" | ||
| fi | ||
|
|
||
| - name: Publish to npm | ||
| working-directory: packages/${{ matrix.package }} | ||
| env: | ||
| NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} | ||
| run: | | ||
| # Check if package is already published at this version | ||
| package_name=$(jq -r '.name' package.json) | ||
| package_version=$(jq -r '.version' package.json) | ||
|
|
||
| if npm view "$package_name@$package_version" version 2>/dev/null; then | ||
| echo "Package $package_name@$package_version already published, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Publish the package | ||
| echo "Publishing $package_name@$package_version" | ||
| npm publish --access public |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| name: Test Packages | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: [main] | ||
| push: | ||
| branches: [main] | ||
|
|
||
| jobs: | ||
| detect-changes: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| changed-packages: ${{ steps.detect-changes.outputs.packages }} | ||
| has-changes: ${{ steps.detect-changes.outputs.has-changes }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Detect changed packages | ||
| id: detect-changes | ||
| run: | | ||
| # For PRs, compare against the base branch | ||
| if [ "${{ github.event_name }}" = "pull_request" ]; then | ||
| base_ref="origin/${{ github.event.pull_request.base.ref }}" | ||
| else | ||
| # For push to main, compare with previous commit | ||
| base_ref="HEAD~1" | ||
| fi | ||
|
|
||
| # Get changed files in packages directory | ||
| changed_files=$(git diff --name-only "$base_ref" HEAD -- packages/ | grep -E "packages/[^/]+/" | cut -d'/' -f1-2 | sort -u || true) | ||
|
|
||
| if [ -z "$changed_files" ]; then | ||
| echo "has-changes=false" >> $GITHUB_OUTPUT | ||
| echo "No packages changed" | ||
| exit 0 | ||
| fi | ||
|
|
||
| changed_packages="" | ||
| for package_dir in $changed_files; do | ||
| package_name=$(basename "$package_dir") | ||
| if [ -f "$package_dir/package.json" ]; then | ||
| if [ -n "$changed_packages" ]; then | ||
| changed_packages="$changed_packages,$package_name" | ||
| else | ||
| changed_packages="$package_name" | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| if [ -n "$changed_packages" ]; then | ||
| echo "has-changes=true" >> $GITHUB_OUTPUT | ||
| echo "packages=$changed_packages" >> $GITHUB_OUTPUT | ||
| echo "Changed packages: $changed_packages" | ||
| else | ||
| echo "has-changes=false" >> $GITHUB_OUTPUT | ||
| echo "No packages with changes found" | ||
| fi | ||
|
|
||
| test: | ||
| needs: detect-changes | ||
| if: needs.detect-changes.outputs.has-changes == 'true' | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| package: ${{ fromJson(format('["{0}"]', join(split(needs.detect-changes.outputs.changed-packages, ','), '","'))) }} | ||
| fail-fast: false | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '18' | ||
|
|
||
| - name: Setup pnpm | ||
| uses: pnpm/action-setup@v4 | ||
| with: | ||
| version: 10 | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| - name: Run tests | ||
| working-directory: packages/${{ matrix.package }} | ||
| run: | | ||
| if grep -q '"test"' package.json && ! grep -q '"test":\s*""' package.json; then | ||
| echo "Running tests for ${{ matrix.package }}" | ||
| pnpm test | ||
| else | ||
| echo "No tests found for ${{ matrix.package }}, skipping" | ||
| fi | ||
|
|
||
| - name: Build package | ||
| working-directory: packages/${{ matrix.package }} | ||
| run: | | ||
| if grep -q '"build"' package.json; then | ||
| echo "Building ${{ matrix.package }}" | ||
| pnpm build | ||
| else | ||
| echo "No build script found for ${{ matrix.package }}, skipping" | ||
| fi | ||
|
|
||
| - name: Check formatting | ||
| working-directory: packages/${{ matrix.package }} | ||
| run: | | ||
| if grep -q '"prettier"' package.json; then | ||
| echo "Checking formatting for ${{ matrix.package }}" | ||
| pnpm prettier --check . | ||
| else | ||
| echo "No prettier script found for ${{ matrix.package }}, skipping" | ||
| fi |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need version 18 here due to some limitation, or can we use a more recent one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'd definitely prefer to bump this to a more recent version to match what Galaxy uses. I picked 18 conservatively for now since I haven't had a chance to fully test the publishing workflow without doing actual npm publishes.
Currently, I've excluded packages that failed during pnpm build:all to get the initial automation working, but the goal is to get all 40 packages building and publishing successfully. Once I figure out a good testing strategy (maybe using a local npm registry or dry-run publishes), I'll update the Node version and bring the remaining packages into the workflow.