Debug ff-matrix gha workflow #12989

cderv · 2025-06-26T16:02:46Z

Update to fix issues
Update to new version of its initial inspirations smoke-tests workflow

posit-snyk-bot · 2025-06-26T16:04:32Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

gordonwoodhull · 2025-06-26T20:14:36Z

Thanks @cderv! With this fix I do see mostly the same failures on the Windows run in CI as I saw locally.

25 mins is not bad, but we probably don't want to add these to the smoke tests.

The timeouts on Linux are troubling.... IIRC we had similar trouble before.

cderv · 2025-06-27T09:19:30Z

The timeouts on Linux are troubling.... IIRC we had similar trouble before.

Yes I recall now this was the main problem. Something related to anything with mermaid and required chrome printing by quarto. I'll check the setup

cderv · 2025-06-27T11:50:30Z

This is interesting because running a single doc that fails previously works

https://github.com/quarto-dev/quarto-cli/actions/runs/15924034685/job/44917132138?pr=12989#step:25:1002

So this means the hanging has to do with multiple calls one after the other somehow

cderv · 2025-06-27T13:05:38Z

So running all crossref test fails:

./run-tests.sh ../dev-docs/feature-format-matrix/qmd-files/crossref/**/*.qmd

And running all floats too

./run-tests.sh ../dev-docs/feature-format-matrix/qmd-files/crossref/float/**/*.qmd

However, running two tests with Mermaid only works

./run-tests.sh ../dev-docs/feature-format-matrix/qmd-files/crossref/float/custom/document.qmd ../dev-docs/feature-format-matrix/qmd-files/crossref/float/figure/document.qmd

So still haven't found the pattern that creates the issue

cderv · 2025-06-27T13:24:18Z

And strangely this works

./run-tests.sh \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/caption-location-bottom/document.qmd \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/caption-location-top/document.qmd \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/code-listing/document.qmd \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/code-listing/executable/document.qmd \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/custom/document.qmd \
../dev-docs/feature-format-matrix/qmd-files/crossref/float/figure/document.qmd

cderv · 2025-06-27T15:44:36Z

So adding the table feature file makes the thing fail... Let's investigate on that.

cderv · 2025-06-27T15:52:37Z

So it seems ../dev-docs/feature-format-matrix/qmd-files/crossref/float/table/document.qmd is failing. But I don't see how it could create the hang... 🤔

Let's see if this is the problem.

cderv · 2025-06-27T16:01:03Z

It is passing ! Let's try full tests.

But there is something about this test failing that makes the all process hang. Still don't know what

cderv · 2025-06-27T16:25:34Z

No that is not the case... Not related to this specific test file. I am now thinking there is a flow somewhere in how testing works ... Won't be easy to debug 😭

cderv · 2025-06-27T17:06:24Z

So I can reproduce by running interactively on the CI runners directly. This is really puzzling.

This is only something when running inside CI runners, non interactively. Like if the stream does not get back information anymore (and not really quarto hanging).

I ran all feature-matrix test locally and got to the end with the failures. Here are the 9 failing test

[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/crossref/float/table/document.qmd --to revealjs => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/crossref/float/table/document.qmd --to typst => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/code-python-label.qmd --to typst => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/code-python-nocaption.qmd --to typst => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/code-r-nocaption.qmd --to typst => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/toplevel-id-caption.qmd --to latex => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/toplevel-id-nocaption.qmd --to latex => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/layout/fig-align/toplevel-noid-nocaption.qmd --to typst => ./test.ts:346:8
[smoke] > quarto render ../dev-docs/feature-format-matrix/qmd-files/raw-blocks/interpreted/pandoc-native/document.qmd --to typst => ./test.ts:346:8

FAILED | 259 passed | 9 failed (5m32s)

gordonwoodhull · 2025-06-27T18:16:16Z

Thanks @cderv for this thorough investigation! This is really weird.

It seems like maybe we should just remove mermaid from some tests until it's working. I don't think it's essential here.

I think I know what the six Typst failures are; Pandoc output has changed for figures. The reveal and latex failures are probably churn as well.

For now still a copy of the smoke test workflow, but we will refactor this to use the smoke test workflow as a base and then add the additional steps needed for the feature format matrix tests.

except table one which is currently hidden

This reverts commit 8bea247.

```` _quarto: tests-on-ci: false ```` `smoke-all.test.ts` should skip tests when this metadata is set to false inside a CI context This can be useful for mermaid test for example

Now internal slide links works

Since #12285

This reverts commit 4a20493.

There is now a `\pandocbounded` macro used

Since #11010 as we insert the class https://github.com/quarto-dev/quarto-cli/blob/d761a8b30cc3e939daaa70724b87c479b0e9f498/src/resources/filters/quarto-post/typst.lua#L127-L131

[skip-ci]

revealjs link can work to id anchor inside slide now follow up on #12989 fixes

cderv force-pushed the gha/ff-matrix branch from 0886392 to a6ac922 Compare June 27, 2025 09:20

cderv changed the title ~~Update ff-matrix gha workflow~~ Debug ff-matrix gha workflow Jun 27, 2025

cderv marked this pull request as draft June 27, 2025 17:57

cderv added 15 commits July 8, 2025 16:23

Use node 20

152455a

No need for playwright

8a14e48

Update ff-matrix workflow based on smoke test one

a988128

For now still a copy of the smoke test workflow, but we will refactor this to use the smoke test workflow as a base and then add the additional steps needed for the feature format matrix tests.

Opt-in on pull requests to run on the current PR

0e92024

Add to ff-matrix workflow too

b04caa3

Is installing node dependencies helping ?

e05af1c

fixup - only run linux to try fix hanging issue

52b76da

fixup - only run linux to try fix hanging issue

dd32e13

Rename step as this is not smoke test but feature-format-matrix

cae3d5c

debug - let's try all crossref tests

63dc851

debug - try floats one only

cfbf2a8

debug - try two in a row

c81fa4e

debug - try all files in floats

f23e2a6

debug - I was missing table file

85b0e6a

debug - only run table document

9b97b07

cderv added 7 commits July 8, 2025 16:23

hide table test and try re-running all floats tests

75381c2

reactivate all tests

883d784

except table one which is currently hidden

Revert "reactivate all tests"

4a20493

This reverts commit 8bea247.

debug - Restore case fail with all floats

95fd58a

Run tmate for live debug

52a2a11

debug remove duplicated call

f87df74

Add a way to skip test on CI based on metadata

aadfee9

```` _quarto: tests-on-ci: false ```` `smoke-all.test.ts` should skip tests when this metadata is set to false inside a CI context This can be useful for mermaid test for example

cderv force-pushed the gha/ff-matrix branch from 2df0062 to aadfee9 Compare July 8, 2025 14:37

cderv added 11 commits July 8, 2025 20:25

Fix revealjs link

81219d1

Now internal slide links works

Typst ref now always have #ref( and supplement

7a7f9d6

Since #12285

remove tmate debugging

a24929e

Reapply "reactivate all tests"

e81299b

This reverts commit 4a20493.

Reactivate Windows

48e7bb7

in typst #align does not have new line anymore inserted before box

7235121

Account for Pandoc 3.2.1 new method for ensuring images don’t overflow

930573f

There is now a `\pandocbounded` macro used

fixup - align

a8e80ef

typo in link

d761a8b

No more figure element with Typst on Table

4f119b0

Since #11010 as we insert the class https://github.com/quarto-dev/quarto-cli/blob/d761a8b30cc3e939daaa70724b87c479b0e9f498/src/resources/filters/quarto-post/typst.lua#L127-L131

use test-smokes.yml for ff-matrix

ed0fd5f

cderv force-pushed the gha/ff-matrix branch from 2329342 to ed0fd5f Compare July 9, 2025 13:30

cderv marked this pull request as ready for review July 9, 2025 13:31

unused import

9391b16

[skip-ci]

cderv merged commit fad85eb into main Jul 9, 2025
5 of 53 checks passed

cderv deleted the gha/ff-matrix branch July 9, 2025 14:00

cderv added a commit that referenced this pull request Jul 10, 2025

ff-matrix - remove revealjs comment as outdated

6e36833

revealjs link can work to id anchor inside slide now follow up on #12989 fixes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Debug ff-matrix gha workflow #12989

Debug ff-matrix gha workflow #12989

Uh oh!

cderv commented Jun 26, 2025

Uh oh!

posit-snyk-bot commented Jun 26, 2025 •

edited

Loading

Uh oh!

gordonwoodhull commented Jun 26, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025 •

edited

Loading

Uh oh!

gordonwoodhull commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

Debug ff-matrix gha workflow #12989

Debug ff-matrix gha workflow #12989

Uh oh!

Conversation

cderv commented Jun 26, 2025

Uh oh!

posit-snyk-bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

gordonwoodhull commented Jun 26, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025

Uh oh!

cderv commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gordonwoodhull commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

posit-snyk-bot commented Jun 26, 2025 •

edited

Loading

cderv commented Jun 27, 2025 •

edited

Loading