Forward envvars from JSON payload #306

MichaelHatherly · 2025-05-22T15:39:43Z

This linked PR is changing the way that several QUARTO_* environment variables are forwarded to the engine. Instead of just being available via ENV they are passed in a .env field in the options JSON that Quarto passes to QNR. Pick out the env vars if they exist and add them to the notebook process's env vars.

Cc @cscheid, this will provide the fix needed for quarto-dev/quarto-cli#12621.

src/server.jl

MichaelHatherly · 2025-05-22T15:47:12Z

@cscheid if you're wanting to run this in CI in quarto-dev/quarto-cli#12621 then temporarily committing a Manifest.toml to https://github.com/quarto-dev/quarto-cli/tree/main/src/resources/julia that points to this branch should do the trick. (Not sure how familiar you are with that part of Julia so let me know if unsure.)

codecov · 2025-05-22T16:30:01Z

Codecov Report

Attention: Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.42%. Comparing base (58e5dc8) to head (fb14ffa).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/server.jl	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #306      +/-   ##
==========================================
+ Coverage   79.40%   79.42%   +0.01%     
==========================================
  Files          39       39              
  Lines        2064     2066       +2     
==========================================
+ Hits         1639     1641       +2     
  Misses        425      425

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cscheid · 2025-05-22T17:50:35Z

@cscheid if you're wanting to run this in CI in quarto-dev/quarto-cli#12621 then temporarily committing a Manifest.toml to https://github.com/quarto-dev/quarto-cli/tree/main/src/resources/julia that points to this branch should do the trick. (Not sure how familiar you are with that part of Julia so let me know if unsure.)

Thanks, I will take you on your offer. I'm trying to do this locally and failing. I've confirmed that the branch I'm working on fails without this version of QuartoNotebookRunner:

 quarto preview julia.qmd
Starting julia control server process. This might take a while...
Julia server process started.
Running [1/1] at line 12:  ENV["QUARTO_DOCUMENT_FILE"]
ERROR: Julia server returned error after receiving "run" command:

Failed to run notebook: /Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd

The underlying Julia error was:

EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd:12
KeyError: key "QUARTO_DOCUMENT_FILE" not found
Stacktrace:
 [1] (::Base.var"#693#694")(k::String)
   @ Base ./env.jl:100
 [2] access_env
   @ ./env.jl:43 [inlined]
 [3] getindex(#unused#::Base.EnvDict, k::String)
   @ Base ./env.jl:100
 [4] top-level scope
   @ ~/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd:13

However, adding the following Manifest.toml file to the .../engine/env directory doesn't appear to do anything:

[[QuartoNotebookRunner]]
git-tree-sha1 = "57f9bd985428b9112fa258a7d889eb8cc90db19d"
uuid = "4c0109c6-14e9-4c88-93f0-2b974d3468f4"
version = "0.17.3"
repo-rev = "mh/env"
repo-url = "https://github.com/PumasAI/QuartoNotebookRunner.jl"

It's almost certain that I'm doing something wrong, but I'm not sure what that would be.

MichaelHatherly · 2025-05-22T19:07:22Z

However, adding the following Manifest.toml file to the .../engine/env directory doesn't appear to do anything:

engine/env/julia.qmd:13

Looks like you're installing QNR into the notebook's environment rather than the server environment.

Needs to be installed into this one: https://github.com/quarto-dev/quarto-cli/tree/main/src/resources/julia

cscheid · 2025-05-22T20:04:42Z

Ok, I think I got it to update, but I don't think the environment is being updated through correctly:

$ git log
commit 6028e0b354b7a15e908940a4c6819aad8e19e7b1 (HEAD -> bugfix/no-env-set-past-startup, origin/bugfix/no-env-set-past-startup)
...
$ quarto render julia.qmd
Starting julia control server process. This might take a while...
    Updating registry at `~/.julia/registries/General.toml`
   Installed QuartoNotebookRunner ─ v0.17.3
    Updating `~/Library/Caches/quarto/julia/Project.toml`
  [4c0109c6] ↑ QuartoNotebookRunner v0.17.0 ⇒ v0.17.3
    Updating `~/Library/Caches/quarto/julia/Manifest.toml`
  [4c0109c6] ↑ QuartoNotebookRunner v0.17.0 ⇒ v0.17.3
Precompiling project...
  1 dependency successfully precompiled in 14 seconds. 20 already precompiled.
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 6 found
      Active artifact files: 158 found
      Active scratchspaces: 4 found
     Deleted 2 package installations (639.244 KiB)
     Deleted 3 artifact installations (8.487 MiB)
Julia server process started.
Running [1/1] at line 12:  ENV["QUARTO_DOCUMENT_FILE"]
ERROR: Julia server returned error after receiving "run" command:

Failed to run notebook: /Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd

The underlying Julia error was:

EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd:12
KeyError: key "QUARTO_DOCUMENT_FILE" not found
Stacktrace:
 [1] (::Base.var"#693#694")(k::String)
   @ Base ./env.jl:100
 [2] access_env
   @ ./env.jl:43 [inlined]
 [3] getindex(#unused#::Base.EnvDict, k::String)
   @ Base ./env.jl:100
 [4] top-level scope
   @ ~/repos/github/quarto-dev/quarto-cli/tests/docs/smoke-all/engine/env/julia.qmd:13

MichaelHatherly · 2025-05-22T20:21:38Z

Does ~/Library/Caches/quarto/Julia/Manifest.toml reference this branch, or just 0.17.3?

cscheid · 2025-05-22T20:49:48Z

[[deps.QuartoNotebookRunner]]
deps = ["BSON", "Base64", "CommonMark", "Compat", "Dates", "IOCapture", "InteractiveUtils", "IterTools", "JSON3", "Logging", "Pkg", "PrecompileTools", "Preferences", "ProgressLogging", "REPL", "Random", "RelocatableFolders", "SHA", "Sockets", "TOML", "YAML"]
git-tree-sha1 = "94a4a9ef9ee5b22b5ea01215f749114f5cf9e220"
uuid = "4c0109c6-14e9-4c88-93f0-2b974d3468f4"
version = "0.17.3"

This commit is 57f9... so I think the answer is "just 0.17.3"

jkrumbiegel · 2025-05-22T20:57:31Z

Isn't the quarto CI running a sufficiently new Julia version that a [sources] section in the Project.toml would suffice, instead of committing a Manifest?
Like this (untested)

[sources]
QuartoNotebookRunner = {url = "https://github.com/PumasAI/QuartoNotebookRunner.jl/", rev = "mh/env"}

MichaelHatherly · 2025-05-22T20:58:49Z

Locally you'd do something like:

julia --project=~/Library/Caches/quarto/Julia

julia> ]add QuartoNotebookRunner#mh/env

to change it.

MichaelHatherly · 2025-05-22T20:59:22Z

Yeah, they're on 1.11 then that'll work as well

MichaelHatherly · 2025-06-02T16:25:38Z

@cscheid did you get a chance to try this out to see whether it was doing what had wanted it to do?

cscheid · 2025-06-11T12:48:27Z

@MichaelHatherly I'm sorry for the delay. I was out of commission for the last 10 days and I'm getting back today. I'll report back here as soon as I have updates (hopefully today even.)

cscheid · 2025-06-16T14:14:07Z

Hi! This worked 🎉

$ julia --project=~/Library/Caches/quarto/Julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(julia) pkg> add QuartoNotebookRunner#mh/env
     Cloning git-repo `https://github.com/PumasAI/QuartoNotebookRunner.jl.git`
    Updating git-repo `https://github.com/PumasAI/QuartoNotebookRunner.jl.git`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/Library/Caches/quarto/julia/Project.toml`
  [4c0109c6] ~ QuartoNotebookRunner v0.17.3 ⇒ v0.17.3 `https://github.com/PumasAI/QuartoNotebookRunner.jl.git#mh/env`
    Updating `~/Library/Caches/quarto/julia/Manifest.toml`
  [4c0109c6] ~ QuartoNotebookRunner v0.17.3 ⇒ v0.17.3 `https://github.com/PumasAI/QuartoNotebookRunner.jl.git#mh/env`
Precompiling project...
  1 dependency successfully precompiled in 14 seconds. 20 already precompiled.

(julia) pkg>
$ quarto render julia.qmd
Check file:///Users/cscheid/repos/github/quarto-dev/quarto-cli/src/quarto.ts
Starting julia control server process. This might take a while...
    Updating registry at `~/.julia/registries/General.toml`
    Updating git-repo `https://github.com/PumasAI/QuartoNotebookRunner.jl.git`
   Installed JSON3 ─ v1.14.3
   Installed YAML ── v0.4.14
  No Changes to `~/Library/Caches/quarto/julia/Project.toml`
    Updating `~/Library/Caches/quarto/julia/Manifest.toml`
  [0f8b85d8] ↑ JSON3 v1.14.2 ⇒ v1.14.3
  [ddb6d928] ↑ YAML v0.4.13 ⇒ v0.4.14
Precompiling project...
  3 dependencies successfully precompiled in 21 seconds. 18 already precompiled.
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 6 found
      Active artifact files: 158 found
      Active scratchspaces: 4 found
     Deleted 1 package installation (483.699 KiB)
     Deleted 9 artifact installations (130.005 MiB)
     Deleted 2 scratchspaces (111.047 KiB)
Julia server process started.
Running [1/1] at line 6:  print("hi")
pandoc
  to: html
  output-file: julia.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  variables: {}

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en

Output created: julia.html

MichaelHatherly commented May 22, 2025

View reviewed changes

src/server.jl Show resolved Hide resolved

MichaelHatherly commented May 22, 2025

View reviewed changes

src/server.jl Show resolved Hide resolved

Forward envvars from JSON payload

fb14ffa

MichaelHatherly force-pushed the mh/env branch from 57f9bd9 to fb14ffa Compare June 20, 2025 14:15

MichaelHatherly marked this pull request as ready for review June 20, 2025 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Forward envvars from JSON payload #306

Forward envvars from JSON payload #306

Uh oh!

MichaelHatherly commented May 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

MichaelHatherly commented May 22, 2025 •

edited

Loading

Uh oh!

codecov bot commented May 22, 2025 •

edited

Loading

Uh oh!

cscheid commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

cscheid commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

cscheid commented May 22, 2025 •

edited

Loading

Uh oh!

jkrumbiegel commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

MichaelHatherly commented Jun 2, 2025

Uh oh!

cscheid commented Jun 11, 2025

Uh oh!

cscheid commented Jun 16, 2025

Uh oh!

Uh oh!

Forward envvars from JSON payload #306

Are you sure you want to change the base?

Forward envvars from JSON payload #306

Uh oh!

Conversation

MichaelHatherly commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelHatherly commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cscheid commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

cscheid commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

cscheid commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkrumbiegel commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

MichaelHatherly commented May 22, 2025

Uh oh!

MichaelHatherly commented Jun 2, 2025

Uh oh!

cscheid commented Jun 11, 2025

Uh oh!

cscheid commented Jun 16, 2025

Uh oh!

Uh oh!

MichaelHatherly commented May 22, 2025 •

edited

Loading

MichaelHatherly commented May 22, 2025 •

edited

Loading

codecov bot commented May 22, 2025 •

edited

Loading

cscheid commented May 22, 2025 •

edited

Loading