allow specifying an oe-eval branch

[davidheineman]

Currently, we can set --oe-eval-commit for a commit on the main branch of oe-eval. This allows changing the branch. (commit logic is unchanged -- one could point to a commit on a different branch)

olmo-cookbook-eval evaluate {model} \
    ...
    --oe-eval-branch cookbook-updates \

Example command:

olmo-cookbook-eval evaluate allenai/OLMo-2-0425-1B \
    --tasks arc_easy:rc::olmes:full \
    --priority high \
    --cluster aus80g \
    --num-gpus 1 \
    --model-backend vllm \
    --partition-size 8 \
    --huggingface-secret davidh_HUGGING_FACE_HUB_TOKEN \
    --vllm-use-v1-spec \
    --dashboard cookbook-debug \
    --workspace ai2/olmo-3-evals \
    --model-args max_length=4096,trust_remote_code=true \
    --use-gantry \
    --oe-eval-branch cookbook-updates

[soldni]

changing the hash should pick the branch already; is that not working? it works fine for me using git CLI directly, e.g.

>>> git checkout main
>>> git log | head -5
commit 94e6db26d5df6b7d2a4a4c6455f8aaa00a8de75f
Author: davidheineman <[email protected]>
Date:   Tue Jun 17 11:33:04 2025 -0700

    Revert "update 7b task set"
>>> git checkout soldni/code
>>> git log | head -5
commit 9c3ccbf607b1c2010464843031694941b781d773
Author: Luca Soldaini <[email protected]>
Date:   Sun Jun 8 10:50:56 2025 -0700

    Refactor YAML configurations for spring2code dataset sources
>>> git checkout main
>>> git checkout 9c3ccbf607b1c2010464843031694941b781d773
Note: switching to '9c3ccbf607b1c2010464843031694941b781d773'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 9c3ccbf Refactor YAML configurations for spring2code dataset sources
>>>  git log | head -5
commit 9c3ccbf607b1c2010464843031694941b781d773
Author: Luca Soldaini <[email protected]>
Date:   Sun Jun 8 10:50:56 2025 -0700

    Refactor YAML configurations for spring2code dataset sources

if it is not working, maybe we should figure out why that is the case?

[davidheineman]

This works in the CLI, yes, but not in the cookbook. Here's an example (using the old implementation):

olmo-cookbook-eval evaluate allenai/OLMo-2-0425-1B --tasks *olmo3:dev:7b:vllm --priority high --cluster aus80g --num-gpus 2 --model-backend vllm --partition-size 8 --huggingface-secret davidh_HUGGING_FACE_HUB_TOKEN --vllm-use-v1-spec --vllm-memory-utilization=0.8 --dashboard olmo-3-intermediate --workspace ai2/olmo3-webdata --model-args max_length=4096,trust_remote_code=true --gantry-args retries=3   --use-gantry --oe-eval-commit cookbook-updates

Found git directory at /oe-eval-default/davidh/eval/olmo-cookbook; using as venv base location
Using virtual environment: /oe-eval-default/davidh/eval/olmo-cookbook/.venv/oe-eval-venv...
Using Python virtual environment at oe-eval-venv
Installing beaker and gantry clients...
Cloning repository from [email protected]:allenai/oe-eval-internal.git to /tmp/tmpkcbksejg...
Cloning into '/tmp/tmpkcbksejg'...
remote: Enumerating objects: 671, done.
remote: Counting objects: 100% (671/671), done.
remote: Compressing objects: 100% (558/558), done.
remote: Total 671 (delta 46), reused 434 (delta 28), pack-reused 0 (from 0)
Receiving objects: 100% (671/671), 7.59 MiB | 19.78 MiB/s, done.
Resolving deltas: 100% (46/46), done.
remote: Enumerating objects: 12313, done.
remote: Counting objects: 100% (12313/12313), done.
remote: Compressing objects: 100% (3692/3692), done.
remote: Total 12038 (delta 8850), reused 11342 (delta 8244), pack-reused 0 (from 0)
Receiving objects: 100% (12038/12038), 19.37 MiB | 12.50 MiB/s, done.
Resolving deltas: 100% (8850/8850), completed with 224 local objects.
From github.com:allenai/oe-eval-internal
 * branch            cookbook-updates -> FETCH_HEAD
error: pathspec 'cookbook-updates' did not match any file(s) known to git
Error cloning repository at '[email protected]:allenai/oe-eval-internal.git'  (commit cookbook-updates): Command '['git', 'checkout', 'cookbook-updates']' returned non-zero exit status 1.
Cleaning up /tmp/tmpkcbksejg...
Traceback (most recent call last):
  File "/root/.conda-remote/envs/oe-eval/bin/olmo-cookbook-eval", line 10, in <module>
    sys.exit(eval.cli())
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/oe-eval-default/davidh/eval/olmo-cookbook/src/cookbook/cli/eval.py", line 369, in evaluate_model
    evaluate_checkpoint(
  File "/oe-eval-default/davidh/eval/olmo-cookbook/src/cookbook/eval/evaluation.py", line 67, in evaluate_checkpoint
    oe_eval_dir = install_oe_eval(
  File "/oe-eval-default/davidh/eval/olmo-cookbook/src/cookbook/cli/utils.py", line 185, in install_oe_eval
    oe_eval_dir = clone_repository(OE_EVAL_GIT_URL, commit_hash)
  File "/oe-eval-default/davidh/eval/olmo-cookbook/src/cookbook/cli/utils.py", line 402, in clone_repository
    raise e
  File "/oe-eval-default/davidh/eval/olmo-cookbook/src/cookbook/cli/utils.py", line 393, in clone_repository
    subprocess.run(shlex.split(f"git checkout  '{commit_hash}'"), check=True)
  File "/root/.conda-remote/envs/oe-eval/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'checkout', 'cookbook-updates']' returned non-zero exit status 1.
Eval command failed: Command '['olmo-cookbook-eval', 'evaluate', 'allenai/OLMo-2-0425-1B', '--tasks', '*olmo3:dev:7b:vllm', '--priority', 'high', '--cluster', 'aus80g', '--num-gpus', '2', '--model-backend', 'vllm', '--partition-size', '8', '--huggingface-secret', 'davidh_HUGGING_FACE_HUB_TOKEN', '--vllm-use-v1-spec', '--vllm-memory-utilization=0.8', '--dashboard', 'olmo-3-intermediate', '--workspace', 'ai2/olmo3-webdata', '--model-args', 'max_length=4096,trust_remote_code=true', '--gantry-args', 'retries=3', '--use-gantry', '--oe-eval-commit', 'cookbook-updates']' returned non-zero exit status 1.

I think this is because we don't clone the entire repo, just the head:

cmd = shlex.split(f"git clone --depth 1 {git_url}")

I like this because it speeds up the clone, but then the branch doesn't exist, so we have to pull from the ref at the origin.

[soldni]

that's a good point! I think we can resolve that with git fetch origin <commit-hash> first, then followed by git checkout <commit-hash>?

that way we can avoid using branches, which can unexpectedly change and cause bugs.

[davidheineman]

Those commands are the current logic. However, git doesn't know what the remote branch is, because we haven't cloned the entire repo (only git clone --depth 1 {git_url}).

This logic will fail:

git clone --depth 1 https://github.com/allenai/oe-eval-internal.git
cd oe-eval-internal
git fetch origin cookbook-updates
git checkout cookbook-updates
# error: pathspec 'cookbook-updates' did not match any file(s) known to git

So, we need to pull the refs for that branch from the remote:

git fetch origin cookbook-updates:refs/remotes/origin/cookbook-updates
git checkout -b cookbook-updates origin/cookbook-updates
# success

This requires us to know the branch name ahead of time. I don't think we can do this with the commit hash only.

Also -- My use-case would find tracking branches to be useful! For example, Shane has custom vLLM branch for OLMo 3. This is currently in an oe-eval branch. When we implement new evals, we need to re-base that oe-eval branch, and it would be nice for cookbook to track the head of that branch directly, just like how cookbook tracks the main branch by default.

[undfined]

Just my 2 cents on basing commands on branches; while convenient they offer lots of ways to shoot yourself in the foot. Commits are atomic and predictable though slightly less convenient.

If we continue to support both options then I'm all for it.

One other thought, why even clone at a depth of 1? In most other workflows (locally and remotely) we just clone the whole repo and then checkout the commit we want (training workflow in cookbook and OLMo-core being another example). We shouldn't have so much cruft in these repos that cloning the entire graph takes more than a second or two.

[davidheineman]

Yes! This commit supports commit only, branch only or both options. Commit hashes are great when you want to pin to a version of oe-eval, but when I'm iterating it makes things quite challenging.

Unfortunately, oe-eval-internal has a lot of bloat in the commit history, a clone takes a bit over 20 seconds IIRC.