CI: Fix slow mamba solver issue by limiting boto3 versions

[datapythonista]

Closes #61531

Probably better to rerun the CI 3 or 4 times to be sure this is the problem and the solution. But based on local tests, seems like boto3 has a huge number of versions (they release almost every day), and that's the problem with the mamba solver. Limiting the number of versions provided to the solver should help. 1.27 is from 2 years ago, consistent with other packages. Why only fails for 3.13? No idea

[datapythonista]

This doesn't seem to work. Removing boto3 from the environment seems to fix the problem locally, I thought limiting the options would also fix it. But seems like even pinning to a specific version doesn't work. I guess the problem may be with a boto3 dependency, but not too sure.

[datapythonista]

The dependencies for boto3 are here: https://github.com/conda-forge/boto3-feedstock/blob/main/recipe/meta.yaml#L23

Not too sure if the problem is really with boto3, maybe it's more complex but removing boto3 makes it work. But doesn't seem like mamba really checks their issues, and I don't think we want to get into the mamba solver internals ourselves, so I guess we need to just keep trying something that makes the solver for 3.13 take a reasonable time.

[datapythonista]

Based on my tests, this is what makes the solver happy. Relaxing the conditions causes the very slow mamba install times. This breaks the check for minimum versions, so it can't be merged as is (and I'll add a comment when we decide what to do).

@mroeschke not sure if you have an opinion on what to do to avoid the mamba problems based on this. Not too sure what's going on.

[mroeschke]

I'm curious if conda's libmamba solver has the same issue. Potentially based on conda/conda-libmamba-solver#668 it may. (It's on my radar to maybe switch to https://github.com/conda-incubator/setup-miniconda given that mamba activity is kinda slow as you mentioned)

It's a shame we have to pin, but I'm OK with it to speed up solve times

[datapythonista]

I had the impression that conda, mamba, micromamba and pixi all use libmamba, so not sure if changing tool would be helpful. But I'm not so sure about the exact details.

[WillAyd]

I don't know what conda is doing with its solver, but I've seen this issue with boto3 and the standard pip dependency resolver, which downloads a copy of every possible boto version and inspects its metadata to find out what constraints it might have to solve for. The sheer number of boto versions to download makes that a really slow process

[datapythonista]

I'll leave out the s3fs>=2025.5.1 for now, and just pin boto3 in all the dependency files. I see mamba being very slow in other versions of pandas now. Since boto3 is not a pandas dependency, but something we use for testing, pinning shouldn't be a problem. Just pining boto3 didn't make mamba resolve immediately in my tests, but with some luck it makes resolve faster, and we can avoid the cancelled jobs.

[datapythonista]

The first run of the CI after just pinning boto3 didn't have long mamba solver times. I'll try a couple more times and see if it continues to be the case. And if it is, I think we can merge this and leave the dependencies like this until we have new problems with the solver. The other options I can think of require changing the versions of s3fs to what we'd like to have, which is not great.

[jorisvandenbossche]

This looks good to me. The one failing test is a xpass, probably because xarray got updated to 2025.6.0 and they might have fixed something. Will do a PR updating this test.

[jorisvandenbossche]

Will do a PR updating this test.

-> #61648

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.3.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 2bf3dc9850bf085242711553982e75f019c31d1d

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #61594: CI: Fix slow mamba solver issue by limiting boto3 versions'

4. Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-61594-on-2.3.x

5. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #61594 on branch 2.3.x (CI: Fix slow mamba solver issue by limiting boto3 versions)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[jorisvandenbossche]

Backport -> #61653