Skip to content

CI: Fix slow mamba solver issue by limiting boto3 versions #61594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jun 13, 2025

Conversation

datapythonista
Copy link
Member

Closes #61531

Probably better to rerun the CI 3 or 4 times to be sure this is the problem and the solution. But based on local tests, seems like boto3 has a huge number of versions (they release almost every day), and that's the problem with the mamba solver. Limiting the number of versions provided to the solver should help. 1.27 is from 2 years ago, consistent with other packages. Why only fails for 3.13? No idea

@datapythonista datapythonista requested a review from mroeschke as a code owner June 6, 2025 21:01
@datapythonista datapythonista added CI Continuous Integration Dependencies Required and optional dependencies labels Jun 6, 2025
@datapythonista
Copy link
Member Author

This doesn't seem to work. Removing boto3 from the environment seems to fix the problem locally, I thought limiting the options would also fix it. But seems like even pinning to a specific version doesn't work. I guess the problem may be with a boto3 dependency, but not too sure.

@datapythonista
Copy link
Member Author

The dependencies for boto3 are here: https://github.com/conda-forge/boto3-feedstock/blob/main/recipe/meta.yaml#L23

Not too sure if the problem is really with boto3, maybe it's more complex but removing boto3 makes it work. But doesn't seem like mamba really checks their issues, and I don't think we want to get into the mamba solver internals ourselves, so I guess we need to just keep trying something that makes the solver for 3.13 take a reasonable time.

@datapythonista
Copy link
Member Author

Based on my tests, this is what makes the solver happy. Relaxing the conditions causes the very slow mamba install times. This breaks the check for minimum versions, so it can't be merged as is (and I'll add a comment when we decide what to do).

@mroeschke not sure if you have an opinion on what to do to avoid the mamba problems based on this. Not too sure what's going on.

@mroeschke
Copy link
Member

I'm curious if conda's libmamba solver has the same issue. Potentially based on conda/conda-libmamba-solver#668 it may. (It's on my radar to maybe switch to https://github.com/conda-incubator/setup-miniconda given that mamba activity is kinda slow as you mentioned)

It's a shame we have to pin, but I'm OK with it to speed up solve times

@datapythonista
Copy link
Member Author

I had the impression that conda, mamba, micromamba and pixi all use libmamba, so not sure if changing tool would be helpful. But I'm not so sure about the exact details.

@WillAyd
Copy link
Member

WillAyd commented Jun 9, 2025

I don't know what conda is doing with its solver, but I've seen this issue with boto3 and the standard pip dependency resolver, which downloads a copy of every possible boto version and inspects its metadata to find out what constraints it might have to solve for. The sheer number of boto versions to download makes that a really slow process

@datapythonista
Copy link
Member Author

I'll leave out the s3fs>=2025.5.1 for now, and just pin boto3 in all the dependency files. I see mamba being very slow in other versions of pandas now. Since boto3 is not a pandas dependency, but something we use for testing, pinning shouldn't be a problem. Just pining boto3 didn't make mamba resolve immediately in my tests, but with some luck it makes resolve faster, and we can avoid the cancelled jobs.

@datapythonista
Copy link
Member Author

The first run of the CI after just pinning boto3 didn't have long mamba solver times. I'll try a couple more times and see if it continues to be the case. And if it is, I think we can merge this and leave the dependencies like this until we have new problems with the solver. The other options I can think of require changing the versions of s3fs to what we'd like to have, which is not great.

@jorisvandenbossche
Copy link
Member

This looks good to me. The one failing test is a xpass, probably because xarray got updated to 2025.6.0 and they might have fixed something. Will do a PR updating this test.

@jorisvandenbossche
Copy link
Member

Will do a PR updating this test.

-> #61648

@jorisvandenbossche jorisvandenbossche added this to the 2.3.1 milestone Jun 13, 2025
@datapythonista datapythonista merged commit 2bf3dc9 into pandas-dev:main Jun 13, 2025
44 checks passed
Copy link

lumberbot-app bot commented Jun 13, 2025

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 2.3.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 2bf3dc9850bf085242711553982e75f019c31d1d
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #61594: CI: Fix slow mamba solver issue by limiting boto3 versions'
  1. Push to a named branch:
git push YOURFORK 2.3.x:auto-backport-of-pr-61594-on-2.3.x
  1. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #61594 on branch 2.3.x (CI: Fix slow mamba solver issue by limiting boto3 versions)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

@jorisvandenbossche
Copy link
Member

Backport -> #61653

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration Dependencies Required and optional dependencies
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI: Micromamba taking too long to resolve the environments in the CI
4 participants