While the R 4.0 migration has been functionally complete for quite a
while, the recent migration of r-java and its dependents gives a good
opportunity to write a retrospective on the technical issues with
large-scale migrations in conda-forge and how we solved them.
The R 4.0 migration rebuilt every package in conda-forge that had
r-base as a requirement, including more than 2200 feedstocks. A
migration of this size in conda-forge faces several hurdles. First,
since every feedstock is a separate GitHub repository, one needs to
merge more 2200 pull requests (PRs). Second, conda-forge's packages
on anaconda.org are behind a CDN (content delivery network). This
service reduces web hosting costs for Anaconda Inc. but introduces an
approximately 30 minute delay from when a package is uploaded to
anaconda.org and when it will appear as available using conda from
the command line. Thus, even if the dependencies of a package have been
built, we have to wait until they appear on the CDN before we can
successfully issue the next PR and have it build correctly. Finally, the
existing bot and conda infrastructure limited the throughput of the
migrations, due in part to the speed of the conda solver.
Given the size of the R 4.0 migration, we took this opportunity to try
out a bunch of new technology to speed up large-scale migrations. The
main enhancements were using GitHub Actions to automerge PRs, using
mamba to quickly check for solvability of package environments, and
enabling long-running migration jobs for the autotick bot. All told, the
bulk of the feedstocks for R 4.0 were rebuilt in less than a week, with
many PRs being merged in 30 minutes or less from when they were issued.
These enhancements to the autotick bot and conda-forge infrastructure
can be used to enhance future migrations (e.g., Python 3.9) and reduce
maintenance burdens for feedstocks.
Automerging conda-forge PRs
In a typical migration on conda-forge, we issue a PR to a feedstock
and then ask the feedstock maintainers to make sure it passes and merge
it. In the case of the R 4.0 migration, the maintainers of R packages on
conda-forge use a maintenance team (i.e., @conda-forge/r) on the
vast majority of feedstocks. This team is small and so merging over 2000
PRs by hand is a big undertaking. Thus, with their permission, we added
the conda-forge automerge functionality to all R feedstocks that they
maintain. The automerge bot, which relies on GitHub Actions, is able to
automatically merge any PR from the autotick bot that passes the recipe
linter, the continuous integration services, and has the special
[bot automerge] slug in the PR title. This feature removed the
bottleneck of waiting for maintainers to merge PRs and reduced the
maintenance burden on the R maintenance team.
Checking Solvability with mamba
While being able to automatically merge PRs removed much of the work of performing the R 4.0 migration, it relied on the PR building correctly the first time it was issued. Due to the CDN delays and the build times of a package's dependencies, the dependencies of a package may not be immediately available after all of their migration PRs are merged. If the bot issued the packages migration PR before the dependents are available, the PR would fail with an unsolvable environment and have to be restarted manually. This failure would negate any of the benefits of using automerge in the first place.
To control for this edge case, we employed the mamba package to check
for the solvability of a PR's environments before the PR was issued.
mamba is a fast alternative to conda that produces solutions for
environments orders of magnitude more quickly. Since, we have to perform
our checks of PR environments many times, an extremely fast solver was
essential for making the code efficient enough to run as part of the
autotick bot. We ended up using mamba to try to install the dependencies
for every variant produced by the feedstock to be migrated. With this
check in place, the autotick bot was able to issue migration PRs that
passed on the first try and were thus automatically merged, many within
30 minutes or less.
Improving the Autotick Bot's Efficiency
Finally, we made several upgrades to the autotick bot infrastructure to increase the uptime of the bot and its efficiency. First, we moved from an hourly cron job to a set of chained CI jobs. This change eliminated downtime between the runs of the bot. Second, we started to refactor the autotick bot from one monolithic piece of code into a distributed set of microservices which perform various independent tasks in parallel. These independent tasks, used for things like checking the statuses of previously issued PRs, are run separately allowing the bot to spend more time issuing PRs. Finally, we optimized the internal prioritization of the PRs to make sure the bot was spending more time on larger migrations where there is more work to do. More work on the autotick bot infrastructure, including work done by Vinicius Cerutti as part of the Google Summer of Code program, will further streamline the bot's operation.
Despite some initial hiccups with the bot infrastructure, the migration
ran quite smoothly for an endeavor of its size. The vast majority of
migration PRs were completed within a week from when we started, which
is a first for a migration of this size on conda-forge. The largest
issue was solved recently, with the fixing of the openjdk recipe and
the removal of aarch64 and ppc64le builds from r-java, enabling
the last large piece of the R ecosystem to be updated.
Looking forward, the improvements we made for the R 4.0 migration seem
broadly applicable to other migration tasks, including the yearly python
minor version bump. These kinds of large-scale migrations are
particularly suitable, since they usually involve few changes to the
feedstock itself and usually fail on CI when a broken package would be
produced. Faster migrations will help to provide the latest features to
downstream users and keep transition times to a minimum, helping to
foster greater stability of the ecosystem and the seamless experience
users have come to expect from conda-forge.