Update conda-forge feedstock
Oct 21, 2017
Curtis Alexander
6 minute read

Summary

Utilize the instructions below to update a conda-forge feedstock.

Background

Before going further, if you don’t know what conda-forge is, I’ll quote directly from their website:

A community led collection of recipes, build infrastructure and distributions for the conda package manager.

For more background information, see the following posts from two members of the PyData community.

Within the conda-forge description above are what I believe to be keys for the success of the project:

  • build infrastructure
    • I know little about how the infrastructure works but I find it to be an awesome example of using continuous integration + git + GitHub to deliver packages to several programming language ecosystems.
    • Without this level of automation, the Python community (and to a lesser extent the R community) was relying on the hard work of Continuum to develop and build packages. I think it is a win for all that the community is in the process of working towards aleviating and eventually taking over the efforts to maintain package recipes. Presumably, this frees up Continuum to push forward so many of their awesome projects.
  • community1 led
    • That means people doing things on their own time because they want to help!

Normally, I would not contribute to a project such as this — not because I have anything against the project but simply because I normally feel that I have nothing to contribute. However, I have benefited so greatly from free, open-source software that I should find some small way to become more involved2.

Figuring out exactly how to build a recipe wasn’t fun3. But once I pieced together how to utilize the build infrastructure, it doesn’t feel burdensome or even difficult to maintain a few packages.

Nix [Aside]

I am quite interested in learning more about package management using nix. In fact, I was just looking at the nix-in-place repository that allows one to install NixOS on top of a Linux distribution. Next step would be to experiment on a Digital Ocean droplet or on an ARM board.


What to Update?

Use feedstockrot in order to determine which feedstocks one needs to update.4

Approximately weekly, I run a meta-update script to determine which feedstocks I need to update. Below is the appropriate section from my meta-update script (which assumes feedstockrot is on one’s PATH).

echo
echo "#################"
echo "## conda forge ##"
echo "#################"
echo
echo "The following feedstocks require updating..."
echo

# update using your GitHub token
export FEEDSTOCKROT_GITHUB_TOKEN="12345abcdefg"
feedstockrot --github

Currently5, I maintain the following feedstocks:

  • openpyxl-feedstock
  • greenlet-feedstock
  • jdcal-feedstock
  • et_xmlfile-feedstock
  • neovim-feedstock
  • trollius-feedstock
  • pyuv-feedstock

How to Update?

All examples will use the openpyxl package as an example.

Github Setup

Begin by forking the main repository for the feedstock. For instance, to update openpyxl, fork the openpyxl-feedstock repository.

The fork now resides within one’s personal account.

Local Setup

Clone the forked repository to one’s local machine.

git clone https://github.com/curtisalexander/openpyxl-feedstock/

Next, create a new branch for the new version. For clarity, branch names should be of the form package_name-version.

git branch openpyxl-2.5.0a2

Then switch to the new branch.

git checkout openpyxl-2.5.0a2

Update the Feedstock

To update the feedstock, simply edit the file recipe/meta.yaml.

Within the meta.yaml file, the following entries need to be updated:

{% set version = "2.5.0a1" %}
{% set sha256 = "fe00ef04c5560504c625cac9f18913a1d3fb414f0560e715af95d894a6b43d63" %}
...
build:
  number: 0

In order to update the sha256 value, the tar.gz file needs to be downloaded. For openpyxl, the file that needs to be downloaded can be found at pypi. The canonical link at the site is for the latest version. For instance, as of this writing the openpyxl link is for version 2.5.0b1.

Again, the canonical link is for the most recent version of the package. If older packages need to be downloaded, navigate to the following link for archived versions.

Once downloaded, the sha256 can be acquired by running the following.

openssl sha256 ~/Downloads/openpyxl-2.5.0a2.tar.gz

Local Commit |> Push |> Create a Pull Request

Commit the changes and push the branch to the origin remote and not the upstream remote. The origin remote points to the fork in one’s personal account and the upstream remote points to the original, or main, repository.

git add meta.yaml
git commit -m 'Update for version 2.5.0a2'
git push origin openpyxl-2.5.0a2

Now navigate back to the main openpyxl-feedstock site and create a pull request by clicking the button Compare & pull request.

Build (and wait)

After clicking Compare & pull request, a pull request will be created within the tab Pull requests within the main openpyxl-feedstock repository. The conda-forge build system will then take over, initiating builds on Linux (Travis CI), MacOS (CircleCI), and Windows (AppVeyor).

The builds can take a little bit of time to complete. Previously, AppVeyor took quite some time to complete but of late they have been running much quicker of late. At this time of this writing, Travis CI is lagging.

If one needs to trigger a rebuild — for whatever reason — then the easiest way to do so is noted at the conda-forge/staged-recipes repository.

git commit --amend
git push -f origin openpyxl-2.5.0a2

Merge

If there are no errors then the pull request can be merged! Just click on the Merge pull request button at the upstream site.

If there are errors, then read through the logs within the relevant CI system and make the necessary changes. When making these updates / changes, increment the build number within meta.yaml.

After merging, the just built packages are uploaded to the conda-forge repository for potential download.

Cleanup

There are three places where things need to be cleaned up.

Forked & Main Repositories

After merging on GitHub, one will have the option to delete the branch at forked repository — curtisalexander:openpyxl-2.5.0a2. Simply click the button Delete branch. At this point the forked repository — curtisalexander/openpyxl-feedstock — and the main repository — conda-forge/openpyxl-feedstock — only have a single, master branch. All other branches have been cleaned up.

Local Repository

Now the master branch at the fork — the origin remote — needs to be synced back with the master branch on one’s local machine. It is at this stage I tend to get myself into trouble.

Following these directions, execute the following locally. Note that this is only done after one has already set the upstream branch.

git fetch upstream
git checkout master
git merge upstream/master
git push origin master
git branch -d openpyxl-2.5.0a2
git remote prune origin
# alternative is git fetch -p

What just happened?

At this point the following should be true:

  • The master branch at the forked repository — curtisalexander:master — should be even with the master branch at the main repository — conda-forge:master.
  • There should only be a single branch — master — at the forked and main repositories — i.e. the just utilized branches should be cleaned up.
  • Locally, the working branch (openpyxl-2.5.0a2) should be deleted.
  • Locally, the reference to the remote (curtisalexander:openpyxl-2.5.0a2) should be pruned and cleaned up.

Check

Finally, to check that the new package is available, run the following.

conda update openpyxl --dry-run

The just updated version of the package should be displayed.


  1. I normally think of the community as something other — but if I’m contributing my small part, then the community includes me.

  2. Yes, I did want to become more involved. But my involvement grew out of the fact that I wanted to manage openpyxl via the conda-forge channel as opposed to the default channel. An innocuous beginning!

  3. I temporarily broke the pandas build.

  4. In the future I may try utilizing tick_my_feedstocks.py.

  5. As of 2018-02-10.