Packaging releases

MetaWards is now fully tested and deployed using GitHub actions. The development process should be;

• New features are developed on feature branches, called feature-{feature}, either in the main MetaWards repository for authorised developers, or in personal forks for new developers.

• Bug fixes or issue fixes are developed on fix branches, called fix-issue-{number} (again in either the main repository or forks).

• Pull requests are issued from these branches to devel. All merge conflicts must be fixed in the branch and all tests must pass before the pull request can be merged into devel. NOTE THAT ONLY AUTHORISED DEVELOPERS CAN ACCEPT THE PULL REQUEST. Authorised developers will review the pull request as quickly as they can. They will be greatly helped if the feature is accompanied with tests, examples and/or tutorial instructions, and if most changes are confined to plugins (e.g. metawards.iterators, metawards.extractors, metawards.mixers or metawards.movers).

The result of this is that “devel” should contain the fully-working and tested, and most up-to-date version of metawards. However, this version should not be used for production runs.

Note

The group of developers authorised to have access to the main MetaWards repository and to accept pull requests is not fixed, and will evolve over time. If you wish to join this group then please complete the tutorial and then demostrate your commitment by submitting good issues and pull requests from a personal fork of the repository. Please get in touch if you find this difficult, or follow this workshop if you need to learn how to use Git, GitHub, feature branching, merging, pull requests etc.

Defining a release

We will release metawards regularly. Releases aim to be backwards compatible and capable of being used for production runs, at least for the functionality that is fully described in the tutorial.

We use semantic versioning and take care not to cause breaking changes in the public API. The public API consists of the core metawards module only - it does not cover metawards.utils or any plugins that are in metawards.iterators, metawards.extractors, metawards.mixers or metawards.movers (although, the core plugin interface API will not change). We will endeavour to retain backwards compatibility outside of the core metawards module, but cannot guarantee it.

Note

It is the job of the release manager (currently chryswoods) to decide when it is time to create a new release. If you are interested in helping join the release management group then please feel free to get in touch.

Creating a release

To create a release first checkout the “main” branch.

git checkout main
git pull

Next, merge in all changes from the “devel” branch.

git pull origin devel

Next, update the Changelog with details about this release. This should include the link at the top of the release that shows the commit differences between versions. This can be easily copied from a previous release and updated, e.g.

`0.11.2 <https://github.com/metawards/MetaWards/compare/0.11.1...0.11.2>`__ - May 11th 2020

could be changed to

`0.12.0 <https://github.com/metawards/MetaWards/compare/0.11.2...0.12.0>`__ - May 18th 2020

when moving from the 0.11.2 to 0.12.0 release.

Now push this change back to GitHub, using;

git push

This will trigger a CI/CD run which will build and test everything on Windows, Mac and Linux for Python 3.7 and 3.8. Everything should work, as “devel” should have been in a release-ready state.

Testing the packages

GitHub actions will produce the source and binary wheels for metawards on all supported platforms. This will be in an artifact called dist which you should download and unpack.

You should unpack these into the dist directory, e.g.

cd dist
unzip ~/Downloads/dist.zip

This should result in six binary wheels and once source package, e.g.

metawards-0.11.1+7.g52b3671-cp37-cp37m-macosx_10_14_x86_64.whl
metawards-0.11.1+7.g52b3671-cp37-cp37m-manylinux1_x86_64.whl
metawards-0.11.1+7.g52b3671-cp37-cp37m-win_amd64.whl
metawards-0.11.1+7.g52b3671-cp38-cp38-macosx_10_14_x86_64.whl
metawards-0.11.1+7.g52b3671-cp38-cp38-manylinux1_x86_64.whl
metawards-0.11.1+7.g52b3671-cp38-cp38-win_amd64.whl
metawards-0.11.1+7.g52b3671.tar.gz

Try to install the package related to you machine, just to double-check that it is working, e.g.

pip install ./metawards-0.11.1+7.g52b3671-cp37-cp37m-macosx_10_14_x86_64.whl
cd ..
pytest tests

Once it is working, remove these temporary packages from your dist folder,

rm dist/*

Tagging a new release

Now that you are happy that the release is ready, you can tag the new version. Do this using the git tag command, e.g.

git tag -a {VERSION} -m "{VERSION} release"

replacing {VERSION} with the version number. For this 0.12.0 release the command would be;

git tag -a 0.12.0 -m "0.12.0 release"

Next, push your tag to GitHub;

git push --tags

The tag will be used by automatic versioning script to generate the version numbers of the code. Building the package (as happens below) will automatically update the _version.py that is included in the package to tag versions.

This will also trigger a full CI/CD to test and build the new version. Again, it should work as this tag was taken from your fully-tested “main” branch.

Uploading packages to pypi

While you are waiting for the CI/CD GitHub Actions to complete, make sure that your version of twine is fully up to date;

pip install --upgrade twine

Once GitHub actions is complete, you will see that another build artifact is ready for download. Download this and unpack it into your dist directory as before. You should now have a dist directory that contains six binary wheels and one source package, named according to the release version. For example, for the 0.11.2 release we had;

$ ls dist
 metawards-0.11.2-cp37-cp37m-macosx_10_14_x86_64.whl
 metawards-0.11.2-cp37-cp37m-manylinux1_x86_64.whl
 metawards-0.11.2-cp37-cp37m-win_amd64.whl
 metawards-0.11.2-cp38-cp38-macosx_10_14_x86_64.whl
 metawards-0.11.2-cp38-cp38-manylinux1_x86_64.whl
 metawards-0.11.2-cp38-cp38-win_amd64.whl
 metawards-0.11.2.tar.gz

Now you can upload to pypi using the command;

python3 -m twine upload dist/*

Note

You will need a username and password for pypi and to have permission to upload code to this project. Currently only the release manager has permission. If you would like join the release management team then please get in touch.

Testing the final release

Finally(!) test the release on a range of different machines by logging in and typing;

pip install metawards=={VERSION}

replacing {VERSION} with the version number, e.g. for 0.11.2

pip install metawards==0.11.2

Play with the code, run the tests and run some examples. Everything should work as you have performed lots of prior testing to get to this stage.