Submitting a Python package with GitHub and PyPI
Update (2017-11-17)
The submission process has changed slightly since I wrote this post.
PyPI is migrating from https://pypi.python.org/pypi/ to http://pypi.org. The old site is still up, but I don’t think it’s accepting submissions anymore - something I discovered while trying to submit a new version of simplestatistics.
The two big differences are:
- The submission url has changed. If you have a
~/pypirc
file, you can probably fix the submission issue by removing a line that begins withrespository=
, allowing the default url to be used. See this post for more information.
- You used to be able to specify a different url in
~/.pypirc
to usetestpypi
, the test server you could use to test the submission. I think this no longer works, and you might need to use twine to be able to make use of that again. I think PyPI is generally encouraging everyone to use twine for all their submission needs.
I maintain and publish a math and statistics Python package called simplestatistics. The library is a port of its javascript ancestor simple-statistics. While building it up, I wanted it to be available on the Python Package Index, PyPI1, and I had to learn how to do that. The process turned out to be more complicated than I expected it to be, and so here are the steps it takes to publish your package, or its updates, to PyPI.
Note: The order of steps matters.
Note 2: Some of these steps are specific to hosting your package on GitHub.
- Do you have a
setup.py
? - Update
changelog.txt
- Update version number in documentation
- Update version number in
setup.py
- Convert
README.md
toREADME.rst
- Add tarball download url to
setup.py
- If there are new files that should be included, edit
MANIFEST.in
- Commit all those changes. Have a clean repo.
- Add/create a git tag
- Push git tag to remote
- Confirm that GitHub has generated the release file
- Release testing
- Release
- Add changelog notes to GitHub release page/tag
- Congratulations!
- See also
Do you have a setup.py
?
You need to create a setup script to publish your package using distutils
. This is the one for simplestatistics
.
from distutils.core import setup
setup(
name = 'simplestatistics',
packages = ['simplestatistics', 'simplestatistics.statistics'],
version = '0.2.5',
description = 'Simple statistical functions implemented in readable Python.',
author = 'Sherif Soliman',
author_email = 'sherif@ssoliman.com',
copyright = 'Copyright (c) 2016 Sherif Soliman',
url = 'https://github.com/sheriferson/simplestatistics',
download_url = 'https://github.com/sheriferson/simplestatistics/tarball/0.2.5',
keywords = ['statistics', 'math'],
classifiers = [
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 3',
'Topic :: Scientific/Engineering :: Mathematics',
'Intended Audience :: Developers',
'Intended Audience :: Education',
'Intended Audience :: End Users/Desktop',
'Intended Audience :: Science/Research',
'Operating System :: MacOS',
'Operating System :: Unix',
'Topic :: Education',
'Topic :: Utilities'
]
)
Update changelog.txt
If you maintain a changelog file, make sure you update it with the release version and date.
If you don’t maintain a changelog file, you should.
I maintain a separate HISTORY.rst
file, so I make sure I update that too.
Update version number in documentation
Good documentation is important to me. It helps you understand your code and project better, and definitely helps anyone else trying to use it. simplestatistics documentation is hosted and generated automatically by Read the Docs using the very useful Sphinx package.2
When I’m publishing a new release, I make sure I update the version number in Sphinx’s conf.py
.
Update version number in setup.py
I mentioned setup.py
in the first step. Make sure to update the version number.
from distutils.core import setup
setup(
...
version = '0.2.5',
...
Convert README.md
to README.rst
PyPI doesn’t like Markdown. Actually, it’s not that it doesn’t like it, it just doesn’t care about it one way or another. PyPI likes reStructuredText (RST from this onwards). If you want PyPI to render the README on the package homepage like you can see on the simplestatistics PyPI page, it has to be in RST.
I’ve had a lot of trouble with RST. In my experience, it’s very fragile. It takes one extra space in a table to break rendering for the whole file.
pandoc is a great tool you could use to convert your README.md
to README.rst
, but PyPI may not like the default output of pandoc’s conversion. In my use case, the conversion of the Markdown tables to RST tables was the part that often angered PyPI rendering.
After some troubleshooting, I found that this command prevents the tables from wrapping around to new lines and causing README rendering on PyPI to fail.
pandoc --columns=100 --output=README.rst --to rst README.md
Here’s a bonus tip: this online reStructuredText editor, made available by Andrey Rublev, has been a huge help in debugging and fixing RST errors.
Add tarball download url to setup.py
In a later step, we will add a git tag and push it to GitHub. This will create a new release on the GitHub page, and this release will include .zip
and .tar.gz
files of the release (see an example here).
These compressed files are the ones that PyPI will pull from when you push your release or update. PyPI gets that download url from setup.py
.
The result of this circle is that you need to anticipate the url for the release on GitHub before you push the release commit to GitHub.
You can set the new download url in setup.py
based on your new version number:
setup(
...
url = 'https://github.com/sheriferson/simplestatistics',
download_url = 'https://github.com/sheriferson/simplestatistics/tarball/0.2.5',
...
Yes, you do set this url and commit it before it actually exists.
If there are new files that should be included, edit MANIFEST.in
The MANIFEST.in
file is how you tell disutils
to include files in the release file that it wouldn’t include otherwise. This is my MANIFEST.in
file:
include LICENSE.txt
include README.rst
HISTORY.rst
Commit all those changes. Have a clean repo.
Commit everything we’ve done so far. Have a consistent commit comment for those changes. I usually add the message: “Prep for 0.2.5 release.”
Add/create a git tag
git tag 1.2.3 -m "Adds 1.2.3 tag for PyPI
Once you have the project in the state you want for creating the release, you add a git tag with the version number of the release. This will be reflected in the “releases” page of your GitHub repository.
Push git tag to remote
git push --tags origin master
Push those tags to GitHub.
Confirm that GitHub has generated the release file
Browse to your releases page (example) and make sure the new version has a release entry with its corresponding files.
Release testing
Update (2017-11-17)
See update at the top for notes about testpypi. The instructions below are here for posterity only. You probably want to use twine.
python setup.py register -r pypitest
You are, or aspire to be a good programmer who wants to be as cautious as possible, and so you’d like to test releasing the update on PyPI before actually doing it.
PyPI provides a test system that you can use to test the registration, upload, and installation of your package. Let’s make use of that gift.
python setup.py register -r pypitest
will register the package on the pypitest
server.
If you get a message indicating the all-clear, continue.
python setup.py sdist upload -r pypitest
… will upload the distribution of your package to pypitest
.
If something in the package is broken, you cannot make changes and reupload the package with the same version number. This means that if you want to make a change or fix something, you will have to change the version number in setup.py
(and accordingly everywhere else) and start all over again.3
pip install -i https://testpypi.python.org/pypi simplestatistics
… is the final step in the process of testing the release. This will try to install the new version of the package from pypi
. If all goes well, you should be able to import the package normally in the REPL or in a Python script.
Once you’ve tested importing the package, it’s a good idea to pip uninstall
your package so you can test the installation from the live PyPI servers.
Release
python setup.py register -r pypi
You are finally at the actual release stage. This will register the new version with PyPI.
python setup.py sdist upload -r pypi
… uploads the distribution to PyPI.
pip install simplestatistics
… tests the installation from PyPI. This is why we pip uninstalled
the version from pypitest
.
Add changelog notes to GitHub release page/tag
This is not necessary, but it’s good practice and shows care for maintaining documentation of your open source project. Edit the new GitHub release and add notes about what changed in nicely formatted Markdown.
Congratulations!
You made it. It feels good to have a package on PyPI. It helps you use your own package in the future, and it’s a good contribution to make it available to everyone else. Go celebrate.
See also
- How to submit a package to PyPI - I learned a large part of what I described here from this helpful article by Peter Downs. It’s a good resource.
-
Which it is!
pip install simplestatistics
↩︎ -
The thing I like most about Sphinx is that you write the code, and in the case of
simplestatistics
the tests, in the docstrings of each.py
file. The thing I like the least is that you have to use reStructured text, which is a bouquet of sadness. ↩︎ -
At the time of writing. Things might change and become more flexible in the future. I hope so. ↩︎