Python packaging for idiots (me)

I have now reached the need to package a Python package and publish it through PyPI for the world (albeit a very small world) to have unfettered access to it. The package is called "Metapype", and its purpose is to simplify generating and managing scientific metadata. This post, however, is not about Metapype; rather, it is about the painful process (try and try again) I learned to package this Python package.

First off, I decided to be modern and package Metapype without invoking setup.py. Instead, I am using PyPA's build tool (pip install build; and not yet in conda-forge) to create both the "wheel" and the "sdist" packages, which uses configurations in pyproject.toml, setupf.cfg, and MANIFEST.in files (more below). Using build avoids the controversial use of setup.py to install and create distributable packages - it still uses setuptools, but does require the reflective execution of setup.py. The bonus about using setup.cfg is that modern pip will read its attribute content just like setup.py to perform a local installation of the package. Unfortunately, I must keep setup.py alive and well (i.e., in parity with setup.cfg) because pip does not yet support "editable" installations using setup.cfg. Oh well.

Sidenote: my experience indicates that build (and I assume its use of setuptools) preferentially uses setup.py over setup.cfg if both are in the same directory, which is causing an issue when pulling in the long_description from README.md. I've not yet figured this one out.

Pre-step

Remember always (and I emphasize "always") to remove lingering build cruft before attempting a new build. Why you ask? I don't know the details, but prior build content seems to get sucked into the new build and be there for the world to see. Make it a habit to clean up the package state before attempting a new build - this will save you grief and time:

rm -rf dist/ src/metapype.egg-info

Step 1: Bump Version

Once I have the local git state set correctly, I bump the Metapype version value in src/metapype/VERSION.txt and perform the perfunctory git commit and push to GitHub. The file VERSION.txt is a one-line file that contains the semantic version value (e.g., 1.4.0). This is picked up by setup.cfg, setup.py, and src/metapype/__init__.py.

Step 2: Create Tag

With Metapype at the latest commit state and in the Metapype project root, I set a tag to the current version (as noted in src/metapype/VERSION.txt) and push it to GitHub:

git tag -a v1.4.0 -m "Release with new evaluation"

git push origin v1.4.0

Step 3: Build Dist

With Metapype ready to be published and distributed through PyPI, the first thing I do is to navigate into the Metapype project root directory (where pyproject.toml and setup.cfg exist) using a new terminal that has the Conda tools virtual environment activated (this is where the build package is installed). Once in the project root, issue the following command (best to double-check the "Pre-step"):

python -m build

This will create the build, dist, and src/metapype/metapype.egg-info artifacts. It is the dist directory that contains the metapype-1.4.0-py3-none-any.whl wheel and metapype-1.4.0.tar.gz sdist packages used for publishing to PyPI.

Step 4: Publish to PyPI with Twine

The final step is to Publish the wheel and sdist packages to PyPI using twine. Again, use the tools virtual environment to execute twine as follows:

twine upload --repository testpypi dist/* for PyPI's test server

or

twine upload dist/* for PyPI's production server.

Either case will require the credentials of the target user (Metapype will use the edi.repository user id).

Note: Pypi no longer accepts login credentials for uploads. You must use an authentication token obtained from pypi.org for your project. See here for a user's perspective on uploading to pypi.org with an auth-token.

Step 5: Perform an installation into a test virtual environment

As a sanity check, it is best to perform an installation of Metapype into a test virtual environment just to make sure everything is working as expected:

pip install metapype

pyproject.toml

[build-system]
requires = [
    "setuptools>=49.6.0",
    "wheel"
]
build-backend = "setuptools.build_meta"

setup.cfg

[metadata]
name = metapype
version = file: src/metapype/VERSION.txt
author = Environmental Data Initiative
author_email = support@environmentaldatainitiative.org
description = Metapype: science metadata manipulation library
long_description = file: README.md
long_description_content_type = text/markdown
license_file = LICENSE
url = https://github.com/PASTAplus/metapype-eml
project_urls =
    Bug Tracker = https://github.com/PASTAplus/metapype-eml/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    =src
packages = find:
include_package_data = True
python_requires =  >=3.8
install_requires =
    click==7.1.2
    daiquiri==3.0.0
    lxml==4.6.2
    rfc3986==1.4.0

[options.packages.find]
where = src
include = metapype, metapype.eml, metapype.model

MANIFEST.in

include src/metapype/eml/rules.json
include src/metapype/VERSION.txt
global-include src/metapype/*.md