APE 17 Migration Guide¶

The Astropy project is now transitioning from using astropy-helpers for infrastructure to more standard Python packaging tools. The motivation and implications of this are discussed in an Astropy Proposal for Enhancements: APE 17: A roadmap for package infrastructure without astropy-helpers. The core astropy package has already migrated and these changes will be included in astropy v4.1.

This page aims to provide a guide to migrating to using the new infrastructure described in APE 17. We assume that your package is currently using astropy-helpers and that you have used a version of the Astropy package-template in the past to set up your package (though this guide might still be useful if the latter is not the case).

Throughout this guide, we will assume that your package is called my-package and that the module is called my_package. We deliberately choose a name where the package name is different from the module name, but for many cases, these will be the same.

Step 0: Re-rendering the template¶

In this guide, we will describe the changes to make to the files in your package. To make some of the steps easier below, you should first re-run cookiecutter to re-render the template to a temporary folder:

cookiecutter gh:astropy/package-template -o my_package_tmp

We will refer to some of the rendered files in several of the steps below.

Step 1: Remove astropy-helpers¶

To remove the astropy-helpers submodule, first, run:

git rm -r astropy_helpers

if astropy_helpers was the only submodule in your repository, the .gitmodules file will be empty, you can remove this if you wish:

git rm -f .gitmodules

Next you should remove the ah_bootstrap.py file:

git rm ah_bootstrap.py

You can now commit your changes with:

git commit -m "Remove astropy-helpers submodule"

Step 2: Update/create `setup.cfg`¶

The next step is to update make sure that you have a setup.cfg file that contains meta-data about the package. If you already have this file, you will likely need to update it, and if you don’t already have this file, you will need to create it by copying over the setup.cfg file generated in Step 0. If you are updating an existing file, you may also find it easier to start from the file generated in Step 0 and add back any customizations.

In any case this file should contain at least the following entries:

[metadata]
name = my-package
author = ...
author_email = ...
license = ...
license_file = LICENSE.rst
url = ...
description = ...
long_description = file: README.rst

[options]
zip_safe = False
packages = find:
install_requires =
    numpy
    astropy
    ...
python_requires = >=3.6

Replace the ... with the information for your package. Make sure that license_file and long_description have the right filename, and specify your required dependencies one per line in the install_requires section. Make sure the python_requires line is set to indicate the correct minimum Python version for your package.

If you already had a file, make sure you remove the following entries (if present):

package_name
version
tests_require
[ah_bootstrap] and all entries in it

Also remove any existing setup_requires entry, though we’ll add back this key for the setuptools-scm package in Step 8.

Step 3 - Define optional, test, and docs dependencies¶

Next up, add a new section to the setup.cfg file (or modify, if it already exists), to specify optional dependencies as well as dependencies required to run tests and build the documentation, for example:

[options.extras_require]
all =
    scipy
    matplotlib
test =
    pytest-astropy
docs =
    sphinx-astropy

If you don’t need any optional dependencies, remove the all section. You will likely need to have at least pytest-astropy in the test section and sphinx-astropy in docs.

Step 4 - Define package data¶

If your package includes non-Python data files, you will need to update how you declare which data files to include. If you have been using the Astropy package template, it is likely that you have functions called get_package_data defined inside setup_package.py files. Remove these functions, and instead define the package data using a [options.package_data] section inside your setup.cfg file, e.g.:

[options.package_data]
* = *.fits, *.csv
my_package.tests = data/*

In the above example, all .fits and .csv in the package will be included as well as all files inside my_package/tests/data.

Step 5 - Update your `setup.py` file¶

Copy the setup.py file you generated in Step 0 and replace your existing one - it should be good to go as-is without any further customizations.

Step 6: add a `pyproject.toml` file¶

The pyproject.toml file is used to declare dependencies needed to run setup.py and build the package. Copy the pyproject.toml file you generated in Step 0 and replace your existing one.

If your package doesn’t have any compiled extensions, the file should contain:

[build-system]
requires = ["setuptools",
            "setuptools_scm",
            "wheel"]
build-backend = 'setuptools.build_meta'

Step 7 - Handling C/Cython extensions¶

If your package has no compiled C/Cython extensions, you can skip this step. Otherwise, if you have C or Cython extensions, you can either define your extensions manually inside the setup.py file or make use of the extension-helpers package to collect extensions in a similar way to astropy-helpers.

Step 7a - Defining extensions manually¶

You can define extensions manually as described here. If you do this, you can remove all setup_package.py files in your package, and you don’t need to include extension-helpers in the pyproject.toml file.

If you have Cython extensions or your extensions use the NumPy C API, proceed to Step 7c, otherwise you can proceed to Step 8.

Step 7b - Using extension-helpers¶

You can use the extension-helpers package to:

Automatically define extensions for Cython files.
Pick up extensions declared in setup_package.py files, as described in the extension-helpers documentation.

The latter works by looking through all the setup_package.py files in your package and executing the get_extensions() functions, which each should return a list of extensions. Check through your existing setup_package.py files (if any), and make sure that any astropy_helpers imports are changed to extension_helpers. Also note that all functions in extension-helpers should now be imported from the top level. See the extension-helpers API documentation for a complete list of functions still provided by extension-helpers. Finally, make sure that any instance of include_dirs='numpy' is changed to include_dirs=np.get_include() and add the import numpy as np import if not already present.

Note that if you have existing setup_package.py files from an older version of the package template, you may have imports that look like:

from astropy_helpers import setup_helpers

and extensions that make use of:

cfg = setup_helpers.DistutilsExtensionArgs()

You should replace the import, from astropy_helpers import setup_helpers, with:

from collections import defaultdict

and change any lines that use setup_helpers.DistutilsExtensionArgs() to instead use:

cfg = defaultdict(list)

Provided you indicated when you generated the template in Step 0 that you wanted to use compiled extensions, you should be good to go. If not, make sure you add:

from extension_helpers.setup_helpers import get_extensions

just under the following lines at the top of the setup.py file:

import sys
from setuptools import setup

In addition, in the same file, add ext_modules=get_extensions() to the call to setup.py.

If you have Cython extensions or your extensions use the NumPy C API, proceed to Step 7c, otherwise you can proceed to Step 8.

Step 7c - Cython and Numpy build-time dependencies¶

If your compiled extensions rely on the NumPy C API, you will need to declare Numpy as a build-time dependency in pyproject.toml. Note that as described in APE 17, you need to pin the build-time Numpy dependency to the oldest supported Numpy version for each Python version. However, rather than doing this manually, you can add the oldest-supported-numpy package to the build dependencies in your pyproject.toml file. In addition if you have Cython extensions, you will need to also add an entry for Cython, pinning it to a recent version. Provided you indicated when you generated the template in Step 0 that you wanted to use compiled extensions, you should be good to go as both oldest-supported-numpy and cython should be in the pyproject.toml file. In this case your pyproject.toml file will look like:

[build-system]
requires = ["setuptools",
            "setuptools_scm",
            "wheel",
            "extension-helpers",
            "oldest-supported-numpy",
            "cython==0.29.14"]
build-backend = 'setuptools.build_meta'

Whenever a new major Python version is released, you will likely need to update the Cython pinning to use the most recent Cython version available.

Step 8 - Using setuptools_scm¶

The setuptools_scm package is now recommended to manage the version numbers for your package. The way this works is that instead of setting the version number manually in, e.g., setup.cfg or elsewhere in your package, the version number is based on git tags.

In Steps 5 and 6, we already added the required entry for setuptools_scm to setup.py and pyproject.toml.

In addition to these, we recommend that you define setup_requires inside the [options] section of your setup.cfg file:

[options]
...
setup_requires = setuptools_scm
...

This will already be the case if you copied the setup.cfg generated in Step 0. Having setup_requires is not strictly necessary but will make it possible for python setup.py --version to work without having to install setuptools_scm manually.

Next, check your .gitignore and make sure that you have a line containing:

my_package/version.py

Finally, copy over the _astropy_init.py file generated in Step 0, or alternatively edit your my_package/_astropy_init.py file and remove the following lines:

try:
    from .version import githash as __githash__
except ImportError:
    __githash__ = ''

and remove '__githash__' from the __all__ list at the top of the file. The git hash is now contained in the version number, so this is no longer needed.

Step 9 - Configuring pytest¶

To make sure that pytest works properly, you can set a few options in a [tool:pytest] section in your setup.cfg file:

[tool:pytest]
testpaths = "my_package" "docs"
astropy_header = true
doctest_plus = enabled
text_file_format = rst
addopts = --doctest-rst

For the testpaths line, make sure you replace my_package with the name of your package. This section will already exist if you copied the setup.cfg generated in Step 0.

The remaining options ensure that the output from pytest includes a header that lists dependencies and system information, and also ensure that the .rst files are picked up and tested by pytest.

Step 10 - Update `MANIFEST.in`¶

Edit your MANIFEST.in file to remove the following lines, if present (and any other line related to astropy_helpers) - those lines might include any of the following:

include ez_setup.py
include ah_bootstrap.py

# the next few stanzas are for astropy_helpers.  It's derived from the
# astropy_helpers/MANIFEST.in, but requires additional includes for the actual
# package directory and egg-info.

include astropy_helpers/README.rst
include astropy_helpers/CHANGES.rst
include astropy_helpers/LICENSE.rst
recursive-include astropy_helpers/licenses *

include astropy_helpers/ez_setup.py
include astropy_helpers/ah_bootstrap.py

recursive-include astropy_helpers/astropy_helpers *.py *.pyx *.c *.h
recursive-include astropy_helpers/astropy_helpers.egg-info *
# include the sphinx stuff with "*" because there are css/html/rst/etc.
recursive-include astropy_helpers/astropy_helpers/sphinx *

prune astropy_helpers/build
prune astropy_helpers/astropy_helpers/tests

Then add a new line near the top with the following:

include pyproject.toml

Step 11 - Updating your documentation configuration¶

You will need to edit the docs/conf.py file to make sure it does not use astropy-helpers. If you see a code block such as:

 try:
    import astropy_helpers
 except ImportError:
    # Building from inside the docs/ directory?
    if os.path.basename(os.getcwd()) == 'docs':
        a_h_path = os.path.abspath(os.path.join('..', 'astropy_helpers'))
        if os.path.isdir(a_h_path):
            sys.path.insert(1, a_h_path)

# Load all of the global Astropy configuration
from astropy_helpers.sphinx.conf import *

# Get configuration information from setup.cfg
try:
    from ConfigParser import ConfigParser
except ImportError:
    from configparser import ConfigParser

you should change it to:

try:
    from sphinx_astropy.conf.v1 import *  # noqa
except ImportError:
    print('ERROR: the documentation requires the sphinx-astropy package to be installed')
    sys.exit(1)

# Get configuration information from setup.cfg
from configparser import ConfigParser
conf = ConfigParser()

Find and replace any instances of package_name in the file with name.

Step 12 - Setting up tox¶

tox is a tool for automating commands, which is well suited to, e.g., running tests for your package or building the documentation. One of the benefits of using tox is that it will (by default) create a source distribution for your package and install it into a virtual environment before running tests or building docs, which means that it will be a good test of whether, e.g., you have declared the package data correctly.

As a starting point, copy over the tox.ini file generated in Step 0. You can always then customize it if needed (although it should work out of the box).

Once you have done this you should be able to do the following:

Run tests with minimal dependencies:

tox -e test

Run tests with astropy LTS and Numpy 1.16:

tox -e test-astropylts-numpy116

Run tests with all optional dependencies:

tox -e test-alldeps

Run tests with minimal dependencies and the latest developer version of numpy and astropy:

tox -e test-devdeps

Build the documentation:

tox -e build_docs

Run code style checks on your code:

tox -e codestyle

The {posargs} corresponds to arguments passed to tox after a -- separator - for example to make pytest verbose in a test environment, you can do:

tox -e test -- -v

Step 13 - Updating your Continuous Integration¶

This step will depend on what continuous integration services you use. Broadly speaking, unless there are dependencies you need that can only be installed with conda, you should no longer need to use ci-helpers to install these. The recommended approach is to use the tox file to set up the different configurations you want to use, and to then keep the CI configuration as simple as possible.

GitHub now has an integrated CI service, GitHub Actions. If you wish to use Actions, a good place to start is the .github/workflows/ci_tests.yml file generated in Step 0. You can then see if any previous customizations you had made need to be copied over. This file shows how one can configure Actions to use tox to test different environments with different versions of Python on different platforms.

Step 14 - Update ReadTheDocs configuration¶

With the set-up described in this migration guide, you should be able to simplify the configuration for ReadTheDocs. This can be done via a .readthedocs.yml or readthedocs.yml file (the former is recommended). See the ReadTheDocs documentation for more information about this file.

You should be able to copy over the .readthedocs.yml file created in Step 0. With this updated file, you should now be able to remove any pip requirements file or conda YAML file that were previously used by readthedocs.yml.

Step 15 - Coverage configuration¶

Preivously, astropy-helpers expected the coverage configuration to be located in my_package/tests/coveragerc. This is now no longer necessary, so you can now define the coverage configuration inside the setup.cfg file, which should help reduce the number of files to keep track of. Add the following to the bottom of your setup.cfg:

[coverage:run]
omit =
    my_package/_astropy_init*
    my_package/conftest.py
    my_package/*setup_package*
    my_package/tests/*
    my_package/*/tests/*
    my_package/extern/*
    my_package/version*
    */my_package/_astropy_init*
    */my_package/conftest.py
    */my_package/*setup_package*
    */my_package/tests/*
    */my_package/*/tests/*
    */my_package/extern/*
    */my_package/version*

[coverage:report]
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover
    # Don't complain about packages we have installed
    except ImportError
    # Don't complain if tests don't hit assertions
    raise AssertionError
    raise NotImplementedError
    # Don't complain about script hooks
    def main\(.*\):
    # Ignore branches that don't pertain to this version of Python
    pragma: py{ignore_python_version}
    # Don't complain about IPython completion helper
    def _ipython_key_completions_

Make sure to replace my_package by your module name. If you had any customizations in coveragerc you can include them here, but otherwise the above should be sufficient and you should now be able to remove the old file:

git rm my_package/tests/coveragerc

Step 16 - conftest.py file updates¶

For the header in your test runs to be correct with the latest versions of astropy, you will need to make sure that you update your conftest.py file as described in the pytest-astropy-header instructions. You can also copy over the file created in Step 0 and add back any customizations you had.

Step 17 - Final cleanup¶

Once you’ve made the above changes, you should be able to remove the following sections from your setup.cfg file:

[build_docs]
[build_sphinx]
[upload_docs]

You should also add pip-wheel-metadata/ to your .gitignore file.

Once you are done, if you would like us to help by reviewing your changes, you can open a pull request to your package and mention @astrofrog or @Cadair to ask for a review.

Note on releasing your package¶

As a result of the changes above, there are some tweaks to the procedure to follow for releasing your package - see the latest instructions in the astropy documentation. The two main changes are that you no longer need to manually update the version number in files, instead the version number is based on the latest git tag, and in addition the source file should be built using the pep517 package.

Note on conda recipes and pyproject.toml¶

While not something you can do until you release your updated package, you will need to take care to update conda recipes (e.g., in conda-forge) for your package. In particular, since conda ignores pyproject.toml files, you will need to make sure that the build dependencies present in pyproject.toml are explicitly listed as build dependencies in the conda recipe. For packages with compiled extensions and Cython, this would look like:

build:
    - {{ compiler('c') }}
host:
    - pip
    - python
    - setuptools
    - setuptools_scm
    - numpy
    - cython
    - extension-helpers
run:
    ...

while for pure-Python packages you will still need to make sure setuptools_scm is included in the build dependencies:

host:
    - pip
    - python
    - setuptools
    - setuptools_scm
run:
    ...

Navigation