Packaging (numpy.distutils
)#
Warning
numpy.distutils
is deprecated, and will be removed for
Python >= 3.12. For more details, see Status of numpy.distutils and migration advice
Warning
Note that setuptools
does major releases often and those may contain
changes that break numpy.distutils
, which will not be updated anymore
for new setuptools
versions. It is therefore recommended to set an
upper version bound in your build configuration for the last known version
of setuptools
that works with your build.
NumPy provides enhanced distutils functionality to make it easier to
build and install sub-packages, auto-generate code, and extension
modules that use Fortran-compiled libraries. A useful Configuration
class is also provided in
numpy.distutils.misc_util
that can make it easier to construct
keyword arguments to pass to the setup function (by passing the
dictionary obtained from the todict() method of the class). More
information is available in the numpy.distutils user guide.
The choice and location of linked libraries such as BLAS and LAPACK as well as
include paths and other such build options can be specified in a site.cfg
file located in the NumPy root repository or a .numpy-site.cfg
file in your
home directory. See the site.cfg.example
example file included in the NumPy
repository or sdist for documentation.
Modules in numpy.distutils
#
- distutils.misc_util
all_strings
allpath
appendpath
as_list
blue_text
cyan_text
cyg2win32
default_config_dict
dict_append
dot_join
exec_mod_from_location
filter_sources
generate_config_py
get_build_architecture
get_cmd
get_data_files
get_dependencies
get_ext_source_files
get_frame
get_info
get_language
get_lib_source_files
get_mathlibs
get_num_build_jobs
get_numpy_include_dirs
get_pkg_info
get_script_files
gpaths
green_text
has_cxx_sources
has_f_sources
is_local_src_dir
is_sequence
is_string
mingw32
minrelpath
njoin
red_text
sanitize_cxx_flags
terminal_has_colors
yellow_text
Provides the CCompilerOpt class, used for handling the CPU/hardware optimization, starting from parsing the command arguments, to managing the relation between the CPU baseline and dispatch-able features, also generating the required C headers and ending with compiling the sources with proper compiler's flags. |
|
|
|
exec_command |
|
|
|
|
notfound_action: |
Returns a list of files named 'fname' from 1) System-wide directory (directory-location of this module) 2) Users HOME directory (os.environ['HOME']) 3) Local directory |
Configuration class#
- class numpy.distutils.misc_util.Configuration(package_name=None, parent_name=None, top_path=None, package_path=None, **attrs)[source]#
Construct a configuration instance for the given package name. If parent_name is not None, then construct the package as a sub-package of the parent_name package. If top_path and package_path are None then they are assumed equal to the path of the file this instance was created in. The setup.py files in the numpy distribution are good examples of how to use the
Configuration
instance.- todict()[source]#
Return a dictionary compatible with the keyword arguments of distutils setup function.
Examples
>>> setup(**config.todict())
- get_subpackage(subpackage_name, subpackage_path=None, parent_name=None, caller_level=1)[source]#
Return list of subpackage configurations.
- Parameters:
- subpackage_namestr or None
Name of the subpackage to get the configuration. ‘*’ in subpackage_name is handled as a wildcard.
- subpackage_pathstr
If None, then the path is assumed to be the local path plus the subpackage_name. If a setup.py file is not found in the subpackage_path, then a default configuration is used.
- parent_namestr
Parent name.
- add_subpackage(subpackage_name, subpackage_path=None, standalone=False)[source]#
Add a sub-package to the current Configuration instance.
This is useful in a setup.py script for adding sub-packages to a package.
- Parameters:
- subpackage_namestr
name of the subpackage
- subpackage_pathstr
if given, the subpackage path such as the subpackage is in subpackage_path / subpackage_name. If None,the subpackage is assumed to be located in the local path / subpackage_name.
- standalonebool
- add_data_files(*files)[source]#
Add data files to configuration data_files.
- Parameters:
- filessequence
Argument(s) can be either
2-sequence (<datadir prefix>,<path to data file(s)>)
paths to data files where python datadir prefix defaults to package dir.
Notes
The form of each element of the files sequence is very flexible allowing many combinations of where to get the files from the package and where they should ultimately be installed on the system. The most basic usage is for an element of the files argument sequence to be a simple filename. This will cause that file from the local path to be installed to the installation path of the self.name package (package path). The file argument can also be a relative path in which case the entire relative path will be installed into the package directory. Finally, the file can be an absolute path name in which case the file will be found at the absolute path name but installed to the package path.
This basic behavior can be augmented by passing a 2-tuple in as the file argument. The first element of the tuple should specify the relative path (under the package install directory) where the remaining sequence of files should be installed to (it has nothing to do with the file-names in the source distribution). The second element of the tuple is the sequence of files that should be installed. The files in this sequence can be filenames, relative paths, or absolute paths. For absolute paths the file will be installed in the top-level package installation directory (regardless of the first argument). Filenames and relative path names will be installed in the package install directory under the path name given as the first element of the tuple.
Rules for installation paths:
file.txt -> (., file.txt)-> parent/file.txt
foo/file.txt -> (foo, foo/file.txt) -> parent/foo/file.txt
/foo/bar/file.txt -> (., /foo/bar/file.txt) -> parent/file.txt
*
.txt -> parent/a.txt, parent/b.txtfoo/
*
.txt`` -> parent/foo/a.txt, parent/foo/b.txt*/*.txt
-> (*
,*
/*
.txt) -> parent/c/a.txt, parent/d/b.txt(sun, file.txt) -> parent/sun/file.txt
(sun, bar/file.txt) -> parent/sun/file.txt
(sun, /foo/bar/file.txt) -> parent/sun/file.txt
(sun,
*
.txt) -> parent/sun/a.txt, parent/sun/b.txt(sun, bar/
*
.txt) -> parent/sun/a.txt, parent/sun/b.txt(sun/
*
,*
/*
.txt) -> parent/sun/c/a.txt, parent/d/b.txt
An additional feature is that the path to a data-file can actually be a function that takes no arguments and returns the actual path(s) to the data-files. This is useful when the data files are generated while building the package.
Examples
Add files to the list of data_files to be included with the package.
>>> self.add_data_files('foo.dat', ... ('fun', ['gun.dat', 'nun/pun.dat', '/tmp/sun.dat']), ... 'bar/cat.dat', ... '/full/path/to/can.dat')
will install these data files to:
<package install directory>/ foo.dat fun/ gun.dat nun/ pun.dat sun.dat bar/ car.dat can.dat
where <package install directory> is the package (or sub-package) directory such as ‘/usr/lib/python2.4/site-packages/mypackage’ (‘C: Python2.4 Lib site-packages mypackage’) or ‘/usr/lib/python2.4/site- packages/mypackage/mysubpackage’ (‘C: Python2.4 Lib site-packages mypackage mysubpackage’).
- add_data_dir(data_path)[source]#
Recursively add files under data_path to data_files list.
Recursively add files under data_path to the list of data_files to be installed (and distributed). The data_path can be either a relative path-name, or an absolute path-name, or a 2-tuple where the first argument shows where in the install directory the data directory should be installed to.
- Parameters:
- data_pathseq or str
Argument can be either
2-sequence (<datadir suffix>, <path to data directory>)
path to data directory where python datadir suffix defaults to package dir.
Notes
Rules for installation paths:
foo/bar -> (foo/bar, foo/bar) -> parent/foo/bar (gun, foo/bar) -> parent/gun foo/* -> (foo/a, foo/a), (foo/b, foo/b) -> parent/foo/a, parent/foo/b (gun, foo/*) -> (gun, foo/a), (gun, foo/b) -> gun (gun/*, foo/*) -> parent/gun/a, parent/gun/b /foo/bar -> (bar, /foo/bar) -> parent/bar (gun, /foo/bar) -> parent/gun (fun/*/gun/*, sun/foo/bar) -> parent/fun/foo/gun/bar
Examples
For example suppose the source directory contains fun/foo.dat and fun/bar/car.dat:
>>> self.add_data_dir('fun') >>> self.add_data_dir(('sun', 'fun')) >>> self.add_data_dir(('gun', '/full/path/to/fun'))
Will install data-files to the locations:
<package install directory>/ fun/ foo.dat bar/ car.dat sun/ foo.dat bar/ car.dat gun/ foo.dat car.dat
- add_include_dirs(*paths)[source]#
Add paths to configuration include directories.
Add the given sequence of paths to the beginning of the include_dirs list. This list will be visible to all extension modules of the current package.
- add_headers(*files)[source]#
Add installable headers to configuration.
Add the given sequence of files to the beginning of the headers list. By default, headers will be installed under <python- include>/<self.name.replace(‘.’,’/’)>/ directory. If an item of files is a tuple, then its first argument specifies the actual installation location relative to the <python-include> path.
- Parameters:
- filesstr or seq
Argument(s) can be either:
2-sequence (<includedir suffix>,<path to header file(s)>)
path(s) to header file(s) where python includedir suffix will default to package name.
- add_extension(name, sources, **kw)[source]#
Add extension to configuration.
Create and add an Extension instance to the ext_modules list. This method also takes the following optional keyword arguments that are passed on to the Extension constructor.
- Parameters:
- namestr
name of the extension
- sourcesseq
list of the sources. The list of sources may contain functions (called source generators) which must take an extension instance and a build directory as inputs and return a source file or list of source files or None. If None is returned then no sources are generated. If the Extension instance has no sources after processing all source generators, then no extension module is built.
- include_dirs
- define_macros
- undef_macros
- library_dirs
- libraries
- runtime_library_dirs
- extra_objects
- extra_compile_args
- extra_link_args
- extra_f77_compile_args
- extra_f90_compile_args
- export_symbols
- swig_opts
- depends
The depends list contains paths to files or directories that the sources of the extension module depend on. If any path in the depends list is newer than the extension module, then the module will be rebuilt.
- language
- f2py_options
- module_dirs
- extra_infodict or list
dict or list of dict of keywords to be appended to keywords.
Notes
The self.paths(…) method is applied to all lists that may contain paths.
- add_library(name, sources, **build_info)[source]#
Add library to configuration.
- Parameters:
- namestr
Name of the extension.
- sourcessequence
List of the sources. The list of sources may contain functions (called source generators) which must take an extension instance and a build directory as inputs and return a source file or list of source files or None. If None is returned then no sources are generated. If the Extension instance has no sources after processing all source generators, then no extension module is built.
- build_infodict, optional
The following keys are allowed:
depends
macros
include_dirs
extra_compiler_args
extra_f77_compile_args
extra_f90_compile_args
f2py_options
language
- add_scripts(*files)[source]#
Add scripts to configuration.
Add the sequence of files to the beginning of the scripts list. Scripts will be installed under the <prefix>/bin/ directory.
- add_installed_library(name, sources, install_dir, build_info=None)[source]#
Similar to add_library, but the specified library is installed.
Most C libraries used with
distutils
are only used to build python extensions, but libraries built through this method will be installed so that they can be reused by third-party packages.- Parameters:
- namestr
Name of the installed library.
- sourcessequence
List of the library’s source files. See
add_library
for details.- install_dirstr
Path to install the library, relative to the current sub-package.
- build_infodict, optional
The following keys are allowed:
depends
macros
include_dirs
extra_compiler_args
extra_f77_compile_args
extra_f90_compile_args
f2py_options
language
- Returns:
- None
See also
Notes
The best way to encode the options required to link against the specified C libraries is to use a “libname.ini” file, and use
get_info
to retrieve the required options (seeadd_npy_pkg_config
for more information).
- add_npy_pkg_config(template, install_dir, subst_dict=None)[source]#
Generate and install a npy-pkg config file from a template.
The config file generated from template is installed in the given install directory, using subst_dict for variable substitution.
- Parameters:
- templatestr
The path of the template, relatively to the current package path.
- install_dirstr
Where to install the npy-pkg config file, relatively to the current package path.
- subst_dictdict, optional
If given, any string of the form
@key@
will be replaced bysubst_dict[key]
in the template file when installed. The install prefix is always available through the variable@prefix@
, since the install prefix is not easy to get reliably from setup.py.
See also
Notes
This works for both standard installs and in-place builds, i.e. the
@prefix@
refer to the source directory for in-place builds.Examples
config.add_npy_pkg_config('foo.ini.in', 'lib', {'foo': bar})
Assuming the foo.ini.in file has the following content:
[meta] Name=@foo@ Version=1.0 Description=dummy description [default] Cflags=-I@prefix@/include Libs=
The generated file will have the following content:
[meta] Name=bar Version=1.0 Description=dummy description [default] Cflags=-Iprefix_dir/include Libs=
and will be installed as foo.ini in the ‘lib’ subpath.
When cross-compiling with numpy distutils, it might be necessary to use modified npy-pkg-config files. Using the default/generated files will link with the host libraries (i.e. libnpymath.a). For cross-compilation you of-course need to link with target libraries, while using the host Python installation.
You can copy out the numpy/_core/lib/npy-pkg-config directory, add a pkgdir value to the .ini files and set NPY_PKG_CONFIG_PATH environment variable to point to the directory with the modified npy-pkg-config files.
Example npymath.ini modified for cross-compilation:
[meta] Name=npymath Description=Portable, core math library implementing C99 standard Version=0.1 [variables] pkgname=numpy._core pkgdir=/build/arm-linux-gnueabi/sysroot/usr/lib/python3.7/site-packages/numpy/_core prefix=${pkgdir} libdir=${prefix}/lib includedir=${prefix}/include [default] Libs=-L${libdir} -lnpymath Cflags=-I${includedir} Requires=mlib [msvc] Libs=/LIBPATH:${libdir} npymath.lib Cflags=/INCLUDE:${includedir} Requires=mlib
- paths(*paths, **kws)[source]#
Apply glob to paths and prepend local_path if needed.
Applies glob.glob(…) to each path in the sequence (if needed) and pre-pends the local_path if needed. Because this is called on all source lists, this allows wildcard characters to be specified in lists of sources for extension modules and libraries and scripts and allows path-names be relative to the source directory.
- get_build_temp_dir()[source]#
Return a path to a temporary directory where temporary files should be placed.
- have_f77c()[source]#
Check for availability of Fortran 77 compiler.
Use it inside source generating function to ensure that setup distribution instance has been initialized.
Notes
True if a Fortran 77 compiler is available (because a simple Fortran 77 code was able to be compiled successfully).
- have_f90c()[source]#
Check for availability of Fortran 90 compiler.
Use it inside source generating function to ensure that setup distribution instance has been initialized.
Notes
True if a Fortran 90 compiler is available (because a simple Fortran 90 code was able to be compiled successfully)
- get_version(version_file=None, version_variable=None)[source]#
Try to get version string of a package.
Return a version string of the current package or None if the version information could not be detected.
Notes
This method scans files named __version__.py, <packagename>_version.py, version.py, and __svn_version__.py for string variables version, __version__, and <packagename>_version, until a version number is found.
- make_svn_version_py(delete=True)[source]#
Appends a data function to the data_files list that will generate __svn_version__.py file to the current package directory.
Generate package __svn_version__.py file from SVN revision number, it will be removed after python exits but will be available when sdist, etc commands are executed.
Notes
If __svn_version__.py existed before, nothing is done.
This is intended for working with source directories that are in an SVN repository.
Building installable C libraries#
Conventional C libraries (installed through add_library) are not installed, and
are just used during the build (they are statically linked). An installable C
library is a pure C library, which does not depend on the python C runtime, and
is installed such that it may be used by third-party packages. To build and
install the C library, you just use the method add_installed_library instead of
add_library, which takes the same arguments except for an additional
install_dir
argument:
.. hidden in a comment so as to be included in refguide but not rendered documentation
>>> import numpy.distutils.misc_util
>>> config = np.distutils.misc_util.Configuration(None, '', '.')
>>> with open('foo.c', 'w') as f: pass
>>> config.add_installed_library('foo', sources=['foo.c'], install_dir='lib')
npy-pkg-config files#
To make the necessary build options available to third parties, you could use
the npy-pkg-config mechanism implemented in numpy.distutils
. This mechanism is
based on a .ini file which contains all the options. A .ini file is very
similar to .pc files as used by the pkg-config unix utility:
[meta]
Name: foo
Version: 1.0
Description: foo library
[variables]
prefix = /home/user/local
libdir = ${prefix}/lib
includedir = ${prefix}/include
[default]
cflags = -I${includedir}
libs = -L${libdir} -lfoo
Generally, the file needs to be generated during the build, since it needs some
information known at build time only (e.g. prefix). This is mostly automatic if
one uses the Configuration
method add_npy_pkg_config. Assuming we have a
template file foo.ini.in as follows:
[meta]
Name: foo
Version: @version@
Description: foo library
[variables]
prefix = @prefix@
libdir = ${prefix}/lib
includedir = ${prefix}/include
[default]
cflags = -I${includedir}
libs = -L${libdir} -lfoo
and the following code in setup.py:
>>> config.add_installed_library('foo', sources=['foo.c'], install_dir='lib')
>>> subst = {'version': '1.0'}
>>> config.add_npy_pkg_config('foo.ini.in', 'lib', subst_dict=subst)
This will install the file foo.ini into the directory package_dir/lib, and the
foo.ini file will be generated from foo.ini.in, where each @version@
will be
replaced by subst_dict['version']
. The dictionary has an additional prefix
substitution rule automatically added, which contains the install prefix (since
this is not easy to get from setup.py).
Reusing a C library from another package#
Info are easily retrieved from the get_info
function in
numpy.distutils.misc_util
:
>>> info = np.distutils.misc_util.get_info('npymath')
>>> config.add_extension('foo', sources=['foo.c'], extra_info=info)
<numpy.distutils.extension.Extension('foo') at 0x...>
An additional list of paths to look for .ini files can be given to get_info
.
Conversion of .src
files#
NumPy distutils supports automatic conversion of source files named <somefile>.src. This facility can be used to maintain very similar code blocks requiring only simple changes between blocks. During the build phase of setup, if a template file named <somefile>.src is encountered, a new file named <somefile> is constructed from the template and placed in the build directory to be used instead. Two forms of template conversion are supported. The first form occurs for files named <file>.ext.src where ext is a recognized Fortran extension (f, f90, f95, f77, for, ftn, pyf). The second form is used for all other cases. See Conversion of .src files using templates.