NumPy 1.21.0 Release Notes#

The NumPy 1.21.0 release highlights are

  • continued SIMD work covering more functions and platforms,

  • initial work on the new dtype infrastructure and casting,

  • universal2 wheels for Python 3.8 and Python 3.9 on Mac,

  • improved documentation,

  • improved annotations,

  • new PCG64DXSM bitgenerator for random numbers.

In addition there are the usual large number of bug fixes and other improvements.

The Python versions supported for this release are 3.7-3.9. Official support for Python 3.10 will be added when it is released.

Warning

There are unresolved problems compiling NumPy 1.20.0 with gcc-11.1.

  • Optimization level -O3 results in many incorrect warnings when running the tests.

  • On some hardware NumPY will hang in an infinite loop.

New functions#

Add PCG64DXSM BitGenerator#

Uses of the PCG64 BitGenerator in a massively-parallel context have been shown to have statistical weaknesses that were not apparent at the first release in numpy 1.17. Most users will never observe this weakness and are safe to continue to use PCG64. We have introduced a new PCG64DXSM BitGenerator that will eventually become the new default BitGenerator implementation used by default_rng in future releases. PCG64DXSM solves the statistical weakness while preserving the performance and the features of PCG64.

See Upgrading PCG64 with PCG64DXSM for more details.

(gh-18906)

Expired deprecations#

  • The shape argument unravel_index cannot be passed as dims keyword argument anymore. (Was deprecated in NumPy 1.16.)

    (gh-17900)

  • The function PyUFunc_GenericFunction has been disabled. It was deprecated in NumPy 1.19. Users should call the ufunc directly using the Python API.

    (gh-18697)

  • The function PyUFunc_SetUsesArraysAsData has been disabled. It was deprecated in NumPy 1.19.

    (gh-18697)

  • The class PolyBase has been removed (deprecated in numpy 1.9.0). Please use the abstract ABCPolyBase class instead.

    (gh-18963)

  • The unused PolyError and PolyDomainError exceptions are removed.

    (gh-18963)

Deprecations#

The .dtype attribute must return a dtype#

A DeprecationWarning is now given if the .dtype attribute of an object passed into np.dtype or as a dtype=obj argument is not a dtype. NumPy will stop attempting to recursively coerce the result of .dtype.

(gh-13578)

Inexact matches for numpy.convolve and numpy.correlate are deprecated#

convolve and correlate now emit a warning when there are case insensitive and/or inexact matches found for mode argument in the functions. Pass full "same", "valid", "full" strings instead of "s", "v", "f" for the mode argument.

(gh-17492)

np.typeDict has been formally deprecated#

np.typeDict is a deprecated alias for np.sctypeDict and has been so for over 14 years (6689502). A deprecation warning will now be issued whenever getting np.typeDict.

(gh-17586)

Exceptions will be raised during array-like creation#

When an object raised an exception during access of the special attributes __array__ or __array_interface__, this exception was usually ignored. A warning is now given when the exception is anything but AttributeError. To silence the warning, the type raising the exception has to be adapted to raise an AttributeError.

(gh-19001)

Four ndarray.ctypes methods have been deprecated#

Four methods of the ndarray.ctypes object have been deprecated, as they are (undocumentated) implementation artifacts of their respective properties.

The methods in question are:

  • _ctypes.get_data (use _ctypes.data instead)

  • _ctypes.get_shape (use _ctypes.shape instead)

  • _ctypes.get_strides (use _ctypes.strides instead)

  • _ctypes.get_as_parameter (use _ctypes._as_parameter_ instead)

(gh-19031)

Expired deprecations#

  • The shape argument numpy.unravel_index cannot be passed as dims keyword argument anymore. (Was deprecated in NumPy 1.16.)

    (gh-17900)

  • The function PyUFunc_GenericFunction has been disabled. It was deprecated in NumPy 1.19. Users should call the ufunc directly using the Python API.

    (gh-18697)

  • The function PyUFunc_SetUsesArraysAsData has been disabled. It was deprecated in NumPy 1.19.

    (gh-18697)

Remove deprecated PolyBase and unused PolyError and PolyDomainError#

The class PolyBase has been removed (deprecated in numpy 1.9.0). Please use the abstract ABCPolyBase class instead.

Furthermore, the unused PolyError and PolyDomainError exceptions are removed from the numpy.polynomial.

(gh-18963)

Compatibility notes#

Error type changes in universal functions#

The universal functions may now raise different errors on invalid input in some cases. The main changes should be that a RuntimeError was replaced with a more fitting TypeError. When multiple errors were present in the same call, NumPy may now raise a different one.

(gh-15271)

__array_ufunc__ argument validation#

NumPy will now partially validate arguments before calling __array_ufunc__. Previously, it was possible to pass on invalid arguments (such as a non-existing keyword argument) when dispatch was known to occur.

(gh-15271)

__array_ufunc__ and additional positional arguments#

Previously, all positionally passed arguments were checked for __array_ufunc__ support. In the case of reduce, accumulate, and reduceat all arguments may be passed by position. This means that when they were passed by position, they could previously have been asked to handle the ufunc call via __array_ufunc__. Since this depended on the way the arguments were passed (by position or by keyword), NumPy will now only dispatch on the input and output array. For example, NumPy will never dispatch on the where array in a reduction such as np.add.reduce.

(gh-15271)

Validate input values in Generator.uniform#

Checked that high - low >= 0 in np.random.Generator.uniform. Raises ValueError if low > high. Previously out-of-order inputs were accepted and silently swapped, so that if low > high, the value generated was high + (low - high) * random().

(gh-17921)

/usr/include removed from default include paths#

The default include paths when building a package with numpy.distutils no longer include /usr/include. This path is normally added by the compiler, and hardcoding it can be problematic. In case this causes a problem, please open an issue. A workaround is documented in PR 18658.

(gh-18658)

Changes to comparisons with dtype=...#

When the dtype= (or signature) arguments to comparison ufuncs (equal, less, etc.) is used, this will denote the desired output dtype in the future. This means that:

np.equal(2, 3, dtype=object)

will give a FutureWarning that it will return an object array in the future, which currently happens for:

np.equal(None, None, dtype=object)

due to the fact that np.array(None) is already an object array. (This also happens for some other dtypes.)

Since comparisons normally only return boolean arrays, providing any other dtype will always raise an error in the future and give a DeprecationWarning now.

(gh-18718)

Changes to dtype and signature arguments in ufuncs#

The universal function arguments dtype and signature which are also valid for reduction such as np.add.reduce (which is the implementation for np.sum) will now issue a warning when the dtype provided is not a “basic” dtype.

NumPy almost always ignored metadata, byteorder or time units on these inputs. NumPy will now always ignore it and raise an error if byteorder or time unit changed. The following are the most important examples of changes which will give the error. In some cases previously the information stored was not ignored, in all of these an error is now raised:

# Previously ignored the byte-order (affect if non-native)
np.add(3, 5, dtype=">i32")

# The biggest impact is for timedelta or datetimes:
arr = np.arange(10, dtype="m8[s]")
# The examples always ignored the time unit "ns":
np.add(arr, arr, dtype="m8[ns]")
np.maximum.reduce(arr, dtype="m8[ns]")

# The following previously did use "ns" (as opposed to `arr.dtype`)
np.add(3, 5, dtype="m8[ns]")  # Now return generic time units
np.maximum(arr, arr, dtype="m8[ns]")  # Now returns "s" (from `arr`)

The same applies for functions like np.sum which use these internally. This change is necessary to achieve consistent handling within NumPy.

If you run into these, in most cases pass for example dtype=np.timedelta64 which clearly denotes a general timedelta64 without any unit or byte-order defined. If you need to specify the output dtype precisely, you may do so by either casting the inputs or providing an output array using out=.

NumPy may choose to allow providing an exact output dtype here in the future, which would be preceded by a FutureWarning.

(gh-18718)

Ufunc signature=... and dtype= generalization and casting#

The behaviour for np.ufunc(1.0, 1.0, signature=...) or np.ufunc(1.0, 1.0, dtype=...) can now yield different loops in 1.21 compared to 1.20 because of changes in promotion. When signature was previously used, the casting check on inputs was relaxed, which could lead to downcasting inputs unsafely especially if combined with casting="unsafe".

Casting is now guaranteed to be safe. If a signature is only partially provided, for example using signature=("float64", None, None), this could lead to no loop being found (an error). In that case, it is necessary to provide the complete signature to enforce casting the inputs. If dtype="float64" is used or only outputs are set (e.g. signature=(None, None, "float64") the is unchanged. We expect that very few users are affected by this change.

Further, the meaning of dtype="float64" has been slightly modified and now strictly enforces only the correct output (and not input) DTypes. This means it is now always equivalent to:

signature=(None, None, "float64")

(If the ufunc has two inputs and one output). Since this could lead to no loop being found in some cases, NumPy will normally also search for the loop:

signature=("float64", "float64", "float64")

if the first search failed. In the future, this behaviour may be customized to achieve the expected results for more complex ufuncs. (For some universal functions such as np.ldexp inputs can have different DTypes.)

(gh-18880)

Distutils forces strict floating point model on clang#

NumPy distutils will now always add the -ffp-exception-behavior=strict compiler flag when compiling with clang. Clang defaults to a non-strict version, which allows the compiler to generate code that does not set floating point warnings/errors correctly.

(gh-19049)

C API changes#

Use of ufunc->type_resolver and “type tuple”#

NumPy now normalizes the “type tuple” argument to the type resolver functions before calling it. Note that in the use of this type resolver is legacy behaviour and NumPy will not do so when possible. Calling ufunc->type_resolver or PyUFunc_DefaultTypeResolver is strongly discouraged and will now enforce a normalized type tuple if done. Note that this does not affect providing a type resolver, which is expected to keep working in most circumstances. If you have an unexpected use-case for calling the type resolver, please inform the NumPy developers so that a solution can be found.

(gh-18718)

New Features#

Added a mypy plugin for handling platform-specific numpy.number precisions#

A mypy plugin is now available for automatically assigning the (platform-dependent) precisions of certain number subclasses, including the likes of int_, intp and longlong. See the documentation on scalar types for a comprehensive overview of the affected classes.

Note that while usage of the plugin is completely optional, without it the precision of above-mentioned classes will be inferred as Any.

To enable the plugin, one must add it to their mypy configuration file:

[mypy]
plugins = numpy.typing.mypy_plugin

(gh-17843)

Let the mypy plugin manage extended-precision numpy.number subclasses#

The mypy plugin, introduced in numpy/numpy#17843, has been expanded: the plugin now removes annotations for platform-specific extended-precision types that are not available to the platform in question. For example, it will remove float128 when not available.

Without the plugin all extended-precision types will, as far as mypy is concerned, be available on all platforms.

To enable the plugin, one must add it to their mypy configuration file:

[mypy]
plugins = numpy.typing.mypy_plugin

(gh-18322)

New min_digits argument for printing float values#

A new min_digits argument has been added to the dragon4 float printing functions format_float_positional and format_float_scientific . This kwd guarantees that at least the given number of digits will be printed when printing in unique=True mode, even if the extra digits are unnecessary to uniquely specify the value. It is the counterpart to the precision argument which sets the maximum number of digits to be printed. When unique=False in fixed precision mode, it has no effect and the precision argument fixes the number of digits.

(gh-18629)

f2py now recognizes Fortran abstract interface blocks#

f2py can now parse abstract interface blocks.

(gh-18695)

BLAS and LAPACK configuration via environment variables#

Autodetection of installed BLAS and LAPACK libraries can be bypassed by using the NPY_BLAS_LIBS and NPY_LAPACK_LIBS environment variables. Instead, the link flags in these environment variables will be used directly, and the language is assumed to be F77. This is especially useful in automated builds where the BLAS and LAPACK that are installed are known exactly. A use case is replacing the actual implementation at runtime via stub library links.

If NPY_CBLAS_LIBS is set (optional in addition to NPY_BLAS_LIBS), this will be used as well, by defining HAVE_CBLAS and appending the environment variable content to the link flags.

(gh-18737)

A runtime-subcriptable alias has been added for ndarray#

numpy.typing.NDArray has been added, a runtime-subscriptable alias for np.ndarray[Any, np.dtype[~Scalar]]. The new type alias can be used for annotating arrays with a given dtype and unspecified shape. 1

1 NumPy does not support the annotating of array shapes as of 1.21, this is expected to change in the future though (see PEP 646).

Examples#

>>> import numpy as np
>>> import numpy.typing as npt

>>> print(npt.NDArray)
numpy.ndarray[typing.Any, numpy.dtype[~ScalarType]]

>>> print(npt.NDArray[np.float64])
numpy.ndarray[typing.Any, numpy.dtype[numpy.float64]]

>>> NDArrayInt = npt.NDArray[np.int_]
>>> a: NDArrayInt = np.arange(10)

>>> def func(a: npt.ArrayLike) -> npt.NDArray[Any]:
...     return np.array(a)

(gh-18935)

Improvements#

Arbitrary period option for numpy.unwrap#

The size of the interval over which phases are unwrapped is no longer restricted to 2 * pi. This is especially useful for unwrapping degrees, but can also be used for other intervals.

>>> phase_deg = np.mod(np.linspace(0,720,19), 360) - 180
>>> phase_deg
array([-180., -140., -100.,  -60.,  -20.,   20.,   60.,  100.,  140.,
       -180., -140., -100.,  -60.,  -20.,   20.,   60.,  100.,  140.,
       -180.])

>>> unwrap(phase_deg, period=360)
array([-180., -140., -100.,  -60.,  -20.,   20.,   60.,  100.,  140.,
        180.,  220.,  260.,  300.,  340.,  380.,  420.,  460.,  500.,
        540.])

(gh-16987)

np.unique now returns single NaN#

When np.unique operated on an array with multiple NaN entries, its return included a NaN for each entry that was NaN in the original array. This is now improved such that the returned array contains just one NaN as the last element.

Also for complex arrays all NaN values are considered equivalent (no matter whether the NaN is in the real or imaginary part). As the representant for the returned array the smallest one in the lexicographical order is chosen - see np.sort for how the lexicographical order is defined for complex arrays.

(gh-18070)

Generator.rayleigh and Generator.geometric performance improved#

The performance of Rayleigh and geometric random variate generation in Generator has improved. These are both transformation of exponential random variables and the slow log-based inverse cdf transformation has been replaced with the Ziggurat-based exponential variate generator.

This change breaks the stream of variates generated when variates from either of these distributions are produced.

(gh-18666)

Placeholder annotations have been improved#

All placeholder annotations, that were previously annotated as typing.Any, have been improved. Where appropriate they have been replaced with explicit function definitions, classes or other miscellaneous objects.

(gh-18934)

Performance improvements#

Improved performance in integer division of NumPy arrays#

Integer division of NumPy arrays now uses libdivide when the divisor is a constant. With the usage of libdivide and other minor optimizations, there is a large speedup. The // operator and np.floor_divide makes use of the new changes.

(gh-17727)

Improve performance of np.save and np.load for small arrays#

np.save is now a lot faster for small arrays.

np.load is also faster for small arrays, but only when serializing with a version >= (3, 0).

Both are done by removing checks that are only relevant for Python 2, while still maintaining compatibility with arrays which might have been created by Python 2.

(gh-18657)

Changes#

numpy.piecewise output class now matches the input class#

When ndarray subclasses are used on input to piecewise, they are passed on to the functions. The output will now be of the same subclass as well.

(gh-18110)

Enable Accelerate Framework#

With the release of macOS 11.3, several different issues that numpy was encountering when using Accelerate Framework’s implementation of BLAS and LAPACK should be resolved. This change enables the Accelerate Framework as an option on macOS. If additional issues are found, please file a bug report against Accelerate using the developer feedback assistant tool (https://developer.apple.com/bug-reporting/). We intend to address issues promptly and plan to continue supporting and updating our BLAS and LAPACK libraries.

(gh-18874)