Ctrl+K

NEP 26 — Summary of missing data NEPs and discussion#

Author:

Mark Wiebe <mwwiebe@gmail.com>, Nathaniel J. Smith <njs@pobox.com>

Status:

Deferred

Type:

Standards Track

Created:

2012-04-22

Context: this NEP was written as summary of the large number of discussions and proposals (NEP 12 — Missing data functionality in NumPy, NEP 24 — Missing data functionality - alternative 1 to NEP 12, NEP 25 — NA support via special dtypes), regarding missing data functionality.

The debate about how NumPy should handle missing data, a subject with many preexisting approaches, requirements, and conventions, has been long and contentious. There has been more than one proposal for how to implement support into NumPy, and there is a testable implementation which is merged into NumPy’s current main. The vast number of emails and differing points of view has made it difficult for interested parties to understand the issues and be comfortable with the direction NumPy is going.

Here is our (Mark and Nathaniel’s) attempt to summarize the problem, proposals, and points of agreement/disagreement in a single place, to help the community move towards consensus.

The NumPy developers’ problem#

For this discussion, “missing data” means array elements which can be indexed (e.g. A[3] in an array A with shape (5,)), but have, in some sense, no value.

It does not refer to compressed or sparse storage techniques where the value for A[3] is not actually stored in memory, but still has a well-defined value like 0.

This is still vague, and to create an actual implementation, it is necessary to answer such questions as:

  • What values are computed when doing element-wise ufuncs.

  • What values are computed when doing reductions.

  • Whether the storage for an element gets overwritten when marking that value missing.

  • Whether computations resulting in NaN automatically treat in the same way as a missing value.

  • Whether one interacts with missing values using a placeholder object (e.g. called “NA” or “masked”), or through a separate boolean array.

  • Whether there is such a thing as an array object that cannot hold missing array elements.

  • How the (C and Python) API is expressed, in terms of dtypes, masks, and other constructs.

  • If we decide to answer some of these questions in multiple ways, then that creates the question of whether that requires multiple systems, and if so how they should interact.

There’s clearly a very large space of missing-data APIs that could be implemented. There is likely at least one user, somewhere, who would find any possible implementation to be just the thing they need to solve some problem. On the other hand, much of NumPy’s power and clarity comes from having a small number of orthogonal concepts, such as strided arrays, flexible indexing, broadcasting, and ufuncs, and we’d like to preserve that simplicity.

There has been dissatisfaction among several major groups of NumPy users about the existing status quo of missing data support. In particular, neither the numpy.ma component nor use of floating-point NaNs as a missing data signal fully satisfy the performance requirements and ease of use for these users. The example of R, where missing data is treated via an NA placeholder and is deeply integrated into all computation, is where many of these users point to indicate what functionality they would like. Doing a deep integration of missing data like in R must be considered carefully, it must be clear it is not being done in a way which sacrifices existing performance or functionality.

Our problem is, how can we choose some incremental additions to NumPy that will make a large class of users happy, be reasonably elegant, complement the existing design, and that we’re comfortable we won’t regret being stuck with in the long term.

Prior art#

So a major (maybe the major) problem is figuring out how ambitious the project to add missing data support to NumPy should be, and which kinds of problems are in scope. Let’s start with the best understood situation where “missing data” comes into play:

“Statistical missing data”#

In statistics, social science, etc., “missing data” is a term of art referring to a specific (but extremely common and important) situation: we have tried to gather some measurements according to some scheme, but some of these measurements are missing. For example, if we have a table listing the height, age, and income of a number of individuals, but one person did not provide their income, then we need some way to represent this:

Person | Height | Age | Income
------------------------------
   1   |   63   | 25  | 15000
   2   |   58   | 32  | <missing>
   3   |   71   | 45  | 30000

The traditional way is to record that income as, say, “-99”, and document this in the README along with the data set. Then, you have to remember to check for and handle such incomes specially; if you forget, you’ll get superficially reasonable but completely incorrect results, like calculating the average income on this data set as 14967. If you’re in one of these fields, then such missing-ness is routine and inescapable, and if you use the “-99” approach then it’s a pitfall you have to remember to check for explicitly on literally every calculation you ever do. This is, obviously, an unpleasant way to live.

Let’s call this situation the “statistical missing data” situation, just to have a convenient handle for it. (As mentioned, practitioners just call this “missing data”, and what to do about it is literally an entire sub-field of statistics; if you google “missing data” then every reference is on how to handle it.) NumPy isn’t going to do automatic imputation or anything like that, but it could help a great deal by providing some standard way to at least represent data which is missing in this sense.

The main prior art for how this could be done comes from the S/S+/R family of languages. Their strategy is, for each type they support, to define a special value called “NA”. (For ints this is INT_MAX, for floats it’s a special NaN value that’s distinguishable from other NaNs, …) Then, they arrange that in computations, this value has a special semantics that we will call “NA semantics”.

NA semantics#

The idea of NA semantics is that any computations involving NA values should be consistent with what would have happened if we had known the correct value.

For example, let’s say we want to compute the mean income, how might we do this? One way would be to just ignore the missing entry, and compute the mean of the remaining entries. This gives us (15000 + 30000)/2, or 22500.

Is this result consistent with discovering the income of person 2? Let’s say we find out that person 2’s income is 50000. This means the correct answer is (15000 + 50000 + 30000)/3, or 31666.67, indicating clearly that it is not consistent. Therefore, the mean income is NA, i.e. a specific number whose value we are unable to compute.

This motivates the following rules, which are how R implements NA:

Assignment:

NA values are understood to represent specific unknown values, and thus should have value-like semantics with respect to assignment and other basic data manipulation operations. Code which does not actually look at the values involved should work the same regardless of whether some of them are missing. For example, one might write:

income[:] = income[np.argsort(height)]

to perform an in-place sort of the income array, and know that the shortest person’s income would end up being first. It turns out that the shortest person’s income is not known, so the array should end up being [NA, 15000, 30000], but there’s nothing special about NAness here.

Propagation:

In the example above, we concluded that an operation like mean should produce NA when one of its data values was NA. If you ask me, “what is 3 plus x?”, then my only possible answer is “I don’t know what x is, so I don’t know what 3 + x is either”. NA means “I don’t know”, so 3 + NA is NA.

This is important for safety when analyzing data: missing data often requires special handling for correctness – the fact that you are missing information might mean that something you wanted to compute cannot actually be computed, and there are whole books written on how to compensate in various situations. Plus, it’s easy to not realize that you have missing data, and write code that assumes you have all the data. Such code should not silently produce the wrong answer.

There is an important exception to characterizing this as propagation, in the case of boolean values. Consider the calculation:

v = np.any([False, False, NA, True])

If we strictly propagate, v will become NA. However, no matter whether we place True or False into the third array position, v will then get the value True. The answer to the question “Is the result True consistent with later discovering the value that was missing?” is yes, so it is reasonable to not propagate here, and instead return the value True. This is what R does:

> any(c(F, F, NA, T))
[1] TRUE
> any(c(F, F, NA, F))
[1] NA
Other:

NaN and NA are conceptually distinct. 0.0/0.0 is not a mysterious, unknown value – it’s defined to be NaN by IEEE floating point, Not a Number. NAs are numbers (or strings, or whatever), just unknown ones. Another small but important difference is that in Python, if NaN: ... treats NaN as True (NaN is “truthy”); but if NA: ... would be an error.

In R, all reduction operations implement an alternative semantics, activated by passing a special argument (na.rm=TRUE in R). sum(a) means “give me the sum of all the values” (which is NA if some of the values are NA); sum(a, na.rm=True) means “give me the sum of all the non-NA values”.

Other prior art#

Once we move beyond the “statistical missing data” case, the correct behavior for missing data becomes less clearly defined. There are many cases where specific elements are singled out to be treated specially or excluded from computations, and these could often be conceptualized as involving ‘missing data’ in some sense.

In image processing, it’s common to use a single image together with one or more boolean masks to e.g. composite subsets of an image. As Joe Harrington pointed out on the list, in the context of processing astronomical images, it’s also common to generalize to a floating-point valued mask, or alpha channel, to indicate degrees of “missingness”. We think this is out of scope for the present design, but it is an important use case, and ideally NumPy should support natural ways of manipulating such data.

After R, numpy.ma is probably the most mature source of experience on missing-data-related APIs. Its design is quite different from R; it uses different semantics – reductions skip masked values by default and NaNs convert to masked – and it uses a different storage strategy via a separate mask. While it seems to be generally considered sub-optimal for general use, it’s hard to pin down whether this is because the API is immature but basically good, or the API is fundamentally broken, or the API is great but the code should be faster, or what. We looked at some of those users to try and get a better idea.

Matplotlib is perhaps the best known package to rely on numpy.ma. It seems to use it in two ways. One is as a way for users to indicate what data is missing when passing it to be graphed. (Other ways are also supported, e.g., passing in NaN values gives the same result.) In this regard, matplotlib treats np.ma.masked and NaN values in the same way that R’s plotting routines handle NA and NaN values. For these purposes, matplotlib doesn’t really care what semantics or storage strategy is used for missing data.

Internally, matplotlib uses numpy.ma arrays to store and pass around separately computed boolean masks containing ‘validity’ information for each input array in a cheap and non-destructive fashion. Mark’s impression from some shallow code review is that mostly it works directly with the data and mask attributes of the masked arrays, not extensively using the particular computational semantics of numpy.ma. So, for this usage they do rely on the non-destructive mask-based storage, but this doesn’t say much about what semantics are needed.

Paul Hobson posted some code on the list that uses numpy.ma for storing arrays of contaminant concentration measurements. Here the mask indicates whether the corresponding number represents an actual measurement, or just the estimated detection limit for a concentration which was too small to detect. Nathaniel’s impression from reading through this code is that it also mostly uses the .data and .mask attributes in preference to performing operations on the MaskedArray directly.

So, these examples make it clear that there is demand for a convenient way to keep a data array and a mask array (or even a floating point array) bundled up together and “aligned”. But they don’t tell us much about what semantics the resulting object should have with respect to ufuncs and friends.

Semantics, storage, API, oh my!#

We think it’s useful to draw a clear line between use cases, semantics, and storage. Use cases are situations that users encounter, regardless of what NumPy does; they’re the focus of the previous section. When we say semantics, we mean the result of different operations as viewed from the Python level without regard to the underlying implementation.

NA semantics are the ones described above and used by R:

1 + NA = NA
sum([1, 2, NA]) = NA
NA | False = NA
NA | True = True

With na.rm=TRUE or skipNA=True, this switches to:

1 + NA = illegal # in R, only reductions take na.rm argument
sum([1, 2, NA], skipNA=True) = 3

There’s also been discussion of what we’ll call ignore semantics. These are somewhat underdefined:

sum([1, 2, IGNORED]) = 3
# Several options here:
1 + IGNORED = 1
#  or
1 + IGNORED = <leaves output array untouched>
#  or
1 + IGNORED = IGNORED

The numpy.ma semantics are:

sum([1, 2, masked]) = 3
1 + masked = masked

If either NA or ignore semantics are implemented with masks, then there is a choice of what should be done to the value in the storage for an array element which gets assigned a missing value. Three possibilities are:

  • Leave that memory untouched (the choice made in the NEP).

  • Do the calculation with the values independently of the mask (perhaps the most useful option for Paul Hobson’s use-case above).

  • Copy whatever value is stored behind the input missing value into the output (this is what numpy.ma does. Even that is ambiguous in the case of masked + masked – in this case numpy.ma copies the value stored behind the leftmost masked value).

When we talk about storage, we mean the debate about whether missing values should be represented by designating a particular value of the underlying data-type (the bitpattern dtype option, as used in R), or by using a separate mask stored alongside the data itself.

For mask-based storage, there is also an important question about what the API looks like for accessing the mask, modifying the mask, and “peeking behind” the mask.

Designs that have been proposed#

One option is to just copy R, by implementing a mechanism whereby dtypes can arrange for certain bitpatterns to be given NA semantics.

One option is to copy numpy.ma closely, but with a more optimized implementation. (Or to simply optimize the existing implementation.)

One option is that described in NEP12, for which an implementation of mask-based missing data exists. This system is roughly:

  • There is both bitpattern and mask-based missing data, and both have identical interoperable NA semantics.

  • Masks are modified by assigning np.NA or values to array elements. The way to peek behind the mask or to unmask values is to keep a view of the array that shares the data pointer but not the mask pointer.

  • Mark would like to add a way to access and manipulate the mask more directly, to be used in addition to this view-based API.

  • If an array has both a bitpattern dtype and a mask, then assigning np.NA writes to the mask, rather than to the array itself. Writing a bitpattern NA to an array which supports both requires accessing the data by “peeking under the mask”.

Another option is that described in NEP24, which is to implement bitpattern dtypes with NA semantics for the “statistical missing data” use case, and to also implement a totally independent API for masked arrays with ignore semantics and all mask manipulation done explicitly through a .mask attribute.

Another option would be to define a minimalist aligned array container that holds multiple arrays and that can be used to pass them around together. It would support indexing (to help with the common problem of wanting to subset several arrays together without their becoming unaligned), but all arithmetic etc. would be done by accessing the underlying arrays directly via attributes. The “prior art” discussion above suggests that something like this holding a .data and a .mask array might actually be solve a number of people’s problems without requiring any major architectural changes to NumPy. This is similar to a structured array, but with each field in a separately stored array instead of packed together.

Several people have suggested that there should be a single system that has multiple missing values that each have different semantics, e.g., a MISSING value that has NA semantics, and a separate IGNORED value that has ignored semantics.

None of these options are necessarily exclusive.

The debate#

We both are dubious of using ignored semantics as a default missing data behavior. Nathaniel likes NA semantics because he is most interested in the “statistical missing data” use case, and NA semantics are exactly right for that. Mark isn’t as interested in that use case in particular, but he likes the NA computational abstraction because it is unambiguous and well-defined in all cases, and has a lot of existing experience to draw from.

What Nathaniel thinks, overall:

  • The “statistical missing data” use case is clear and compelling; the other use cases certainly deserve our attention, but it’s hard to say what they are exactly yet, or even if the best way to support them is by extending the ndarray object.

  • The “statistical missing data” use case is best served by an R-style system that uses bitpattern storage to implement NA semantics. The main advantage of bitpattern storage for this use case is that it avoids the extra memory and speed overhead of storing and checking a mask (especially for the common case of floating point data, where some tricks with NaNs allow us to effectively hardware-accelerate most NA operations). These concerns alone appears to make a mask-based implementation unacceptable to many NA users, particularly in areas like neuroscience (where memory is tight) or financial modeling (where milliseconds are critical). In addition, the bit-pattern approach is less confusing conceptually (e.g., assignment really is just assignment, no magic going on behind the curtain), and it’s possible to have in-memory compatibility with R for inter-language calls via rpy2. The main disadvantage of the bitpattern approach is the need to give up a value to represent NA, but this is not an issue for the most important data types (float, bool, strings, enums, objects); really, only integers are affected. And even for integers, giving up a value doesn’t really matter for statistical problems. (Occupy Wall Street notwithstanding, no-one’s income is 2**63 - 1. And if it were, we’d be switching to floats anyway to avoid overflow.)

  • Adding new dtypes requires some cooperation with the ufunc and casting machinery, but doesn’t require any architectural changes or violations of NumPy’s current orthogonality.

  • His impression from the mailing list discussion, esp. the “what can we agree on?” thread, is that many numpy.ma users specifically like the combination of masked storage, the mask being easily accessible through the API, and ignored semantics. He could be wrong, of course. But he cannot remember seeing anybody besides Mark advocate for the specific combination of masked storage and NA semantics, which makes him nervous.

  • Also, he personally is not very happy with the idea of having two storage implementations that are almost-but-not-quite identical at the Python level. While there likely are people who would like to temporarily pretend that certain data is “statistically missing data” without making a copy of their array, it’s not at all clear that they outnumber the people who would like to use bitpatterns and masks simultaneously for distinct purposes. And honestly he’d like to be able to just ignore masks if he wants and stick to bitpatterns, which isn’t possible if they’re coupled together tightly in the API. So he would say the jury is still very much out on whether this aspect of the NEP design is an advantage or a disadvantage. (Certainly he’s never heard of any R users complaining that they really wish they had an option of making a different trade-off here.)

  • R’s NA support is a headline feature and its target audience consider it a compelling advantage over other platforms like Matlab or Python. Working with statistical missing data is very painful without platform support.

  • By comparison, we clearly have much more uncertainty about the use cases that require a mask-based implementation, and it doesn’t seem like people will suffer too badly if they are forced for now to settle for using NumPy’s excellent mask-based indexing, the new where= support, and even numpy.ma.

  • Therefore, bitpatterns with NA semantics seem to meet the criteria of making a large class of users happy, in an elegant way, that fits into the original design, and where we can have reasonable certainty that we understand the problem and use cases well enough that we’ll be happy with them in the long run. But no mask-based storage proposal does, yet.

What Mark thinks, overall:

  • The idea of using NA semantics by default for missing data, inspired by the “statistical missing data” problem, is better than all the other default behaviors which were considered. This applies equally to the bitpattern and the masked approach.

  • For NA-style functionality to get proper support by all NumPy features and eventually all third-party libraries, it needs to be in the core. How to correctly and efficiently handle missing data differs by algorithm, and if thinking about it is required to fully support NumPy, NA support will be broader and higher quality.

  • At the same time, providing two different missing data interfaces, one for masks and one for bitpatterns, requires NumPy developers and third-party NumPy plugin developers to separately consider the question of what to do in either case, and do two additional implementations of their code. This complicates their job, and could lead to inconsistent support for missing data.

  • Providing the ability to work with both masks and bitpatterns through the same C and Python programming interface makes missing data support cleanly orthogonal with all other NumPy features.

  • There are many trade-offs of memory usage, performance, correctness, and flexibility between masks and bitpatterns. Providing support for both approaches allows users of NumPy to choose the approach which is most compatible with their way of thinking, or has characteristics which best match their use-case. Providing them through the same interface further allows them to try both with minimal effort, and choose the one which performs better or uses the least memory for their programs.

  • Memory Usage

    • With bitpatterns, less memory is used for storing a single array containing some NAs.

    • With masks, less memory is used for storing multiple arrays that are identical except for the location of their NAs. (In this case a single data array can be re-used with multiple mask arrays; bitpattern NAs would need to copy the whole data array.)

  • Performance

    • With bitpatterns, the floating point type can use native hardware operations, with nearly correct behavior. For fully correct floating point behavior and with other types, code must be written which specially tests for equality with the missing-data bitpattern.

    • With masks, there is always the overhead of accessing mask memory and testing its truth value. The implementation that currently exists has no performance tuning, so it is only good to judge a minimum performance level. Optimal mask-based code is in general going to be slower than optimal bitpattern-based code.

  • Correctness

    • Bitpattern integer types must sacrifice a valid value to represent NA. For larger integer types, there are arguments that this is ok, but for 8-bit types there is no reasonable choice. In the floating point case, if the performance of native floating point operations is chosen, there is a small inconsistency that NaN+NA and NA+NaN are different.

    • With masks, it works correctly in all cases.

  • Generality

    • The bitpattern approach can work in a fully general way only when there is a specific value which can be given up from the data type. For IEEE floating point, a NaN is an obvious choice, and for booleans represented as a byte, there are plenty of choices. For integers, a valid value must be sacrificed to use this approach. Third-party dtypes which plug into NumPy will also have to make a bitpattern choice to support this system, something which may not always be possible.

    • The mask approach works universally with all data types.

Recommendations for moving forward#

Nathaniel thinks we should:

  • Go ahead and implement bitpattern NAs.

  • Don’t implement masked arrays in the core – or at least, not yet. Instead, we should focus on figuring out how to implement them out-of-core, so that people can try out different approaches without us committing to any one approach. And so new prototypes can be released more quickly than the NumPy release cycle. And anyway, we’re going to have to figure out how to experiment with such changes out-of-core if NumPy is to continue to evolve without forking – might as well do it now. The existing code can live in the main branch, be disabled, or live its own branch – it’ll still be there once we know what we’re doing.

Mark thinks we should:

  • The existing code should remain as is, with a global run-time experimental flag added which disables NA support by default.

A more detailed rationale for this recommendation is:

  • A solid preliminary NA-mask implementation is currently in NumPy main. This implementation has been extensively tested against scipy and other third-party packages, and has been in main in a stable state for a significant amount of time.

  • This implementation integrates deeply with the core, providing an interface which is usable in the same way R’s NA support is. It provides a compelling, user-friendly answer to R’s NA support.

  • The missing data NEP provides a plan for adding bitpattern-based dtype support of NAs, which will operate through the same interface but allow for the same performance/correctness tradeoffs that R has made.

  • Making it very easy for users to try out this implementation, which has reasonable feature coverage and performance characteristics, is the best way to get more concrete feedback about how NumPy’s missing data support should look.

Because of its preliminary state, the existing implementation is marked as experimental in the NumPy documentation. It would be good for this to remain marked as experimental until it is more fleshed out, for example supporting struct and array dtypes and with a fuller set of NumPy operations.

I think the code should stay as it is, except to add a run-time global NumPy flag, perhaps numpy.experimental.maskna, which defaults to False and can be toggled to True. In its default state, any NA feature usage would raise an “ExperimentalError” exception, a measure which would prevent it from being accidentally used and communicate its experimental status very clearly.

The ABI issues seem very tricky to deal with effectively in the 1.x series of releases, but I believe that with proper implementation-hiding in a 2.0 release, evolving the software to support various other ABI ideas that have been discussed is feasible. This is the approach I like best.

Nathaniel notes in response that he doesn’t really have any objection to shipping experimental APIs in the main numpy distribution if we’re careful to make sure that they don’t “leak out” in a way that leaves us stuck with them. And in principle some sort of “this violates your warranty” global flag could be a way to do that. (In fact, this might also be a useful strategy for the kinds of changes that he favors, of adding minimal hooks to enable us to build prototypes more easily – we could have some “rapid prototyping only” hooks that let prototype hacks get deeper access to NumPy’s internals than we were otherwise ready to support.)

But, he wants to point out two things. First, it seems like we still have fundamental questions to answer about the NEP design, like whether masks should have NA semantics or ignore semantics, and there are already plans to majorly change how NEP masks are exposed and accessed. So he isn’t sure what we’ll learn by asking for feedback on the NEP code in its current state.

And second, given the concerns about their causing (minor) ABI issues, it’s not clear that we could really prevent them from leaking out. (He looks forward to 2.0 too, but we’re not there yet.) So maybe it would be better if they weren’t present in the C API at all, and the hoops required for testers were instead something like, ‘we have included a hacky pure-Python prototype accessible by typing “import numpy.experimental.donttrythisathome.NEP” and would welcome feedback’?

If so, then he should mention that he did implement a horribly klugy, pure Python implementation of the NEP API that works with NumPy 1.6.1. This was mostly as an experiment to see how possible such prototyping was and to test out a possible ufunc override mechanism, but if there’s interest, the module is available here: njsmith/numpyNEP

It passes the maskna test-suite, with some minor issues described in a big comment at the top.

Mark responds:

I agree that it’s important to be careful when adding new features to NumPy, but I also believe it is essential that the project have forward development momentum. A project like NumPy requires developers to write code for advancement to occur, and obstacles that impede the writing of code discourage existing developers from contributing more, and potentially scare away developers who are thinking about joining in.

All software projects, both open source and closed source, must balance between short-term practicality and long-term planning. In the case of the missing data development, there was a short-term resource commitment to tackle this problem, which is quite immense in scope. If there isn’t a high likelihood of getting a contribution into NumPy that concretely advances towards a solution, I expect that individuals and companies interested in doing such work will have a much harder time justifying a commitment of their resources. For a project which is core to so many other libraries, only relying on the good will of selfless volunteers would mean that NumPy could more easily be overtaken by another project.

In the case of the existing NA contribution at issue, how we resolve this disagreement represents a decision about how NumPy’s developers, contributors, and users should interact. If we create a document describing a dispute resolution process, how do we design it so that it doesn’t introduce a large burden and excessive uncertainty on developers that could prevent them from productively contributing code?

If we go this route of writing up a decision process which includes such a dispute resolution mechanism, I think the meat of it should be a roadmap that potential contributors and developers can follow to gain influence over NumPy. NumPy development needs broad support beyond code contributions, and tying influence in the project to contributions seems to me like it would be a good way to encourage people to take on tasks like bug triaging/management, continuous integration/build server administration, and the myriad other tasks that help satisfy the project’s needs. No specific meritocratic, democratic, consensus-striving system will satisfy everyone, but the vigour of the discussions around governance and process indicate that something at least a little bit more formal than the current status quo is necessary.

In conclusion, I would like the NumPy project to prioritize movement towards a more flexible and modular ABI/API, balanced with strong backwards-compatibility constraints and feature additions that individuals, universities, and companies want to contribute. I do not believe keeping the NA code in 1.7 as it is, with the small additional measure of requiring it to be enabled by an experimental flag, poses a risk of long-term ABI troubles. The greater risk I see is a continuing lack of developers contributing to the project, and I believe backing out this code because these worries would create a risk of reducing developer contribution.

References and footnotes#

NEP 12 — Missing data functionality in NumPy describes Mark’s NA-semantics/mask implementation/view based mask handling API.

NEP 24 — Missing data functionality - alternative 1 to NEP 12 (“the alterNEP”) was Nathaniel’s initial attempt at separating MISSING and IGNORED handling into bit-patterns versus masks, though there’s a bunch he would change about the proposal at this point.

NEP 25 — NA support via special dtypes (“miniNEP 2”) was a later attempt by Nathaniel to sketch out an implementation strategy for NA dtypes.

A further discussion overview page can be found at: njsmith/numpy

On this page