Как заменить nan на 0 в python
NaN stands for Not A Number and is one of the common ways to represent the missing value in the data. It is a special floating-point value and cannot be converted to any other type than float. NaN value is one of the major problems in Data Analysis. It is very essential to deal with NaN in order to get the desired results.
Methods to Replace NaN Values with Zeros in Pandas DataFrame
In Python, there are two methods by which we can replace NaN values with zeros in Pandas dataframe. They are as follows:
Replace NaN Values with Zeros using Pandas fillna()
The fillna() function is used to fill NA/NaN values using the specified method. Let us see a few examples for a better understanding.
Replace NaN values with zeros for a column using Pandas fillna()
Syntax to replace NaN values with zeros of a single column in Pandas dataframe using fillna() function is as follows:
Handling Missing Data
The difference between data found in many tutorials and data in the real world is that real-world data is rarely clean and homogeneous. In particular, many interesting datasets will have some amount of data missing. To make matters even more complicated, different data sources may indicate missing data in different ways.
In this section, we will discuss some general considerations for missing data, discuss how Pandas chooses to represent it, and demonstrate some built-in Pandas tools for handling missing data in Python. Here and throughout the book, we’ll refer to missing data in general as null, NaN, or NA values.
Trade-Offs in Missing Data Conventions¶
There are a number of schemes that have been developed to indicate the presence of missing data in a table or DataFrame. Generally, they revolve around one of two strategies: using a mask that globally indicates missing values, or choosing a sentinel value that indicates a missing entry.
In the masking approach, the mask might be an entirely separate Boolean array, or it may involve appropriation of one bit in the data representation to locally indicate the null status of a value.
In the sentinel approach, the sentinel value could be some data-specific convention, such as indicating a missing integer value with -9999 or some rare bit pattern, or it could be a more global convention, such as indicating a missing floating-point value with NaN (Not a Number), a special value which is part of the IEEE floating-point specification.
None of these approaches is without trade-offs: use of a separate mask array requires allocation of an additional Boolean array, which adds overhead in both storage and computation. A sentinel value reduces the range of valid values that can be represented, and may require extra (often non-optimized) logic in CPU and GPU arithmetic. Common special values like NaN are not available for all data types.
As in most cases where no universally optimal choice exists, different languages and systems use different conventions. For example, the R language uses reserved bit patterns within each data type as sentinel values indicating missing data, while the SciDB system uses an extra byte attached to every cell which indicates a NA state.
Missing Data in Pandas¶
The way in which Pandas handles missing values is constrained by its reliance on the NumPy package, which does not have a built-in notion of NA values for non-floating-point data types.
Pandas could have followed R’s lead in specifying bit patterns for each individual data type to indicate nullness, but this approach turns out to be rather unwieldy. While R contains four basic data types, NumPy supports far more than this: for example, while R has a single integer type, NumPy supports fourteen basic integer types once you account for available precisions, signedness, and endianness of the encoding. Reserving a specific bit pattern in all available NumPy types would lead to an unwieldy amount of overhead in special-casing various operations for various types, likely even requiring a new fork of the NumPy package. Further, for the smaller data types (such as 8-bit integers), sacrificing a bit to use as a mask will significantly reduce the range of values it can represent.
NumPy does have support for masked arrays – that is, arrays that have a separate Boolean mask array attached for marking data as «good» or «bad.» Pandas could have derived from this, but the overhead in both storage, computation, and code maintenance makes that an unattractive choice.
With these constraints in mind, Pandas chose to use sentinels for missing data, and further chose to use two already-existing Python null values: the special floating-point NaN value, and the Python None object. This choice has some side effects, as we will see, but in practice ends up being a good compromise in most cases of interest.
Data Science for Beginners: Handling Missing Values With Pandas
Missing values can appear as ‘NaN’ (Not a Number), ‘NA’ (Not Available), ‘n/a’, ‘na’, ‘?’, a blank space, an out-of-range value and in many other forms depending on the user(s) filling in the data. In real datasets, missing values are almost unavoidable and they can be caused by several reasons like corrupted data or unrecorded observations.
Learning to handle missing values in a dataset is very important because most machine learning models cannot handle missing values. In this tutorial, you’ll learn how to handle missing values using Pandas. Let’s get started!
Identifying Missing Values
To handle missing data, we need to first identify them. Pandas identifies some missing value forms by default as NaN values (e.g blank entry, ‘NA’, ‘null’, ‘nan’, ‘n/a’, ‘NULL’) but not some others. To make Pandas recognize other non-default missing value forms (e.g ‘?’, ’na’, ‘Nil’), we can make a list of them and pass them into Pandas’ .read_excel() method (since the dataset is an excel file in this case) as is done below:
This gives an output such that all missing value forms used are now recognized by Pandas:
When Pandas has been made to identify all missing value forms, we can then mark them out. To mark out missing values in a given dataset, Pandas isnull() and notnull() functions come in handy. Pandas isnull() marks all missing values as True while notnull() marks all missing values as False.
To check for missing values in the dataset or the number of missing values per column:
To check for the number of missing values in each column:
Ways of handling missing values
Missing values can be handled by:
Deletion
Here rows or columns containing missing values are deleted. This is however not a good way to handle missing values especially when there are many missing values in the dataframe as it leads to loss of data.
To delete rows or columns containing missing values, Pandas dropna() function is used:
Imputation
In this method, missing values are replaced by a constant value or they are replaced based on other observations in the dataset. Pandas fillna() function is used for imputing values. Imputation can be done by:
i. Replacing missing values with a predetermined constant value.
To fill all missing values in the dataframe with a constant:
To fill a particular column in the dataframe with a constant:
ii. replacing missing values in a column with values from previous rows (forward or backward).
Forward fill: this fills missing values with values from previous rows in a forward manner
Backward fill: this fills missing values using values from the later rows in a backward manner
iii. replacing missing values in a column with the mean, median or mode of that column.
Interpolation
In this method, missing values are handled using Pandas interpolate() function. Interpolation can be done using different methods e.g linear, pad, nearest, quadratic method. Pandas however carries out linear interpolation by default.
The way you choose to handle the missing values in your dataset will depend on the type and description of the dataset you are working on.
Замена значения NaN на ноль в DataFrame Pandas
Вы можете заменить значения NaN на 0 в Pandas DataFrame с помощью метода DataFrame.fillna(). Передайте ноль в качестве аргумента методу fillna() и вызовите этот метод в DataFrame, в котором вы хотите заменить значения NaN на ноль.
Метод fillna() возвращает новый DataFrame со значениями NaN, замененными указанным значением.
Образец фрагмента кода
Ниже приведен пример фрагмента кода для замены значений NaN на 0.
Пример 1
В следующей программе Python мы берем DataFrame с некоторыми значениями, как NaN (numpy.nan). Затем мы воспользуемся методом fillna(), чтобы заменить эти значения numpy.nan на ноль.
Все значения NaN в DataFrame заменяются на 0.
Пример 2: замена в указанных столбцах
Вы также можете заменить значения NaN на 0 только в определенных столбцах. В следующем примере программы показано, как заменить значения numpy.nan на 0 для столбца «a».
В этом руководстве примеров Python мы узнали, как заменить значения NaN на 0 в DataFrame.