{"id":16171,"date":"2022-05-30T12:49:35","date_gmt":"2022-05-30T12:49:35","guid":{"rendered":"https:\/\/blog.datumo.com\/en\/?p=16171"},"modified":"2024-10-22T08:23:36","modified_gmt":"2024-10-22T08:23:36","slug":"handling-missing-data-in-pandas-dataframes-in-python","status":"publish","type":"post","link":"https:\/\/blog.datumo.com\/en\/tech\/16171","title":{"rendered":"Handling Missing Data in Pandas Dataframes in Python"},"content":{"rendered":"[vc_row pix_particles_check=””][vc_column]
[vc_raw_html]JTNDbWV0YSUyMGh0dHAtZXF1aXYlM0QlMjJyZWZyZXNoJTIyJTIwY29udGVudCUzRCUyMjAlM0IlMjB1cmwlM0RodHRwcyUzQSUyRiUyRmRhdHVtby5jb20lMkZlbiUyRmhhbmRsaW5nLW1pc3NpbmctZGF0YS1pbi1wYW5kYXMtZGF0YWZyYW1lcy1pbi1weXRob24lMkYlMjIlM0U=[\/vc_raw_html][vc_column_text css=”.vc_custom_1653914996046{padding-top: 40px !important;padding-right: 40px !important;padding-bottom: 0px !important;padding-left: 40px !important;}”]Missing data, as the name suggests, are data observations that do not contain any value. Missing values can have a huge impact on the performance of data science models. Therefore, it is important to understand the reasons behind missing data, and the techniques that can be adopted to handle missing data.\r\n\r\nIn this article, you will study some of the common reasons for having missing data in your observations. Finally, you will study how to handle missing values in data stored in Pandas dataframe, which is one of the most common structures for storing data in the Python programming language.[\/vc_column_text]<\/div><\/div>
<\/div>

Reasons for Missing Data<\/h2><\/div>[vc_column_text css=”.vc_custom_1653915132335{padding-top: 40px !important;padding-bottom: 40px !important;}”]There can be multiple reasons for having missing data in your observations. Three of the most common reasons are listed below:\r\n
    \r\n \t
  1. Sometimes the data is not recorded intentionally. For instance, while completing market surveys, some people might not enter their annual revenue. In such cases, you will have missing observations in your data.<\/li>\r\n \t
  2. The data is oftentimes not available at the time when the observation is recorded. For instance, some of the information about the patients who are brought to emergencies in hospitals might not be immediately available at the time of registering patients.<\/li>\r\n \t
  3. Information is often lost due to technical reasons or calculations. For instance, you might have recorded a person\u2019s weight in a computer but due to application crashing or power failure the data is not recorded.<\/li>\r\n<\/ol>\r\n \r\n
    Types of Missing Data<\/strong><\/h5>\r\n \r\n\r\nMissing data can be divided into three main categories:\r\n\r\n \r\n
    Missing Data Completely Randomly<\/strong><\/h5>\r\n \r\n\r\nIf a missing observation for a particular attribute doesn\u2019t have any relation with other attributes, we can say that the data is missing completely randomly. For instance, if a person forgets to write his city name in a survey, you can say that this data is missing completely randomly since there is no other logical conclusion of missing data.\r\n\r\n \r\n
    Missing Data Randomly<\/strong><\/h5>\r\n \r\n\r\nIn case if a missing observation has a relationship with one of the other attributes of the data, we can say that the data is missing randomly. For instance, it is likely that overweight patients do not write their weights while filling marketing surveys, in such cases we can say that the data is missing randomly\r\n\r\n \r\n
    Missing Data Not Randomly<\/strong><\/h5>\r\n[\/vc_column_text]
    <\/div>[\/vc_column][\/vc_row][vc_section full_width=”stretch_row” pix_over_visibility=”” css=”.vc_custom_1650444445523{padding-top: 80px !important;padding-bottom: 80px !important;background-color: #f8f9fa !important;}” el_id=”pix_section_program”][vc_row full_width=”stretch_row” pix_particles_check=””][vc_column content_align=”text-center” offset=”vc_col-lg-offset-0 vc_col-lg-12 vc_col-md-offset-1 vc_col-md-10″]

    Disadvantages of Missing Data<\/h2><\/div>[vc_column_text css=”.vc_custom_1653915187992{padding-top: 40px !important;padding-bottom: 40px !important;}”]\r\n

    There are multiple disadvantages of having missing data in your datasets. Some of the disadvantages are enlisted below:<\/p>\r\n\r\n

      \r\n \t
    1. Some data science and machine learning tools such as Scikit learn don’t expect your data to have missing values. You have to either remove or somehow handle missing values before you could feed your data to train models from these libraries,<\/li>\r\n \t
    2. The data imputation techniques may distort the overall distribution of your data.<\/li>\r\n<\/ol>\r\n

      Enough of the theory, let\u2019s now see some of the techniques used for handling missing data stored in Pandas dataframes in Python.<\/p>\r\n[\/vc_column_text][\/vc_column][\/vc_row][\/vc_section][vc_row pix_particles_check=””][vc_column]

      <\/div>

      Importing and Analysing the Dataset for Missing Values<\/h2><\/div><\/div><\/div>[vc_column_text css=”.vc_custom_1653915528895{padding-top: 40px !important;padding-right: 20px !important;padding-bottom: px !important;padding-left: 20px !important;}”]The dataset that you will be using in this article can be downloaded in the form of a CSV file from the following Kaggle link.\r\n\r\nhttps:\/\/www.kaggle.com\/code\/dansbecker\/handling-missing-values\/data?select=melb_data.csv<\/a>\r\n\r\nThe script below imports the dataset and displays its header.\r\n\r\n \r\n
      import pandas as pddata_path = \"\/home\/manimalik\/Datasets\/\"house_data = pd.read_csv(data_path + \"melb_data.csv\")house_data.head()<\/pre>\r\n \r\n\r\nThe database consists of information about houses in Melbourn (city of Australia). Some of the data attributes are Room, Price, Postcode, Building Area, Year Built etc.\r\n\r\n \r\n\r\nOutput:\r\n\r\n\"\"\r\n\r\n \r\n\r\nLet\u2019s try to see the number of records in our dataset.\r\n\r\nhouse_data.shape\r\n\r\n \r\n\r\nOutput:\r\n\r\n(13580, 21)\r\n\r\n \r\n\r\nThe dataset consists of 13580 records and 21 attributes.\r\n\r\n \r\n\r\nNext, let\u2019s try to see which attributes or columns contain null values.\r\n
      house_data.isnull().sum()<\/pre>\r\n \r\n\r\n \r\n\r\nOutput:\r\n\r\n\"\"\r\n\r\n \r\n\r\nFrom the above output, you can see that Car, BuildingArea, YearBuilt, and CouncilArea are the columns having missing data. The BuildingArea attribute has null values in 6450 records which is almost 47.50% of the total dataset.\r\n\r\nLet\u2019s filter the columns with the missing values. We will be working with these columns only for handling missing data.\r\n
      house_data = house_data.filter([\"Car\", \"BuildingArea\", \"YearBuilt\", \"CouncilArea\"])house_data.head()<\/pre>\r\n \r\n\r\nOutput:\r\n\r\n\"\"\r\n\r\n \r\n\r\nFinally, let\u2019s try to print the data types of the columns having missing values.\r\n
      house_data.dtypes<\/pre>\r\n \r\n\r\nOutput:\r\n\r\n\"\"\r\n\r\n \r\n\r\nThe above output shows that Car, BuildingArea, and YearBuilt are numeric attributes (of type float), whereas only the CouncilArea attribute contains categorical values.[\/vc_column_text]
      <\/div>
      <\/div>[\/vc_column][\/vc_row][vc_section full_width=”stretch_row” pix_over_visibility=”” css=”.vc_custom_1650444445523{padding-top: 80px !important;padding-bottom: 80px !important;background-color: #f8f9fa !important;}”][vc_row full_width=”stretch_row” pix_particles_check=””][vc_column content_align=”text-center” offset=”vc_col-lg-offset-0 vc_col-lg-12 vc_col-md-offset-1 vc_col-md-10″]
      Remove Complete Rows or Columns with Missing Data<\/h2><\/div><\/div><\/div>[vc_column_text css=”.vc_custom_1653915610586{padding-top: 40px !important;}”]\r\n
      The first and the simplest approach of handling missing data is by removing all the records where any column contains missing value. Or alternatively, you can also drop a column if the majority of values in that column is null.<\/p>\r\n
      Let\u2019s try to remove all the rows containing null values from our sample dataset<\/p>\r\n\r\n
      house_data_filteres = house_data.dropna()print(house_data_filteres.shape)<\/pre>\r\n \r\n
      Output:<\/p>\r\n
      (6196, 21)<\/p>\r\n
      After removing all the rows where any column contains a null or missing value, we are left with only 6196 records.<\/p>\r\n
      Another option is to only remove those rows where a specific column contains missing values. For example, the script below removes all the rows where the BuildingArea attribute contains null values.<\/p>\r\n\r\n
      house_data_filteres = house_data[house_data['BuildingArea'].notna()]print(house_data_filteres.shape)<\/pre>\r\n \r\n
      Output:<\/p>\r\n
      (7130, 21)<\/p>\r\n \r\n
      The main advantage of removing all the rows containing missing values is that this technique is extremely simple to implement and works for both numeric and categorical data types.<\/p>\r\n
      The main disadvantage of removing rows with missing values is that if a large number or rows contain missing values, you will lose a large chunk of useful data contained in attributes that do not have any missing values.<\/p>\r\n
      As a rule of thumb, handle missing data by removing rows only if less than 5% of the rows contain missing data.<\/p>\r\n
      In addition to removing complete rows, imputation techniques exist that can be used to fill missing data by inferencing values via interpolation from the dataset. In the next sections, you will see imputation techniques for handling missing numeric and categorical data.<\/p>\r\n[\/vc_column_text][\/vc_column][\/vc_row][\/vc_section][vc_row pix_particles_check=””][vc_column]
      <\/div>
      <\/div>
      Handling Numeric Missing Values using Imputation<\/h2><\/div><\/div><\/div>[vc_column_text css=”.vc_custom_1653915651682{padding-top: 40px !important;padding-bottom: 0px !important;}”]The main imputation techniques for handling missing numeric values are:\r\n