Data Cleaning with Pandas in Python

Data cleaning is an important step in the data science process as it helps to ensure the quality of the data we use for analysis. In this blog post, we’ll go over the basic data cleaning tasks using the pandas library in Python.

Importing Libraries

First, we need to import the necessary libraries for our analysis. In this case, we will be using the pandas library for data manipulation and analysis.

import pandas as pd

Input Customer Feedback Dataset

Next, we will input our customer feedback dataset. We will be using a sample customer feedback dataset for this example.

df = pd.read_csv("customer_feedback.csv")
print(df.head())

Here is a preview of the input customer feedback dataset:

print(df.head())

   customer_id  satisfaction_score customer_feedback
0            1                  4.0               Good
1            2                  5.0              Great
2            3                  NaN                NaN
3            4                  2.0               Bad
4            5                  4.0             Okay

Locate Missing Data

It’s important to identify and handle missing data in our dataset as it can negatively impact our analysis. We can use the .isnull() method to find missing values in our dataset and the .sum() method to see the total number of missing values for each column.

missing_values = df.isnull().sum()
print(missing_values)

customer_id            0
satisfaction_score     1
customer_feedback      1
dtype: int64

Check for Duplicates

Duplicate data can also impact our analysis, so it’s important to identify and remove any duplicate data. We can use the .duplicated() method to find any duplicates in our dataset and the .drop_duplicates() method to remove them.

duplicates = df.duplicated()
print(duplicates.sum())
df = df.drop_duplicates()

0

Here is a preview of the customer feedback dataset after removing duplicates:

print(df.head())

   customer_id  satisfaction_score customer_feedback
0            1                  4.0               Good
1            2                  5.0              Great
2            3                  NaN                NaN
3            4                  2.0               Bad
4            5                  4.0             Okay

Detect Outliers

Outliers can also impact our analysis, so it’s important to identify and handle any outliers in our dataset. We can use the .describe() method to get a summary of the statistics for our data and identify any potential outliers.

df.describe()

       customer_id  satisfaction_score
count    5.000000            4.000000
mean     3.000000            3.750000
std      1.581139            1.290994
min      1.000000            2.000000
25%      2.250000            3.000000
50%      3.000000            4.000000
75%      4.000000            4.750000
max      5.000000            5.000000

Normalize Casing

In order to ensure consistency in our data, it’s important to standardize the casing of our text data. We can use the .lower() method to convert all text data to lowercase.

df["customer_feedback"] = df["customer_feedback"].str.lower()
print(df.head())

   customer_id  satisfaction_score customer_feedback
0            1                  4.0               good
1            2                  5.0              great
2            3                  NaN                NaN
3            4                  2.0               bad
4            5                  4.0             okay

And that’s it! These are the basic data cleaning tasks that can be performed with the pandas library in Python. It’s important to always check and clean your data before conducting any analysis to ensure the accuracy and quality of your results.

Conclusion:

The takeaway from this data cleaning process is that it is an important step in the data science process to ensure the accuracy and quality of the data used for analysis. By locating missing data, checking for duplicates, detecting outliers, and normalizing casing, we can help ensure that our data is reliable and free of any inaccuracies.

Using the pandas library in Python, these basic data cleaning tasks can be easily performed and automated, making the data cleaning process more efficient and streamlined. It’s essential to use data cleaning before conducting any analysis as it can greatly impact the accuracy of the results. Therefore, investing time and effort into cleaning your data is a critical step in ensuring the success of your data science projects.