Pandas: The Essential Python Library for Data Analysis

Learn Key features of Python pandas with coding examples. Also practice pandas on the Loan Recovery dataset. Download dataset and practice with the code shared below. 

Introduction

Pandas is an open source Python library providing high performance data manipulation and analysis tools. Pandas is built on top of NumPy, it offers intuitive data structures and functions essential for handling structured data.

Pandas is widely used in data science analytics, finance, statistics, and machine learning applications. It simplifies complex data processing tasks, making Python a powerful language for data analysis.

” Pandas is designed for efficient data manipulation and analysis.”

                                           Mark zukewin

What is the Key Features of Pandas?

• Data Structures: Provides Series (1D) and DataFrame (2D) objects.
• Data Cleaning: Handles unstructured data. Cleaning data helps to identify or bring data into a pattern that later on can be analyzed for achieving business goals.
• Handling Missing Values: It provides multiple ways like backward fill, forward fills, others to handle missing data.
• Data Transformation: Supports filtering, sorting, and reshaping data.
• Multiple File Formats Supports(File Handling): Reads and writes multiple file formats (CSV, Excel, JSON, SQL, etc.).
• Grouping & Aggregation: Enables data summarization using group operations.
• Time-Series Analysis: Supports date/time indexing and time-based functions.

Installing Pandas

How to ensure pandas is installed in your local machine? We can Ensure Pandas is installed in our Python environment using:

pip install pandas

Alternatively, if using Anaconda type following code on conda terminal:

conda install pandas

Pandas Data Structures

Series

A Pandas Series is a one dimensional labeled array, similar to a column in an Excel sheet. How to create a series using pandas library. First use import keyword then follow the library name.

import pandas as pd

# Creating a Series
s = pd.Series([10, 20, 30, 40], index=['A', 'B', 'C', 'D'])
print(s)

DataFrame

A DataFrame is a two dimensional table with labeled rows and columns. A DataFrame is collection of rows and columns. It’s similar to excel spreadsheet. DataFrame provides structured arrangement of data. This make data analysis very simple and easy to understand.

# Creating a DataFrame
data = {
    'Name': ['Ram', 'jack', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['New Delhi', 'bengaluru', 'Chicago']
}
df = pd.DataFrame(data)
print(df)

Reading and Writing Data

Pandas support multiple file formats.  Different pandas functions are availabel that makes easy to read files. We can read different formats of file such as excel spreadsheets, csv(Coma seperated values) and other formats. Pandas also helps to convert one format of file into other format. Let’s implement in python.

How to Read CSV Files in Pandas:

df = pd.read_csv('data.csv')
print(df.head())

Writing to CSV Files:

df.to_csv('output.csv', index=False)

Reading Excel Files:

df = pd.read_excel('data.xlsx', sheet_name='Sheet1')

Exploring Data

Understanding the dataset is crucial before analysis:

print(df.info())  # Displays column data types and non-null values
print(df.describe())  # Summary statistics for numerical columns
print(df.columns)  # List of column names

Data Selection and Filtering

Selecting Columns:

print(df['Name'])  # Selecting a single column
print(df[['Name', 'City']])  # Selecting multiple columns

Filtering Rows:

filtered_df = df[df['Age'] > 28]  # Filtering rows based on condition
print(filtered_df)

Data Cleaning and Handling Missing Values

Missing values can be managed using:

df.dropna()  # Remove rows with missing values
df.fillna({'Age': df['Age'].mean()})  # Fill missing Age with mean value

Data Manipulation

Adding a New Column:

df['Salary'] = [60000, 75000, 82000]

Applying Functions:

def categorize_age(age):
    return 'Young' if age < 30 else 'Senior'

df['Category'] = df['Age'].apply(categorize_age)

Grouping and Aggregation

Summarizing data using groupby:

df.groupby('City')['Salary'].mean()

Merging and Joining DataFrames

Combining datasets is common in data analysis.

df1 = pd.DataFrame({'Name': ['Alice', 'Bob'], 'Department': ['HR', 'IT']})
df2 = pd.DataFrame({'Name': ['Alice', 'Bob'], 'Salary': [70000, 80000]})
merged_df = pd.merge(df1, df2, on='Name')
print(merged_df)

Time Series Analysis

Pandas excels in handling time-series data:

df['Date'] = pd.to_datetime(df['Date'])  # Convert to datetime format
df.set_index('Date', inplace=True)  # Set datetime column as index
print(df.resample('M').mean())  # Resample data by month

Let’s Learn Pandas with Loan Recovery Dataset