Exploratory data analysis

Learning outcome statements

The learning outcome statements relevant for this section are:

1. describe exploratory data analysis and how it is used to reveal patterns and discover insights

Definition of exploratory data analysis

Definitions

Exploratory data analysis (EDA)

An approach used to analyze datasets to summarize their main characteristics, identify patterns, detect outliers, and uncover relationships between variables.

EDA is typically the first step in the data analysis process, as it helps analysts understand the structure of the data before applying more complex models or statistical techniques.

Watch out: EDA generates hypotheses - it doesn’t confirm them. A pattern it turns up, like two variables moving together, is a starting point for confirmatory analysis (such as regression), not proof of causation.

Purpose and role of EDA

One of the primary goals of EDA is to identify data quality issues, such as missing values or inconsistencies, which need to be addressed before further analysis. It also helps organizations discover hidden trends and patterns, providing a foundation for hypothesis generation and predictive modeling.

Example: Reviewing expense claims

An analyst reviews 24 months of expense-claim data. A histogram shows the amounts are right-skewed. A box plot flags one month as an outlier, which turns out to be a data-entry error and gets corrected. A scatter plot of claims against headcount suggests a positive relationship.

Answer: That relationship is a hypothesis to test later with regression - not a proven cause-and-effect result.

EDA is a crucial step because it ensures that conclusions drawn from the data are based on a clear understanding of its characteristics.

Tools used in EDA

By applying these techniques, analysts can make informed decisions about how to proceed with predictive or prescriptive modeling, leading to more accurate and reliable insights.

EDA employs both graphical and quantitative methods:

• Graphical tools: histograms show the distribution of a single variable, box plots show the median, quartiles, and outliers, scatter plots show the relationship between two variables, and correlation matrices show pairwise correlation coefficients across many variables at once.
• Statistical tools: descriptive statistics, including measures of central tendency (mean, median, mode) and dispersion (range, variance, standard deviation), plus correlation coefficients that measure how strongly two variables move together.

Software and programming tools used in EDA:

• Python libraries (Pandas, Matplotlib, Seaborn) for visualization and analysis.
• R packages (ggplot2, dplyr) for advanced data manipulation and visualization.
• SQL for extracting and preparing structured data.