Regression and statistical methods

Logistic regression is the better choice when the dependent variable is categorical rather than continuous - often binary, such as whether a customer defaults on a loan (yes/no) or whether a transaction is flagged as fraud (yes/no). Instead of predicting a dollar amount, logistic regression estimates the probability of the outcome, a value between 0 and 1. Use multiple regression when Y is a continuous number, like sales dollars; use logistic regression when Y is a category, like default or no default.

Benefits and shortcomings of regression analysis

Benefits

• Predictive power: Regression analysis allows companies to forecast outcomes, such as sales or costs, based on different levels of input (independent) variables.
• Decision-making: By understanding the relationship between variables, businesses can make informed decisions about how changes in one factor (e.g., production volume) will impact others (e.g., costs or revenue).
• Identifying trends: Regression can help identify underlying trends in historical data, allowing organizations to project future performance.

Shortcomings

• Assumption of linearity: Linear regression, whether simple or multiple, assumes a linear relationship between variables, which may not always be the case in real-world situations.
• Data limitations: Regression analysis depends on the quality of data. Inaccurate or incomplete data can lead to unreliable predictions.
• Overfitting: Using too many variables in a regression model can lead to overfitting, where the model becomes too tailored to historical data and fails to predict future outcomes accurately.
• Correlation is not causation: a strong relationship between two variables doesn’t prove that one causes the other - an omitted third variable may be driving both.
• Multicollinearity: when independent variables are highly correlated with each other, individual coefficients become unstable and harder to interpret.

Coefficient of determination (R²) and correlation coefficient $\text{(r)}$

Two key statistical measures help assess relationships between variables:

1. coefficient of determination (R²) and
2. correlation coefficient $\text{(r)}$

Correlation coefficient $\text{(r)}$

The $\text{correlation coefficient (r)}$ measures the strength and direction of a linear relationship between two variables. It ranges from $\text{-1 to 1}$

• $\text{r = 1}$ indicates a perfect positive correlation, meaning as one variable increases, the other also increases.
• $\text{r = -1}$ indicates a perfect negative correlation, meaning as one variable increases, the other decreases.
• $\text{r = 0}$ suggests no linear correlation between the variables.

In data analytics, correlation helps determine whether a relationship exists between two variables, such as marketing spend and revenue.

Coefficient of determination (R²)

The $\text{R² value}$ tells us how well the independent variable(s) explain the variability of the dependent variable. It ranges from $\text{0 to 1}$:

• $\text{R² = 0}$ means the independent variable explains none of the variation in the dependent variable.
• $\text{R² = 1}$ means the independent variable explains 100% of the variation.

In data analytics, a higher $\text{R²}$ value indicates a better model fit, meaning the independent variables are strong predictors of the dependent variable.

For example, in sales forecasting, if an R² value is 0.85, it means 85% of the variation in sales can be explained by the independent variables (such as advertising spend, promotions, and seasonality). This helps businesses evaluate the reliability of their predictive models.

In simple regression (one independent variable), R² equals r squared - for example, an r of −0.87 gives an R² of about 0.76. Notice that r carries a sign showing the direction of the relationship, while R² is always non-negative, since squaring removes the sign.

Standard error of the estimate (SEE), goodness of fit, and confidence interval

In data analytics, these statistical concepts help evaluate how well a model fits data and how reliable predictions are.

Standard error of the estimate (SEE)

The SEE measures the accuracy of predictions in a regression model. Specifically, it is the standard deviation of the residuals, the differences between actual and predicted values of Y. It is expressed in the same units as Y, so an SEE of $2,000 on a sales forecast means actual sales typically land within about $2,000 of the prediction. A lower SEE means that the predicted values are closer to the actual values, indicating a better model fit. It is useful in data analytics to assess the reliability of predictive models, such as forecasting customer demand based on past sales data.

Goodness of fit

This term refers to how well a statistical model matches the observed data. $\text{R²}$ is a commonly used measure of goodness of fit. A model with poor goodness of fit may lead to inaccurate insights and poor decision-making.

Confidence interval (CI)

A confidence interval (CI) is a statistical range that estimates where a true population parameter (such as a mean or regression coefficient) is likely to fall, given a specified level of confidence (e.g., 95%). It reflects the degree of uncertainty around an estimate rather than providing a single point value.

For example, if revenue is forecasted to increase by $10 million with a 95% CI of ±$1 million, it means we can be 95% confident that the actual increase will fall between $9 million and $11 million.

For a regression coefficient, a 95% confidence interval that includes zero means the variable may have no real effect on Y. Equivalently, its p-value is above 0.05, so the coefficient is not statistically significant. A low p-value (commonly below 0.05) indicates the relationship is unlikely to be due to chance, and a coefficient of zero means the variable has no effect on Y.

Drawing insights and making recommendations

Deciding whether to act on a regression result means checking R² and each coefficient’s confidence interval (or p-value) together - R² shows how well the model fits overall, while a coefficient’s confidence interval shows whether that specific coefficient is reliable enough to rely on.

Returning to the sales forecasting example: suppose the model’s R² is 0.85, meaning the independent variables explain 85% of the variation in sales, and the advertising coefficient’s 95% confidence interval is $3.00 to $4.00, which excludes zero. Because the model fits well overall and the advertising coefficient is statistically significant, management can act on the $51,200 forecast - for example, shifting more budget toward advertising, since every dollar spent there is associated with approximately $3.50 of predicted sales.

The SEE adds a sense of precision to the forecast itself. If the model’s SEE is $2,000, the predicted sales of $51,200 fall within an approximate 95% prediction interval of $51,200 − 2($2,000) to $51,200 + 2($2,000), or $47,200 to $55,200, since about 95% of observations fall within two standard errors of the predicted value.