Regression Model Evaluation Metrics: R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE (2024)

Brandon Wohlwend

19 min read

Jul 14, 2023

The Role of Evaluation in Building Effective Regression Models

Evaluation plays a critical role in building effective regression models. By quantifying a model’s performance, evaluation metrics provide a tangible measure of how well the model is doing. These metrics give us a way to compare different models and select the one that performs the best.

Consider this: if you’ve developed several models to predict housing prices, how do you decide which one to use? Is it the one with the most complex features or perhaps the one with the least? Without evaluation metrics, we’re left to guesswork and gut feelings. Evaluation metrics remove this ambiguity. They allow us to rank models based on their performance and select the one that gives the most accurate predictions.

Beyond model selection, evaluation also helps in fine-tuning our models. By tweaking parameters and monitoring how these changes affect the evaluation metrics, we can optimize our models and push their performance even further.

The Pitfalls of Not Properly Evaluating Regression Models

Failing to properly evaluate regression models can lead to several pitfalls. A poorly evaluated model might perform excellently on your training data but fail miserably when faced with unseen data. This scenario, known as overfitting, is a common consequence of neglecting proper model evaluation. By solely focusing on the training data and failing to consider how the model generalizes, we risk developing models that are too complex and fail to capture the underlying pattern.

On the other end of the spectrum, we have underfitting, where our model is too simple to capture the underlying pattern even in the training data. Without proper evaluation, we might overlook this issue, leading to a model with subpar performance.

Without proper evaluation, it’s also easy to fall into the trap of the “accuracy paradox” — a model that is biased towards predicting the majority class can have a high accuracy rate, but this doesn’t mean the model is good. For instance, if 95% of emails are non-spam, a model that always predicts “non-spam” would be 95% accurate, but it would fail to catch any actual spam.

Proper evaluation protects us from these pitfalls, guiding us towards models that not only perform well on our training data but also generalize well to unseen data. As we navigate through the complex landscape of regression modeling, evaluation metrics serve as our north star, guiding us towards effective and reliable models.

In the process of model evaluation, we use a variety of metrics to assess the performance of our regression models. Each metric provides a unique perspective on how well the model is doing, and together they provide a comprehensive overview of the model’s performance.

Before we dive into the specifics of each metric, it’s important to understand what these metrics are fundamentally trying to achieve. Essentially, they are quantifying the difference between the predicted values our model is generating and the actual values that we observed in the data. The smaller this difference, the better our model is performing.

That said, not all differences are treated equally. Some metrics focus more on larger differences, penalizing models more if they make a few big mistakes. Others treat all differences equally, regardless of their size. Understanding these nuances is critical when selecting and interpreting evaluation metrics.

In this article, we’re going to explore five key metrics used for evaluating regression models:

R-Squared (R²): This is probably the most well-known metric for regression models. It measures the proportion of the total variation in the dependent variable that is captured by the model.
Adjusted R-Squared: This is a modified version of R-Squared that has been adjusted for the number of predictors in the model. It increases only if the new predictor improves the model more than would be expected by chance.
Mean Squared Error (MSE): This is the average of the squared differences between the predicted and actual values. It gives more weight to larger differences and is particularly useful when we have unexpected values that we want to take into account.
Root Mean Squared Error (RMSE): This is the square root of the MSE. By square rooting the MSE, we return the error metric to the same unit as the target variable, which can often make it easier to interpret.
Mean Absolute Error (MAE): This is the average of the absolute differences between the predicted and actual values. Unlike MSE, it treats all differences equally and is less sensitive to outliers.

In the following sections, we’ll explore each of these metrics in more detail, discussing how they’re calculated, how to interpret them, and their strengths and weaknesses.

Introduction to R-Squared

R-Squared, also known as the coefficient of determination, is one of the most commonly used metrics for evaluating the goodness of fit of a regression model. It provides a measure of how well the observed outcomes are replicated by the model, based on the proportion of total variation of outcomes explained by the model.

In simpler terms, R-Squared tells us the proportion of the variance in the dependent variable that is predictable from the independent variable(s). An R-Squared of 100% indicates that all changes in the dependent variable are completely explained by changes in the independent variable(s).

Mathematical Formulation of R-Squared

R-Squared is defined as the proportion of the variance in the dependent variable that is predictable from the independent variable(s). Mathematically, it can be calculated as:

Regression Model Evaluation Metrics: R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE (2)

Where:

SSR (Sum of Squared Residuals) is the sum of the squares of the residuals. The residuals are the difference between the actual values of the dependent variable and the predicted values from the regression model.
SST (Total Sum of Squares) is the total sum of squares, which is the sum of the differences between the actual values of the dependent variable and the mean value of the dependent variable, all squared.

The value of R² lies between 0 and 1. A value of 1 means the model perfectly predicts the dependent variable using the independent variable(s). A value of 0 means the model cannot predict the dependent variable at all using the independent variable(s).

How to Interpret R-Squared

Interpreting R-Squared is relatively straightforward. It is a decimal value between 0 and 1, and is often represented as a percentage when discussing model fit.

An R-Squared of 100% (or 1 when not multiplied by 100) indicates that all changes in the dependent variable are completely explained by changes in the independent variable(s). In other words, our model perfectly fits the data.

On the other hand, an R-Squared of 0% indicates that the dependent variable cannot be predicted from the independent variable(s) at all.

For instance, if the R-Squared of a model is 0.85, we can say that 85% of the variability in the output variable can be explained by the input variables that the model has used. The remaining 15% can be attributed to factors not included in the model, inherent randomness, or errors in the data.

Limitations of R-Squared

Despite its widespread use and ease of interpretation, R-Squared has its limitations:

It Only Measures Explained Variance: R-Squared does not tell us if the chosen model is good or bad, and it doesn’t convey the reliability of the model. It only quantifies the amount of variability in the target variable that’s accounted for by the predictors in the model.
Sensitive to Unnecessary Features: R-Squared value will either stay the same or increase with the addition of more variables, even if those variables are only weakly associated with the response. This can lead to overfitting, especially when dealing with many features.
Not Suitable for Comparing Different Datasets: R-Squared is not a good metric to compare model performances across different datasets. Because it measures the proportion of variance, its value can vary significantly with varying variances of different datasets.
Lower Performance with Non-linear Data: While R-Squared is a good measure for linear regression, it doesn’t perform as well when dealing with non-linear data patterns.

Understanding these limitations is crucial for the proper use and interpretation of R-Squared. In the next sections, we’ll explore more evaluation metrics that can complement R-Squared and provide a more comprehensive picture of model performance.

Introduction to Adjusted R-Squared

Adjusted R-Squared is a modified version of R-Squared that has been adjusted for the number of predictors in the model. Like R-Squared, it provides a measure of the proportion of the total variance in the dependent variable that is explained by the independent variables. However, it also takes into account the number of predictors used, adding a penalty for model complexity.

In other words, Adjusted R-Squared not only considers the goodness of fit, but it also takes into account the parsimony of the model, reflecting the principle of Occam’s razor: the simplest model that fits the data is the best.

Difference between R-Squared and Adjusted R-Squared

While R-Squared and Adjusted R-Squared both provide measures of how well the model fits the data, there’s a crucial difference between them: R-Squared assumes that every single variable explains the variation in the dependent variable, while Adjusted R-Squared adds a penalty for unnecessary complexity in the model.

The problem with R-Squared is that it tends to overestimate the performance of the model as more variables are added, even if those variables are only weakly associated with the response. This can lead to overfitting and misleadingly high R-Squared values.

Adjusted R-Squared overcomes this issue by decreasing the value when unnecessary predictors are included in the model. This makes Adjusted R-Squared a more robust measure for evaluating the overall quality of the regression model, especially when comparing models with a different number of predictors.

When to Use Adjusted R-Squared

Adjusted R-Squared should be used when you have multiple regression models with a different number of predictors. As you add more predictors to a model, R-Squared will always increase, even if those predictors offer no real improvement to the model. This can make it difficult to discern whether an increase in R-Squared is due to a truly better model, or merely due to the model’s increased complexity.

In contrast, Adjusted R-Squared increases only if the new variable improves the model more than would be expected by chance, and it decreases when a predictor improves the model by less than expected by chance. This makes it a more reliable metric when comparing models of different complexities.

How to Interpret Adjusted R-Squared

Like R-Squared, Adjusted R-Squared is a decimal between 0 and 1, and is often expressed as a percentage. An Adjusted R-Squared of 100% indicates that all changes in the dependent variable are completely accounted for by the independent variables in the model. A score of 0% indicates the model explains none of the variability of the response data around its mean.

However, unlike R-Squared, Adjusted R-Squared takes into account the number of predictors in the model. For example, if you have two models with the same R-Squared but different numbers of predictors, the model with fewer predictors will have a higher Adjusted R-Squared, reflecting the fact that it achieved the same goodness of fit with fewer predictors.

Interpreting Adjusted R-Squared in isolation can be misleading, as it doesn’t provide information on how much of the variance it doesn’t explain could be explained by a better model. As with any metric, it’s always best to use Adjusted R-Squared alongside other metrics to get a more complete picture of your model’s performance.

Introduction to MSE

Mean Squared Error (MSE) is another metric used to evaluate the performance of regression models. Unlike R-Squared and Adjusted R-Squared, which are measures of explained variance, MSE is a measure of prediction error. Specifically, it quantifies the average squared difference between the actual and predicted values.

MSE gives a higher penalty to large errors by squaring the residuals. This means models that produce larger errors will result in a larger MSE. As such, when using MSE as a metric, our goal is to minimize its value.

Mathematical Formulation of MSE

The MSE of an estimator measures the average of the squares of the errors — that is, the average squared difference between the estimated values and the actual value. It is defined mathematically as:

Regression Model Evaluation Metrics: R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE (3)

where:

n is the number of observations,
Yi is the actual value of the dependent variable for the i-th observation,
Ŷi is the predicted value of the dependent variable for the i-th observation.

The MSE is always non-negative, and a value of 0 indicates a perfect fit to the data. In practice, this would rarely happen outside of overfitting scenarios.

How to Interpret MSE

The MSE gives us an absolute measure of the magnitude of the model’s error term. It tells us how closely the model’s predictions match the observed data. The smaller the MSE, the closer the fit is to the data.

However, one of the challenges with interpreting the MSE is that it’s not immediately intuitive. It doesn’t tell us directly how much our prediction will deviate from the actual value on average. Instead, it tells us that on average, our prediction will deviate from the actual value by the square root of the MSE.

Another challenge is that MSE is scale-dependent. This means it’s difficult to compare the MSE of models for different datasets unless those datasets are measuring the same variable on the same scale.

Pros and Cons of MSE

Like any metric, the MSE has its pros and cons:

Pros of MSE:

Emphasizes larger errors: By squaring the residuals, MSE places heavier weight on larger errors. This can be beneficial when larger errors are particularly undesirable.
Differentiability: The square function has derivatives, which makes MSE more tractable for optimization in machine learning algorithms.

Cons of MSE:

Sensitive to outliers: Because MSE squares the residuals, it can be highly sensitive to outliers. A single outlier can potentially have a large effect on the MSE.
Scale-dependent: MSE is scale-dependent, which means you cannot compare the MSEs of different variables that are on different scales.
Not directly interpretable: The units of MSE are not the same as the units of the target variable. This makes it harder to interpret in a business setting.

When to Use RMSE

RMSE is a good measure to use when you care more about penalizing large errors. By squaring the errors before averaging them, RMSE gives higher weight to large errors. This means that RMSE is most useful in contexts where large errors are particularly undesirable.

Like MSE, RMSE is commonly used in regression analysis and forecasting where the aim is often to minimize large errors. It’s also a handy metric to use when you want to explain the performance of a model in a more interpretable way, since its units are the same as the target variable.

How to Interpret RMSE

The RMSE measures the average magnitude of the error term. It tells you how much error the system typically makes in its predictions, with a higher weight for large errors.

The value of RMSE is interpreted in the same units as the response variable, making it easier to relate to the variable you’re predicting. A smaller value of RMSE would indicate a better fit to the data, while a larger value indicates a poorer fit.

However, an RMSE of zero is not necessarily always the ultimate goal. A model that fits the training data too perfectly can result in overfitting, where the model performs well on the training data but poorly on new, unseen data.

As with any metric, it’s important to use RMSE in conjunction with other metrics to get a comprehensive understanding of your model’s performance.

Introduction to MAE

Mean Absolute Error (MAE) is another metric used to measure the performance of a regression model. Similar to MSE and RMSE, it quantifies the difference between the actual and predicted values. However, unlike MSE and RMSE, which square the residuals, MAE takes the absolute value of these differences.

By taking the absolute value, MAE gives equal weight to all errors, whether they’re big or small. This makes MAE a great measure to use when you want to know the average magnitude of the error, but don’t want to overly penalize large errors.

Mathematical Formulation of MAE

Mathematically, MAE is defined as the average of the absolute differences between the predicted and actual values. It is calculated as follows:

Regression Model Evaluation Metrics: R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE (4)

where:

n is the number of observations,
Yi is the actual value of the dependent variable for the i-th observation,
Ŷi is the predicted value of the dependent variable for the i-th observation.

MAE is always non-negative, with a lower value indicating a better fit to the data.

How to Interpret MAE

The MAE gives an idea of how wrong the predictions were, on average. Like RMSE, it’s measured in the same units as the target variable, so it’s relatively straightforward to interpret.

A smaller MAE indicates a better fit of the model to the data. An MAE of 0 means that the model makes perfect predictions (which is practically unlikely unless you’re overfitting your model).

One way to interpret the MAE is as follows: Given a MAE of ‘x’, you can say that on average, your prediction misses the true value by ‘x’ units.

Pros and Cons of MAE

As with any metric, the MAE has its pros and cons:

Pros of MAE:

Easy to Understand and Calculate: MAE is simple to understand and calculate. It provides a straightforward way to represent average error.
Less Sensitive to Outliers: Since MAE doesn’t square the residuals, it is less sensitive to outliers compared to MSE and RMSE. This makes it a better metric when outliers are not of particular concern.

Cons of MAE:

No Emphasis on Large Errors: While being less sensitive to outliers can be an advantage, it can also be a disadvantage when large errors are particularly undesirable.
Not Differentiable at Zero: Unlike MSE and RMSE, MAE isn’t differentiable at zero, which makes it less suitable for certain machine learning algorithms that rely on differentiation.

In the following sections, we’ll compare all the metrics discussed so far and provide a guideline on when to use which metric.

Comparison of the Different Metrics

Each metric that we have discussed so far — R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE — offers a unique perspective on the performance of a regression model. Let’s compare them:

R-Squared and Adjusted R-Squared: Both of these metrics provide a measure of how much of the variability in the dependent variable is explained by the independent variables. However, they don’t give us any information on the absolute size of the error. Adjusted R-Squared has an added benefit of taking into account the number of predictors in the model, which helps to avoid overfitting.
MSE and RMSE: Both MSE and RMSE give more weight to larger errors by squaring the residuals. They are useful when large errors are particularly undesirable. The key difference between them is that RMSE is in the same units as the dependent variable, making it easier to interpret. Both MSE and RMSE can be heavily influenced by outliers.
MAE: Unlike MSE and RMSE, MAE treats all errors equally by taking the absolute value of the residuals. It provides a clear representation of the average error and is less sensitive to outliers. However, it doesn’t put as much emphasis on large errors.

In general, no one metric is “the best” in all situations. The choice of which metric to use depends on the specific context, the presence of outliers, the importance of larger errors, and the needs of the stakeholders involved.

Discuss Scenarios to Choose One Over the Others

The choice of which metric to use depends on the particular circumstances of your modeling problem. Here are some scenarios where you might choose one metric over the others:

When outliers are a concern: If your data has outliers, or extreme values, and you don’t want these to heavily influence the model evaluation, then MAE would be a better choice as it is less sensitive to outliers compared to MSE and RMSE.
When larger errors are more significant: If your problem is such that larger errors are more costly than smaller ones, you might want to choose MSE or RMSE as these metrics penalize larger errors more heavily due to the squaring of residuals.
When comparing models with different numbers of predictors: If you are comparing models with different numbers of predictors, then Adjusted R-Squared would be a better choice than R-Squared. Adjusted R-Squared takes into account the number of predictors in the model, adding a penalty for model complexity.
When interpreting to non-technical stakeholders: If you are presenting your model’s performance to stakeholders who are not familiar with technical metrics, RMSE or MAE would be more intuitive since they’re in the original units of the target variable.
When the scale of errors matters: If it’s important to preserve the scale of errors, choosing RMSE over MSE would be beneficial since RMSE is in the same unit as the response variable.

Remember, no single metric can tell the whole story. It’s always a good idea to use a combination of metrics to evaluate your model from different perspectives.

Let’s use the popular Boston Housing dataset to implement a regression model and calculate these metrics. This dataset is a collection of housing values in the suburbs of Boston and is included in the sklearn library.

# Import necessary libraries
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
import numpy as np# Load dataset
boston = datasets.load_boston()
# Split dataset into features and target variable
X = boston.data
y = boston.target
# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create a linear regression model
model = LinearRegression()
# Train the model using the training sets
model.fit(X_train, y_train)
# Make predictions using the testing set
y_pred = model.predict(X_test)
# Calculate metrics
r2 = metrics.r2_score(y_test, y_pred) # R-Squared
print('R-Squared:', r2)
adjusted_r2 = 1 - (1-r2)*(len(y_test)-1)/(len(y_test)-X_test.shape[1]-1) # Adjusted R-Squared
print('Adjusted R-Squared:', adjusted_r2)
mse = metrics.mean_squared_error(y_test, y_pred) # Mean Squared Error
print('Mean Squared Error:', mse)
rmse = np.sqrt(mse) # Root Mean Squared Error
print('Root Mean Squared Error:', rmse)
mae = metrics.mean_absolute_error(y_test, y_pred) # Mean Absolute Error
print('Mean Absolute Error:', mae)

This script will output the values of R-Squared, Adjusted R-Squared, MSE, RMSE, and MAE for the implemented linear regression model.

Interpretation of Results

In the example script we used above, we ran a linear regression model on the Boston Housing dataset and evaluated its performance using five metrics. These metrics provide a variety of insights into the model’s performance:

R-Squared and Adjusted R-Squared: These metrics provide a measure of how much variation in the target variable (median home price) is explained by our predictors (features like crime rate, number of rooms, etc.). The closer these values are to 1, the more of the variation our model is capturing. Remember, however, that R-Squared always increases with the addition of more predictors, while Adjusted R-Squared incorporates a penalty for unnecessary predictors.
Mean Squared Error (MSE): This metric indicates the average squared difference between the actual and predicted home prices. A lower value is better and indicates fewer errors. But remember, since it squares the errors, MSE is sensitive to outliers — a few large prediction errors can greatly increase the value of MSE.
Root Mean Squared Error (RMSE): This is simply the square root of the MSE, so it’s on the same scale as the target variable (in this case, thousands of dollars). This makes it a more interpretable metric: the RMSE tells you, on average, how much our model’s predictions deviate from the actual home prices.
Mean Absolute Error (MAE): This metric represents the average absolute difference between actual and predicted home prices. Like RMSE, it’s also on the same scale as the target variable, but unlike RMSE and MSE, it treats all errors equally regardless of their magnitude. This makes it less sensitive to outliers.

Interpreting these metrics in combination can provide a more comprehensive picture of your model’s performance. For instance, a high R-Squared coupled with a low RMSE would generally indicate a good model fit. But it’s also important to remember that these metrics only speak to your model’s performance on the given dataset. To gauge how the model might perform on new data, you’ll need to use techniques like cross-validation.

In this article, we’ve delved deep into the world of regression model evaluation. We’ve learned about several key metrics: R-Squared, Adjusted R-Squared, Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE).

Each of these metrics offers a unique perspective on your model’s performance. R-Squared and Adjusted R-Squared give a sense of how much of the variance in the target variable your model can explain. MSE and RMSE provide a measure of the average prediction error, with RMSE giving larger errors more weight. And MAE gives you an idea of the average absolute error, without giving additional weight to large errors.

We also discussed the scenarios in which each of these metrics might be most useful, from dealing with outliers to avoiding overfitting. Finally, we saw how to calculate these metrics in Python using the Boston Housing dataset.

While getting a model to work might feel like the end of a data science project, the reality is that it’s just the beginning. Evaluating the model’s performance, tuning it, and re-evaluating it is an essential part of the process.

Remember that no single metric can tell you everything you need to know about your model’s performance. It’s important to consider multiple metrics and understand what each one tells you and what its limitations are.

As you continue your journey in data science, I encourage you to keep these metrics in mind. Always question how well your model is performing and how you can measure that performance in a meaningful way. Your projects will be all the stronger for it.

Thank you for taking the time to read this article. I hope you found it helpful and informative. Here’s to better models and more insightful evaluations!