DataTerminology.comNEW
Also known as: Ordinary Least Squares, OLS
Modeling approach that fits a linear relationship between predictors and a continuous outcome by minimizing residual error.
DESCRIPTION
Linear regression estimates the relationship between a dependent variable and one or more independent variables by fitting a linear function. Parameters are typically estimated via ordinary least squares, minimizing the sum of squared residuals. Assumptions include linearity, independence, homoscedasticity, and normal errors for inference. Extensions include regularization (ridge, lasso), generalized linear models, and robust regression. It is foundational for prediction, causal analysis under assumptions, and interpretability in analytics.
ALTERNATIVE DEFINITIONS
Wikipedia
In statistics, linear regression is a linear approach to modeling the relationship between a scalar response and one or more explanatory variables.
Highlights scalar response and linear modeling approach.
scikit-learn
LinearRegression fits a linear model with coefficients w = (w1, …, wp) to minimize the residual sum of squares between the observed targets and the predicted targets.
Operationalizes the model via least squares and coefficient vector.
RELATED TERMS
Multiple Regression Least Squares Residuals R-squared Gradient Descent Ridge Regression Lasso Heteroscedasticity Gauss–Markov Theorem Confidence Interval
COMPETING TERMS
RELATIONS
- Broader: Regression
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist Data Analyst AI Engineer Information Analyst
EXTERNAL SOURCES
- Wikipedia - Linear regression
- scikit-learn - Documentation on OLS and linear models
- Penn State STAT 501 - Course notes introducing linear regression
- NIST - Engineering statistics on linear regression
TOP QUESTIONS
What problems is Linear Regression best suited for?
Predicting a continuous outcome from numerical or categorical predictors. It excels when relationships are approximately linear and interpretability is important.
Category: Usage
What assumptions underpin Linear Regression?
Linearity, independence, homoscedasticity, and normally distributed errors for inference. Violations can bias estimates or invalidate significance tests.
Category: Requirements
How do you check model fit in Linear Regression?
Use residual plots, R-squared/adjusted R-squared, cross-validation error, and tests for heteroscedasticity or autocorrelation.
Category: Validation
How does regularization relate to Linear Regression?
Ridge and lasso add penalties to reduce overfitting and handle multicollinearity; elastic net combines both penalties.
Category: Related
What is the biggest misunderstanding about Linear Regression?
A high R-squared does not imply causality. Causal inference requires design or assumptions beyond predictive fit.
Category: Misunderstanding
When should you prefer nonlinear models over Linear Regression?
If residual patterns show strong curvature or interactions not captured linearly, consider polynomials, splines, or tree/ML models.
Category: Alternatives
How do categorical variables enter Linear Regression?
Through one-hot (dummy) encoding or contrast coding, allowing estimation of group differences relative to a reference.
Category: Practice
What is multicollinearity and why is it a problem?
Highly correlated predictors inflate variance of estimates, making coefficients unstable. Regularization or dimensionality reduction helps.
Category: Risk
How is Linear Regression optimized in large-scale settings?
Use closed-form solutions with linear algebra factorizations or iterative methods like gradient descent and stochastic variants.
Category: Implementation