What is the difference between linear discriminant analysis and logistic regression?

While studying multivariate statistics, I have learned that many of the functions to perform multivariate statistic computations can actually be found in Python’s machine learning library, sklearn.

The multivariate statistical functions that I am aware of that are methods in sklearn are:-

• Principal component analysis,
• Linear regression,
• Logistic regression,
• Canonical correlation analysis,
• Factor analysis,
• Kmeans clustering,
• Hierarchical clustering, and
• Linear discriminant analysis.

I am familiar with Python’s machine learning library, sklearn, because I began my journey into data science using the data science website, Kaggle. I studied machine learning before statistics, so I was familiar with some of these functions already.

Linear discriminant analysis

Linear discriminant analysis is a method that can be found in sklearn, but it is a function in multivariate statistics. One reason for this is because statistics has predated machine learning. Machine learning became a scientific endeavour in the 1950’s, while statistics has been around since the 19th century.

Because statistics is older than machine learning and the two sciences are related, many of the statistical functions have become machine learning functions through the course of time.

One statistical function found in sklearn is linear discriminant analysis. Discriminant analysis is a powerful tool in multivariate statistics used to classify observations into predefined groups based on their characteristics. It assumes that observations come from known categories and attempts to find the best separation between these groups.

The primary goal of discriminant analysis is to predict group membership (e.g., classifying customers as high-risk or low-risk based on financial data. It achieves this by identifying the boundaries between categories using multivariate data.

There are several types of discriminant analysis, being:-

• Linear Discriminant Analysis (LDA): Assumes the classes are normally distributed and shares equal covariance matrices. It is best for data where the relationship between variables is linear.
• Quadratic Discriminant Analysis (QDA): Allows classes to have different covariance matrices, making it more flexible but also prone to overfitting if sample sizes are small.
• Canonical Discriminant Analysis (CDA): Focuses on reducing the dimensions of the data while maximizing separation between groups.

Discriminant analysis considers multiple variables simultaneously, meaning it can capture complex relationships between features that determine group membership.

The steps in performing discriminant analysis are:-

1. Define Groups: Assign observations to distinct groups (e.g., customers categorized as “default” vs “no default”).

2. Select Predictors: Choose variables that are likely to explain group membership (e.g., income, credit score).

3. Fit the Model: For LDA, the algorithm estimates the means and covariance for each group to create a linear decision boundary. For QDA, it estimates separate covariance matrices for each group and creates nonlinear boundaries.

4. Classification: New observations are classified based on the group that has the highest posterior probability under the fitted model.

Applications of LDA include:-

• Finance: Predicting loan default risk based on customer data.
• Medicine: Classifying patients into disease categories based on symptoms or diagnostic results.
• Marketing: Grouping customers based on purchasing behavior.
• Education: Categorizing students into performance levels based on academic records.

Logistic regression

Logistic regression is a fundamental statistical technique used for classification, particularly in multivariate statistics, where multiple independent variables influence a categorical dependent variable. It models the probability that an observation belongs to a particular class.

Key Concepts in Multivariate Logistic Regression include:-

1. Binary vs. Multivariate Logistic Regression: Binary Logistic Regression is used when the dependent variable has two categories (e.g., “pass” or “fail”). Multivariate Logistic Regression includes multiple predictor variables influencing the outcome. Multinomial Logistic Regression is used when the dependent variable has more than two categories (e.g., “low,” “medium,” and “high risk”).

2. The Logit Function:

Logistic regression models the probability of belonging to a class using the logit (log-odds) function: Logit(p) = log(p/(1-p). The coefficients (β) indicate how each predictor variable influences the probability of the outcome.

3. Interpretation of Coefficients is as follows: Positive coefficient indicates an increase of the likelihood of the target class. A negative coefficient indicates a decrease in the likelihood. Exponentiated coefficients (odds ratio) provide insights into how each predictor affects the probability of classification.

Applications of Multivariate Logistic Regression include:-

• Medical Diagnostics: Predicting the probability of disease based on multiple symptoms.
• Financial Risk Analysis: Assessing loan default risk based on factors like income, credit score, and debt ratio.
• Marketing: Predicting customer behavior (e.g., likelihood of purchasing a product).
• Education: Classifying students into performance groups based on demographics and test scores.

Why Logistic Regression is used in Multivariate Statistics

• It effectively handles multiple predictors and identifies relationships between them.
• Unlike linear regression, it is tailored for classification problems.
• It provides probabilistic outputs instead of direct numeric predictions.
• It supports feature selection, helping determine which variables are most significant.

The difference between linear discriminant analysis and logistic regression

Logistic regression and linear discriminant analysis (LDA) are both used for classification problems, but they approach the task differently.

Logistic Regression is:-

• A discriminative model that estimates the probability of class membership directly using the logistic function.
• It models the relationship between input features and the likelihood of belonging to a specific class.
• Works well when features have little correlation and when the dataset is relatively small.

Linear Discriminant Analysis (LDA) is:-

• A generative model that assumes that each class follows a normal (Gaussian) distribution with shared covariance.
• It finds a linear boundary by maximizing the ratio of between-class variance to within-class variance.
• Performs well when feature distributions are approximately normal and works effectively with small datasets.

Key Differences of the models are:-

An example of how logistic regression differs from linear discriminant analysis cab be seen in the Python code below:-

When experimenting with a model, it is important to try it out using several models and selecting the model that has the greatest accuracy.

I have prepared a code review on this blog post and it can be viewed here:- https://youtu.be/IU4SoeRROOA