The dashed vertical line shows the decision boundary. The right side shows histograms of randomly chosen observations. The dashed line again is the Bayesian decision boundary. The solid vertical line is the LDA decision boundary estimated from the training data. When the Bayesian decision boundary and the LDA decision boundary are close, the model is considered to perform well.
When multiple regressors are used, then observations are assumed to be drown from a multivariate Gaussian distribution. Quadratic Discriminant Analysis QDA is similar to LDA based on the fact that there is an assumption of the observations being drawn form a normal distribution.
LDA is less flexible with lower variance. However, in LDA, observations share a common covariance matrix, resulting in higher bias. The model fits a Gaussian density to each class, assuming that all classes share the same covariance matrix. Source: Sci-Kit Learn - Click for more.
Each class is fitted with a Gaussian density. The first block loads all necessary libraries, creates the regressors and the dependent variable required by sklearn. Finally, the data set is partitioned into train and test sets. Since we are charged with creating the best model possible, let us create new features. Creation of these terms will bring up some issues but will spend more time on that in a bit.
For now, just notice that creating third order polynomials increased our column count from 23 to 1, When fitting LDA models, standardizing or scaling is a good idea. There are several articles out there explaining why standardizing is a must. Here we have to remember to standardize all of our data sets.
Since we created interaction terms and polynomials, multicollinearity will certainly be an issue. Here, we check if multicollinearity exists in the original data set, and then we go through the newly created two data sets containing second and third degree polynomials the same way. We simply will eliminate these variables from the analysis.
And now the painful task of eliminating variables begins. This may be a slow process if the dataset is large. Think about it: We need to create a matrix of correlations among all variables. In both cases, the number of variables was reduced significantly. The second order polynomial file now has only features, while the third order polynomial file contains only features vs.
We are ready to fit some models. Our first task is to fit a generic LDA model without any parameters tuned for two reasons: understand code structure and get a baseline accuracy. For fitting, we are using the first degree polynomial, e. So let us try to tune the parameters of the LDA model. There are three different solvers one can try, but one of them svd does not work with shrinkage. This can be good when we have only a few observations in our training data set so we lower the variance.
On the other hand, when the K classes have very different covariance matrices then LDA suffers from high bias and QDA might be a better choice. So, what is comes down to is the bias-variance trade-off. Therefore, it is crucial to test the underlying assumptions of LDA and QDA on the data set and then use both methods to decide which one is more appropriate.
I hope you have enjoyed the Linear vs. Quadratic Discriminant Analysis tutorial. If you have any questions, let me know in the comments below. Notify me of follow-up comments by email.
Notify me of new posts by email. Skip to content Linear vs. Linear Discriminant Analysis can only learn linear boundaries, while Quadratic Discriminant Analysis can learn quadratic boundaries and is therefore more flexible. QDA can perform better in the presence of a limited number of training observations because it does make some assumptions about the form of the decision boundary. QDA assumes a quadratic decision boundary, it can accurately model a wider range of problems than can the linear methods.
It can perform both classification and transform for LDA. It can be used for both classification and transform, and it supports shrinkage.
Ledoit O, Wolf M. The Journal of Portfolio Management 30 4 , , Toggle Menu. Prev Up Next. Linear and Quadratic Discriminant Analysis 1. Dimensionality reduction using Linear Discriminant Analysis 1. QDA 1. LDA 1.
0コメント