Dimensionality Reduction Algorithms
Dimension Reduction is very important when you are working with data with a high dimension.
Updated
On this page
Updated November 2026. This was first published on 9 November 2020. The idea is the same. Any number below is what the original piece said then. It is not a new count.
Key takeaways
- Dimension Reduction is very important when you are working with data with a high dimension.
- It is very tiring and time-consuming work to analyze each dimension of your data; and especially when you like to go for data visualization.
- Many times we observed and collect data with similar information or redundant features related to our project.
Dimension Reduction is very important when you are working with data with a high dimension. It is very tiring and time-consuming work to analyze each dimension of your data; and especially when you like to go for data visualization. Many times we observed and collect data with similar information or redundant features related to our project. It is good that we have more information related to our target variable but it will be time-consuming as well as expensive to use such data. So, it is very important to do dimension reduction before building our prediction model or implementing a machine learning model for high dimension data. Dimension Reduction is used when you want to remove noise from your data or your data have different scale features.
Here I am listing few advantages of dimension reduction which can help you make a decision when you should go for dimension reduction.
The Dimension Reduction Algorithms are used in many applications such as 1) text classification like email classification, 2) Customer relationship management analyzing customer feedback or response messages, 3) Image compression, 4) Intrusion detection system, 5) Face recognition, 6) Handwritten digit recognition, 7) Medical data such as Protein Classification, Gene expression micro-array data analysis, and many others.
The Dimension Reduction Algorithms are categories in two parts these are as follows.
Principal Component Analysis — Principal Component Analysis is commonly known as PCA. It i #
PCA is used to decompose high dimension data into a set of successive orthogonal components while keeping a maximum amount of data variance. PCA is used to center each feature before applying SVD to it. PCA does not scale the input data. we need to scale our data before applying PCA on it to get the best result out of it.
An example of PCA setup on image processing #
The above-given image is the scatter plot of 2 component PCA results of the IRIS dataset. The IRIS dataset is very famous among data Scientists for testing their machine learning algorithms. In this dataset, there is 4 feature variable and 150 rows. We have performed PCA on this data and reduced 4 features to 2 features so that we can visualize this data.
Randomized PCA** is nothing but a randomized SVD. In this case, the system performs linear dimension reduction using SVD but keeps only the most significant singular vectors to project given data to a lower dimension. This can be used for face recognition where an image has 64 x 64 pixels (64*64=4096 dimension) for the gray-level image. We can use PCA to linearly transform data into a lower dimension while preserving most of the explained variance.
There are two other methods also under PCA; these are incremental PCA and Kernel PCA. These PCA methods can be used based on a different dataset.
PCA is used in many applications such as image compression, face recognition, feature extraction, media editing, etc.
ISOMAP (Isometric Mapping) ? #
ISOMAP is another method of dimension reduction. It is a non-linear method unlike PCA based on spectral theory. An ISOMAP method is one of the widely used methods of low dimensional embedding.
ISOMAP starts with creating a neighborhood network and use graph distance to measure the geodesic distance between all pairs of data points. And then using eigenvalue decomposition of the geodesic distance matrix it creates low dimensional embedding of a given dataset. Here ISOMAP uses geodesic distance because we considering neighborhood networks is non-linear. And for non-linear network Euclidean distance is a misleading distance matrix.
ISOMAP defined geodesic distance as a sum of edge weights along the shortest path between two data points. Dijkstra Algorithm can be used to find the shorted path between data points. MDS (Multidimensional scaling) performs low dimensional embedding based on the pairwise distance between data points.
An Example of ISOMAP for Handwritten digits #
ISOMAP is used for recognizing handwritten digits. An MNIST dataset consists of 4096 features to explain different handwritten digits. The images of handwritten digits are of 4096 dimensions(64 x 64 pixel) are reduced to just 2 dimensions using ISOMAP. We find the ISOMAP algorithm gives better results compared to PCA.
The above image is a subset of the MNIST dataset after ISOMAP dimension reduction.
The disadvantage of ISOMAP - ISOMAP does not perform well when the manifold of data has holes.
Spectral Embedding — Spectral Embedding is another method to calculate a non-linear low di #
Laplacian Eigen maps use the spectral decomposition of graph Laplacian to find low dimensionality of given data. Graph Laplacian is generated as a discrete approximation of low dimensional manifold in high dimensional space. Minimization of the Cost function is used to ensure that points close to each other on the manifold are mapped according to low dimensional space as well. There are three-step involved in the Spectral Embedding method. The first step is constructing a weighted graph, the second step is building graph Laplacian and the third and final step is Eigenvalue decomposition on graph Laplacian.
The Graph Laplacian has been widely used for clustering and data partition problems. The Locality preserving property of Graph Laplacian makes it insensitive to outliers and noise in data. As long as embedding is isometric, the representation of data will not change. Images taken from moving cameras, different resolution of the camera leads to the embedding of the same underlying manifold spaces of different resolution. Spectral Embedding will produce similar images irrespective of camera resolution.
The Spectral Embedding (Laplacian Eigen map) fails to produce a coherent representation of a classification problem data when training data consists of instances varying significantly in terms of different classes.
An example of Spectral Embedding to identify human speech #
Spectral Embedding algorithm can be used to differentiate human speech. Human speech data are high-dimensional data but distinctive phonetic dimensions are very less. There were 685 features used to recognize speech. These features are reduced to three features like vowels, nasals, and semivowels. Phonetic homogeneity of the data points helps us to represent data points with the same label originating from different occurrences. You can find more details on it in the reference.
LLE (Locally-Linear Embedding) ? #
LLE is a graph-based method of creating low dimension representation of high dimensional datasets by preserving the locally linear structure of nearby input patterns. LLE is discovered by Roweis and Saul in the year 2000. It is very much different from ISOMAP because its outputs are derived from the bottom eigenvectors of the sparse matrix, as compared to the top eigenvectors of the Gram Matrix.
Let’s take a Data set with N dimension (n columns) which lies on the nonlinear manifold of dimensionality D where D < N. This data set is mapped into a single global coordinate system with lower dimensionality D such that the relationships between neighboring data points are preserved. The global nonlinear structure can be recovered by locally linear embedding. LLE algorithms can be processed in the following three steps.
Identify the neighbors of each data point; we can use the K Nearest Neighbors method for this.
Compute weight matrix that best linearly constructs each data points from its neighbors.
LLE calculates the bar centric coordinates of each data point and its neighbors. The same weight will be used to reconstruct the same data points in lower-dimensional space. A neighborhood map is created by preserving data points and its neighbors. Each data point Xi in N-dimensional space is mapped onto to Yi in D dimensional space by minimizing the cost function.
In the above cost function, weights are kept constant and Yi is going to be optimized to minimize the cost function.
The disadvantage of LLE is to handle non-uniform samples because there is no fixed unit to prevent weights to vary as various regions vary in sample densities. Also, LLE has no internal model as such.
An example of LLE for facial recognition #
LLE algorithm can be used for Facial Recognization. Most of the images are nonlinear, so it is better to use LLE compare to PCA. Each image is of 92*112 dimension which is reduced to 6 dimensions using LLE. After dimension reduction, the dimension of data is reduced but the nonlinear structure is maintained, This helps in recognizing image more easily.
Kernel Approximation — Kernel Approximation is a non-linear transformation of input data t #
There are four different methods for Kernel Approximation.
Nystroem Method — It is a general method for low ranked approximation of kernels. Nystroem method creates sub-samples of given data and evaluates different kernel functions on it. By default, it takes RBF kernel but it can use any kernel function. The dimensionality of features is computed by giving parameter n_components.
Radial Basis Function — Radial Basis Function is a non-linear method of kernel construction. It usages Monte Carlo sampling in fit function and maps data into a lower dimension with the help of transform function. The fit function takes two arguments n_components, which is output dimensionality of features transformation and gamma, which is a parameter for RBF kernel.
Additive Chi-Squared — Additive Chi-Squared Kernel method is implemented by additiveChi2Sampler function. This method uses Fourier transformation in a regular interval for sampling. As the kernel is additive, so it treats each component separately for embedding.
Skewed Chi-Squared — Skewed Chi-Squared Kernel function has similar properties as an Exponentiated chi-squared kernel. Exponentiated chi-squared kernel method maps features by combining the approximation feature map of AdditiveChi2Samplerand RBFSampler function. It is also used for computer vision feature mapping.
Kernel Approximation is better to compare to implicitly feature maps for large sample data because Kernel Approximate takes less cost of learning and also standardize large scale dataset.
The Disadvantage of the kernel method is that it might need more memory allocation and time as it might be necessary to store many kernel values during optimization.
Questions and answers #
ISOMAP (Isometric Mapping) ? #
ISOMAP is another method of dimension reduction. It is a non-linear method unlike PCA based on spectral theory.
LLE (Locally-Linear Embedding) ? #
LLE is a graph-based method of creating low dimension representation of high dimensional datasets by preserving the locally linear structure of nearby input patterns. LLE is discovered by Roweis and Saul in the year 2000.