Using UMAP For Faster And Effective Outlier Detection
Machine learning algorithms are getting fancier by the day, and today we can find an ML technique to do any task we want.
Updated
On this page
Updated November 2026. This was first published on 24 November 2022. The idea is the same. Any number below is what the original piece said then. It is not a new count.
Key takeaways
- Machine learning algorithms are getting fancier by the day, and today we can find an ML technique to do any task we want.
- If you are well-versed in machine learning, you will know what outliers mean.
- In case you are not, outliers are the data points in a dataset that are significantly different from the rest of the dataset.
Machine learning algorithms are getting fancier by the day, and today we can find an ML technique to do any task we want. If you are well-versed in machine learning, you will know what outliers mean. In case you are not, outliers are the data points in a dataset that are significantly different from the rest of the dataset.
These outliers can be eliminated from any data collection in a variety of methods. You can plot a scatterplot or a KDE and identify the points that are the furthest away from the group for straightforward datasets. But, these visual outlier detection methods would not be effective with a high-dimensional dataset.
Now, sophisticated machine learning methods that are effective against outliers in high-dimensional data, such as Local Outlier Factors or Isolation Forests, come to mind. But there are numerous restrictions when using ML techniques to find outliers. An elliptical Envelope is a method that uses a covariance estimation but presumes that the data is normally distributed (which is rarely the case). Isolation Forest is significantly slower than the Local Outlier Factor but at the expense of accuracy.
So, given that working with large, million-row datasets is so prevalent, how can we excel at outlier detection both in terms of speed and accuracy?
This is where UMAP comes in to save the day!
What is UMAP? #
UMAP of Uniform Manifold Approximation and Projection is a dimensionality reduction technique founded in 2018, that can be used for clustering as well as outlier detection. The approach can be applied to general non-linear dimension reduction as well as visualization techniques like t-SNE. It combines the best aspects of both Principal Component Analysis and t-SNE, projecting data to low-dimensional space more skillfully and attractively than tSNE while scaling to huge datasets quickly.
Three suppositions regarding the data form the basis of the algorithm.
The distribution of the data on the Riemannian manifold is uniform; 2. The Riemannian metric can be estimated as being locally constant; 3.
UMAP is intended to keep as much variance and topological structure in the data as possible, regardless of the dimension you project the data. What does it have to do with the detection of outliers, though?
We may use UMAP to project the data to a smaller space first, then apply any other outlier identification technique much more quickly because we know that it keeps all the quirks and characteristics of the dataset even in reduced dimensions.
Significance of Outlier Detection #
A question must have arisen in your mind, “Why are we even detecting these outliers?”. Let us clear that out of the way before understanding the working of UMAP or any other machine-learning algorithm.
Data preprocessing and cleaning is a crucial stage in the machine learning pipeline since it makes the data easier to interpret. You deal with missing numbers, find outliers, and more in this stage. The existence of outliers can frequently affect the findings of statistical studies performed on the dataset since they have highly distinct values (abnormally low or abnormally high). This can result in models that are less effective and helpful.
So, prior to using any model, we must find and eliminate any outliers to train the model. If we don’t, the outliers may distort the data distribution and produce unexpected results.
But, handling outliers frequently calls for domain knowledge, and none of the outlier detection strategies should be used without first comprehending the data distribution and the use case.
Further in this blog, you will read how we combine UMAP with the isolation forest algorithm to detect the outliers in any dataset.
Outlier Detection using UMAP and Isolation Forest #
Before starting off with UMAP, we will establish a base using just Isolation Forest. Despite its strength, Isolation Forest only offers a few tuning options. The most crucial one is n_estimators, which regulate the number of trees to be constructed. But doing this will give us an estimated number of outliers.
The snag is that we do not know if they are outliers when we find the number of outliers after that. Since we can’t really use conventional visualization techniques for confirmation, we must first project the data to two dimensions and create a scatterplot.
The singleton points that are the furthest apart from the clusters are instantly visible (they are presented in tiny sizes because we didn’t change the UMAP settings). We may categorize these points as outliers with confidence and have faith in the Isolation Forest results.
The exciting part is when UMAP and Isolation Forest are combined. We must project the data into a particular number of dimensions, let’s say 5, then fit the isolation forest with fewer trees to optimize retention as much as feasible. In this phase, reducing the number of trees will be simpler because there won’t be as many characteristics to fit in.
The entire process of transformation and detection will take way less time than it would take with another outlier detection algorithm.
Summary #
Outlier detection is not a complicated process and can be done through various methods. our goal was to represent how it can be done faster using UMAP.
Towards data science has actually performed an analysis where they initially set the number of trees (n_estimators) to 3000 and waited for about 50 minutes for the Isolation Forest to detect the number of outliers. The estimated number of outliers came out to be 2270.
Then they tested the same dataset after combining UMAP and Isolation Forest, and the entire process took less than one-third of the time that is taken previously. In fact, this time Isolation Forest detected around 163k outliers out of 950k.
Although in that experiment, 3000 trees were probably not enough for the huge dataset, still with the help of UMAP, the process of outlier detection became much less time-consuming and resource-consuming.
Questions and answers #
What is the main point? #
Machine learning algorithms are getting fancier by the day, and today we can find an ML technique to do any task we want.
What else did the original piece say? #
If you are well-versed in machine learning, you will know what outliers mean.