Data Discovery Algorithm Techniques that you can coach
Data being the new oil in 2020, many enterprises around the world are now mature in big data practices and a data-first approach. The organizations have now.
Updated
On this page
Key takeaways
- Data being the new oil in 2020, many enterprises around the world are now mature in big data practices and a data-first approach.
- After the data file and type are detected the data is scanned to just capture the header and first 5 data elements.
- Every unicorn startup that would have challenged your industry would have somehow solved this challenge unhouse.
Data being the new oil in 2020, many enterprises around the world are now mature in big data practices and a data-first approach. The organizations have now been able to store massive data generated by there organization in some kind of data lakes, hubs or on-premise / public cloud. With big data comes the big responsibility to manage, organize, secure the data.
Over the years, big enterprises gather hundreds of data hubs and the data becomes disparate to a level that you cannot use it. This means the organization has data, in fact, lots of data? say big data. This data being big is still of no value as it is incapable of driving business results. According to Forester, 74% of firms say they want to be ‘data-driven,? but only 29% are successful at connecting data to business value. In 2020 Industry maturity is at a stage where storing the data has been accomplished for many organizations but drawing meaningful insights into that data in a timely fashion(real-time) is still a challenge.
In big organizations, the data is collected by multiple channels and the same data can have different fields and different definitions based on the source of origin.
To date, there is no defined way to automate the meaning of data in real-time. Many big companies have tried to solve this approach. The most innovative solution today in the market are podium data, waterline data. IBM is also a key player. The approach these companies use is to detect the headers of the data and capture the data type of the data. Sometimes it would even use a bit of NLP to come to a potential accurate meaning of data. But, most of the time, the business meaning of the data has to be manually mapped. Data discovery platforms would detect the headers to a certain level and then will provide a semi-automated way to define the data. Manual intervention to define the business definitions of each field is still a common practice in today’s world of artificial intelligence and machine learning.
The most innovative technology companies like Apple, Google, Amazon, and Microsoft ( recently ) growth is dependent on their ability to utilize the data of their customers in almost near real-time and also to handle the data securely is a critical part. As this is the secret sauce to their individual success story, the methodology is different for different companies. Tools might be available as open-source but it is difficult to understand — How they do it end to end?
Being an open-source technologist and an innovator, I try to solve these kinds of problems for the industry and share.
Like the famous Eureka, one day I thought why not create open-source algorithm ideation for data discovery that can be utilized by data teams around the globe.
Outline of the Data Discovery Algorithm: #
So consider the piece of code sitting between the source of the data and the destination of the data. Call it a data discovery filter. This filter fundamentally eliminates the need for costly ETL/ELT & Metadata management Tools and the implementation nightmare that a CEO or CTO goes through when doing digital transformation or legacy modernization in a complex organization.
The data is filtered in 2 stages or states: First Stage ( Data Discovery and Standardization ) and Second stage ( Change Data Discovery & Distribution standards )
Original diagram for Techfastly by Vin Patel.
First Stage Methodology: #
The first stage is when the data discovery filter detects the data. The Algorithm first detects the type of the file Is the data coming from a database( tabular) or is an image, audio, video, document. It can even be a script, geospatial data or an API. After the data file and type are detected the data is scanned to just capture the header and first 5 data elements. The Algorithm then maps each data element with 5 standard datatypes - Integer, Float, Character, String or Boolean. A further classification can happen by leveraging modern machine learning methodologies to find the data if it is a Name, Telephone Number, Social Security Number. The data can be then classified as a PII ( Personal Identifiable Information) or a Non-PII. PII information can be tokenized or scrubbed based on the need. Finally, a clean metadata file is generated with information on the data. If you are wondering that can this be done in real-time and on the fly, the answer is yes, it can be done. Yes, a data engineering culture is needed in your organization to do it. This is better than spending millions of dollars on expensive vendors.
Second Stage Methodology: #
The second pass is the clean version of the data and it tracks the data for changes and creates a standardized format of any data so that the data becomes meaningful and distributable to numerous channels in the organization. This will be a true data democratization
For example, if a data is coming to the filter, unchanged, the second pass will ignore it, if a data distribution format is created. If the same data enters the second stage changed it goes back to the first pass to refresh the metadata with changed data attributed.
The second pass is the final form of data and it has the 2020 standard data distribution format: JSON destination mappers like: send it to an API, or send it to a BI dashboard and so on. XML is created so that legacy data sources can be feedback back with clean data if needed.
The Data is a unique and most valuable asset to an Organization and the algorithm mentioned above has to be made relevant to each organization. If this approach is implemented correctly, the time to market and utilize the organizational data improves drastically.
The organization that can comprehend its data and the signals/insights that it generates is the organization that leaps ahead of its competition. Every unicorn startup that would have challenged your industry would have somehow solved this challenge unhouse.
#
Related reading