How Data and Machine Learning are Used within This Industry

How Data and Machine Learning are Used within This Industry
Market Sector
Lecture 6 explores the enterprise market sector, specifically how Data and Machine Learning are used within this industry. Enterprise Search is used in this lecture as a case study to demonstrate the various ML and information retrieval techniques used within the enterprise sector.
Description
Enterprise content is distributed across multiple data sources in various formats. For example, in a company, employee data can be found in the employee database, the company’s internal collaboration system, and the finance ERP. This wide content distribution within the enterprise sector can make retrieving relevant content challenging. Therefore, Enterprise Search makes it easier for enterprises to find relevant content from multiple data sources without worrying about where the information is stored.
Key Data Science Skills and Technologies
Indexing
Indexing systems are an essential part of enterprise search. Their significance goes beyond information extraction, filtering, and classification. These tools enable users to search the textual content of webpages by converting pages in web repositories into appropriate index structures. Through pipelines, indexing systems pre-process documents and perform other extraction tasks on webpages. To excel in enterprise search, data science professionals must be conversant with the concept of full-text indexing and text document properties such as metadata and structure. Additionally, one must understand the use of pipelines for document processing, term selection, and the removal of bearing words.
Deep Learning
Deep learning is an essential data science technology in the enterprise market because it helps query interpretation. It influences matching users’ queries to relevant documents. Query term expansion, one of the mechanisms used in a query interpretation system, utilizes deep learning by applying synonyms, tokenization, and inverted index to maximize the successful matching of queries and results (documents).
Machine Learning
The benefits of machine learning in the enterprise sector are extensive. It plays a significant part in enterprise search because it is involved in processes such as tokenization, stopword removal stemming, and query interpretation. Supervised ML is specifically substantial when data science professionals need to develop a model that addresses the needs of their users.
Use of Data and Computing-related methods
All three high-level architecture components of Web search utilize data and computing-related methods to execute their tasks. Since queries have statistical patterns, data and computing is often leveraged to speed up the search. For example, optimization of the search results to be more user-specific is a complex process that requires the application of several computer-related methods. Search engines also often construct links and word frequencies and run this data through their ML algorithm.
Challenges in Enterprise Search
Redundancy
Redundancy is a common problem in the enterprise sector because project teams traditionally load data marts from existing data sources into the data lake. Teams would then add their unique data that matched their needs. While this approach is convenient, simple, and faster, it contributes to the proliferation of similar data marts. Blindly copying data from other sources and adding our modifications will cause extreme levels of redundancy.
Reliance on Tribal Knowledge
Various actors in the enterprise sector, including companies, rely heavily on...