Feature Engineering

Converting Data into Relevant Formats

What is Automated Feature Engineering?

Feature engineering is the process of using domain knowledge to extract features from raw data through data mining techniques. Feature engineering is more an art than a science in that it uses domain knowledge of the data to create features that increase the predictive power of machine learning algorithms. Data Engineering will allow you to represent the underlying structure of the data accurately and therefore create the best model. Features can be engineered by decomposing or splitting features, from external data sources, or aggregating or combining features to create new features. Automated Feature Engineering is a process of preparing a dataset for machine learning by changing features or deriving new features to improve machine learning model performance. It is one of the fundamental blocks of Tibil’s iDEA framework and data analytics platform where we transform the customer’s data through our in-depth knowledge of our customer’s domain.

Common Data Management Platform Challenges

In several engagements we found that the customer data sets primarily contained messy, unstructured data which was then processed, cleaned and stored. This data can contain missing values, outliers or redundant data. With data being collected from a wide variety of sources, data redundancy becomes an unavoidable predicament. These lead to lot of challenges when it comes to implementing feature engineering for machine learning.

While working on data, some of the common challenges are:

Analysing huge
volumes of data from disparate sources

Availability of data for analytics in recognizable patterns for tools or statistical models

Overlaying business context and process to the patterns to ensure relevant analysis to deliver business insights

Visualizing the outcome of data analytics in appealing and relevant dashboards for quick decision making

Improving the
time-to-value of the
data analytics process

Tibil’s Feature Engineering Solution

We have helped customers solve this problem through feature selection: after categorizing or selecting the features, those features which are highly correlated to another are removed, eliminating the data redundancy. We have used the feature generation process to create new features from one or multiple existing features, for use in statistical analysis. The feature generation process adds new information that can be accessed during the model construction in the process of machine learning in data analysis to result in a more accurate model. Over the years, Tibil has gained significant domain knowledge, primarily in the verticals of BFSI, Retail, Healthcare, Education, Energy/Utilities and Manufacturing.

Benefits of Automated Feature Engineering

We believe that by leveraging our Feature Engineering solution in automated ML, our customers can benefit with:

Rapid time-to-value: Derived variables rather than data that is fed into analytics engine thereby speeding up the data analytics process
Fit-for-purpose analytics: Ensuring the availability of formatted data based on business context or process so that the analytics process drives business value

We have leveraged this domain expertise to create a feature engineering process that involves:

Brainstorming or testing data feature
Deciding what data features to create and creating them
Checking how the data features work with the data analytics model
Improving the data features further if needed
Go back to brainstorming/creating more data features

An Example of Feature Engineering

For an Energy/Utilities customer in North America, TIBIL created a comprehensive, cloud-based data management system for collecting, normalizing and storing data; used proven statistical models for data analysis; and developed a sophisticated reporting system for geo-mapping of all sensors. The result of our data analytics were valuable insights on key performance parameters like the Percentage of sensors violating industry standards, Number of sensors with high frequency of outages and brownouts and Sensors with a high standard deviation.

Schedule a meet