Up till now we have discussed on how we can create some features and how we can encode them. We’ve, hopefully, ended up with lots of variables and we are seeking a way to keep the top-performing ones. In this post I’m going to explain ‘why’ and ‘how’ we should cleverly select our features.
In this post I’m going to talk about one of the most creative aspects of data science: feature encoding. If you haven’t followed my previous posts on feature engineering, you can find them in this link before reading this one.
After creating all features, it is quickly evident that there are a lot of missing data. Missing data should be dealt with before going on to the next phases of our model building. In this post, I’m going to quickly describe the reasons behind the missing values and ways to treat them.
As I was brainstorming possible features for the ML model, I created a lot of graphs that helped me get a better understanding of what is hidden in the data and what features may be helpful for the ML algorithms. Here, I share some of the most interesting and weird ones!
Having explored the data, checked what is available and what’s not, found any inconsistencies and potential problems, it is time to get creative. Feature extraction is the step where an experienced data scientist can really make the difference and improve the subsequent model’s accuracy; many times more than it will be possible through algorithm selection…
After discussing in my previous post about Data Acquisition, here I’m going to describe the most important part of any Machine Learning pipeline i.e. Feature Engineering.