Using Bigger pre-trained models, K-Fold Cross-Validation, CutMix, MixUp, and Ensemble learning
Published in · 10 min read · Feb 22, 2021
--
A project manager once told me about the 80/20 rule. He complained that the last part of a project was taking too long. Implementing the last 20% of features was taking over 80% of the time.
Vilfredo Pareto called this the 80/20 rule or the Pareto principle. It states that 20 percent of your efforts produce 80 percent of the results.
The 80/20 rule also holds for improving the accuracy of my deep learning model. It was straightforward to create a model with 88% accuracy. I have the feeling that improving it an extra 3 percent to make it to the top of the leaderboard will take a lot more time.
If you do not know what I am talking about, I invite you to read my previous article. That article ends with five possible techniques to improve the model’s accuracy. I learned these five techniques from the Kaggle community.
- Use bigger pre-trained models
- Use K-Fold Cross Optimization
- Use CutMix to augment your images
- Use MixUp to augment your images
- Using Ensemble learning
I tried each of these techniques and combined them. This is what happened.
All of the source code is available in this GitHub repository.
Before, we used EfficientNet-B3. This model was a good trade-off between performance and accuracy. See below. But EfficientNet offers other models that provide even greater accuracy — for example, EfficientNet-B4.
These more complex models have more parameters. More parameter needs more computing power and memory during training. I started with EfficientNet-B4, which gave an excellent result. The validation accuracy went up to 90%, and the validation loss to 0.32.
