Why cant you test the hyperparameter tuning on the test set itself, instead of testing on a separate validation dataset?

Hey Prajwal, I have a question, why cant you test the hyperparameter tuning on the test set itself, instead of testing on a separate validation dataset?

1 Like

By using test set to do any kind of hyperparameter tuning, you are trying to improve your score on the test set, the more you tweek the more accurate you will get on the test set, and finally when you deploy on production you will see the model performs poorly, because all the time you spent on tweeking hyperparameter you’re essencially overfitting to test set.

1 Like

The test set will only learn to recognise itself. A separate validation set tests it for “the unkown”

Because you want validate that your model is learning well with validation set and to evaluate whether it has learnt generalized features you’ll be using test set. If you tune your model to test set, then again you will not have a idea of what your model will perform for unseen/real world values