Why couldn’t we use directly the test set to evaluate our model and adjust hyperparameters?

Q&A, Help and Support Data Science & ML Help
#1

Hi Prajwal,

Why couldn’t we use directly the test set to evaluate our model and adjust hyperparameters? I am still confused about the utility of the validation set. I think that the test set can also do the job of the validation set which is smaller. After all, we want to get a minimum loss on the test set, so if we evaluate our model and adjust it directly based on the test set, it will be better, won’t it?

Thank you very much and have a nice day,

Zheyu XIE

1 Like
#2

I had the same doubt for a long time.

  1. You’ll get a better sense of this (train/validation/test) split in the Kaggle competition. You may not always have ready access to the labels for the test set. Sometimes this evaluation might be done by a third party or an external system.
  2. As you use the validation set to adjust hyperparameters several times, you might end up overfitting to the specific validation set you’re using, and your model may still perform poorly in production. The test set provides a final sanity check at the very end that your model is in fact generalizable. Remember, our end goal is not to get the best result on your validation or test set, but to ensure that your model works well on real world data.
    Watch this for more: https://www.coursera.org/lecture/deep-neural-network/train-dev-test-sets-cxG1s
    Courtesy- @aakashns
1 Like