Start Small and Grow

During working on my Course Project with a large dataset, I became frustrated with the speed of processing and how many times I had to rerun my notebook and re-process cells. Of course, this re-processing was down to my learning and correcting mistakes.
Necessity being the mother of invention, I decided to reduce the size of my dataset to a few thousand rows, which then allowed me to process cells and notebook very quickly. When I was happy that the notebook ran without errors and produced reasonable results, I gradually increased the size of my dataframe, running the notebook after each iteration.
I hope this simple tip helps you avoid the frustration of slow notebook processing.

1 Like