'num_workers' argument in 'torch.utils.data.DataLoader'

I want to understand the num_workers argument in the torch.utils.data.DataLoader.

As far as I can understand, it has something to do with the loading of data into memory and num_workers>0 means something about subprocesses in the memory.
I want to know what do subprocesses mean and what difference will it make to provide the num_workers value 0 or more?

Does subprocesses mean that the data will be loaded into memory into the parts? If it is so, how is it different from the batching of data. Or, is it mean that the memory will be divided into parts. If it is so, what difference does it make?

Or, is it something else? :thinking:

Well, hello again :laughing:

num_workers describes how many threads will be used to load the data.

This is especially useful when working with images, because loading every image (especially for datasets with thousand of images) into memory would require a lot of such memory.

Instead of that, the images reside on the disk, and are loaded by the DataLoaders. After the batch is processed it is probably removed from the computer memory, so the next batch can be loaded without approaching the memory limit (no idea, maybe they’re cached but I doubt that).

Since reading up an image can take a while, it’s good to split this into separate CPU threads, which read each image independently. Each such thread runs a “worker” that loads the data, transforms it, and creates a batch.

num_workers equal to 0 means that only a single thread (the one executing the python code) will be also used to load the data. Which means multi-threaded CPUs (common nowadays) aren’t fully utilized.

PS. Also applies to any audio files etc. Basically anything that would be slightly impossible to load all at once into the memory.

Thanks again, for the answer!
I looked up CPU threads and learned a lot. Then I checked my own PC’s CPU thread. It has only 4 threads (i.e. 2 cores). I don’t know if it is good or bad. :laughing:

4 threads is ok, I think my PC has the same amount :stuck_out_tongue:

You could go up with 5 or 6 workers - sometimes one worker is performing something non CPU-intensive, so the operating system can allocate the power to other worker and still get good efficiency.

BTW. not always 2 cores = 4 threads. There are different configurations, sometimes 1 core/thread (but expensive). Anyway nice to see that you delved that much into this subject :stuck_out_tongue:

1 Like