Mazenon Stack Overflow • Recommends
On “How to train model with multiple GPUs in pytorch？”
There are two different ways to train on multiple GPUs:
Data Parallelism = splitting a large batch that can't fit into a single GPU memory into multiple GPUs, so every GPU will process a small batch that can fit into its GPU
Model Parallelism = splitting the layers within the model into different devices is a bit tricky to manage and deal with.
Please refer to this post for more information
To do Data Parallelism in pure PyTorch, please refer to this example that I created a while back to the latest changes of PyTorch (as of today, 1.12).
To utilize other libraries to do multi-GPU training without engineering many things, I would suggest using PyTorch Lightning as it has a straight…