Analyzing Performance and Scaling of DLRM Model in Distributed Run

The DLRM model is widely used in distributed machine learning applications. In this article, we will analyze the performance and scaling characteristics of the DLRM model in a distributed run. First, let's look at the maximum ranks to scale. The table shows that for different configurations, the maximum ranks range from 8 to 64. This indicates the maximum number of parallel processes that can be used to scale the DLRM model. Next, let's consider the total A reduce size. The table shows that the size ranges from 9.5 MB to 1047 MB. This represents the total size of the A reduce operation in the DLRM model. It is important to note that as the size increases, the performance of the model may be affected. Moving on to the strong scaling all-to-all volume, the table shows that the volume ranges from 15.8 MB to 1024 MB. This represents the amount of data that needs to be communicated between processes in the distributed run. It is crucial to optimize the communication overhead to achieve better performance. In the ML.Perf config, we observed contention with the terabyte dataset, leading to a slowdown in the embedding update operation. This slowdown can be up to 10 times compared to our race-free algorithm. However, after optimizations, we found that the embeddings take about 30% of the total time for small configurations, matching it with the MLP time. For the MLPerf config, embeddings take less than 20% of the total time. It is worth mentioning that the runtime per iteration for the small config decreased significantly from 4288 ms to 38 ms. This improvement in performance is remarkable. Additionally, previous work timed an NVIDIA V100 GPU at 62 ms for the same DLRM problem using the Caffe 2 frontend. However, it is important to note that the V100 GPU has more FP32-FLOPS and available bandwidth compared to the Skylake/Cascade CPU. Therefore, a fully-optimized GPU software stack can be expected to be faster than our optimized single-socket CPU version. In the end-to-end analysis of multi-socket/multi-node DLRM results, we aimed to answer questions about the performance and scaling of the DLRM model. By analyzing the performance characteristics and optimizing the model, we can achieve better scalability and efficiency in distributed machine learning applications. In conclusion, the performance and scaling of the DLRM model in a distributed run are crucial factors to consider. By understanding the characteristics of the model and optimizing it, we can achieve better performance and scalability in distributed machine learning applications.