Python has become the de facto language for scientific computing. Programming in Python is highly productive, mainly due to its rich science-oriented software ecosystem built around the NumPy module. As a result, the demand for Python support in High Performance Computing (HPC) has skyrocketed. However, the Python language itself does not necessarily offer high performance. In this work, we present a workflow that retains Python's high productivity while achieving portable performance across different architectures.

The workflow's key features are HPC-oriented language extensions and a set of automatic optimizations powered by a data-centric intermediate representation. We show performance results and scaling across CPU, GPU, FPGA, and the Piz Daint supercomputer (up to 23,328 cores), with 2.47x and 3.75x speedups over previous-best solutions, first-ever Xilinx and Intel FPGA results of annotated Python, and up to 93.16% scaling efficiency on 512 nodes. We will also dive into modern machine-learning and large-scale AI workloads and their data-centric optimizations.

Speaker Bio: Torsten Hoefler is a Professor of Computer Science at ETH Zurich, Switzerland. His research aims at understanding the performance of parallel computing systems ranging from parallel computer architecture through parallel programming to parallel algorithms. He is also active in the application areas of weather and climate as well as machine learning focusing on distributed deep learning.

Torsten won best paper awards at the ACM/IEEE Supercomputing Conference 2010, 2013, 2014 (SC10, SC13, SC14), EuroMPI 2013, IPDPS 2015, ACM HPDC’15, and other conferences. He published numerous peer-reviewed scientific conference and journal articles, authored chapters of the MPI-2.2 and MPI-3.0 standards, and co-authored the book “Using Advanced MPI”. For his work, Torsten received the SIAM SIAG/Supercomputing Junior Scientist Prize in 2012 and the IEEE TCSC Young Achievers in Scalable Computing Award in 2013. He received the Young Alumni Award 2014 from Indiana University, the Latsis prize of ETH Zurich, and the ACM Gordon Bell Prize 2019. Torsten was elected into the first steering committee of ACM’s SIGHPC in 2013. He is an IEEE Fellow. Additional information about Torsten can be found on his homepage at htor.inf.ethz.ch.