Science on day one. That’s the goal for each new supercomputer introduced at the U.S. Department of Energy’s (DOE) Argonne National Laboratory, including its forthcoming exascale system, Aurora.

But how do you prepare for a machine before it is even built?

With Aurora’s 2021 arrival date drawing closer, the Argonne Leadership Computing Facility (ALCF), a DOE Office of Science User Facility, has been ramping up its efforts to ready the system and its future users for science in the exascale era.

In late February, the ALCF hosted its third Aurora hands-on workshop in the past year. Around 100 researchers participating in the ALCF’s Aurora Early Science Program (ESP) and DOE’s Exascale Computing Project (ECP) attended the three-day workshop to advance their code and software development efforts for the Intel-Cray system.

“The primary goal of the workshop was to bring project developers together with Intel and Argonne experts for hands-on work with their applications,” said Tim Williams, manager of the Aurora ESP and deputy director of Argonne’s Computational Science Division.

Over the past year, Intel has unveiled new architectural details and tools that are helping researchers to accelerate their preparatory work. Intel’s initial oneAPI beta software toolkit aims to simplify application development across diverse architectures. Existing Intel graphics processing units (GPUs) are helping researchers gain a better understanding of Aurora’s “Ponte Vecchio” GPU architecture, and serve as a development platform.

With access to the early Aurora software development kit (a more frequently updated version of the publicly available oneAPI toolkit) and Intel Iris (Gen9) GPUs through Argonne’s Joint Laboratory for System Evaluation, the workshop attendees had an opportunity to test code performance and functionality using programming models that will be supported on Aurora.

Getting some hands-on time compiling and executing application kernels on Intel GPUs was one of the primary draws for Jack Deslippe, applications performance group lead at the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility at Lawrence Berkeley National Laboratory. Deslippe is the lead developer of BerkeleyGW (a massively parallel simulation package used to compute electron excited-state properties) and co-principal investigator of an Aurora ESP project that will use simulation and machine learning tools to identify new materials for solar cells.

“We succeeded in building and running a couple of central BerkeleyGW C++ kernels on Intel Gen9 GPUs,” said Deslippe, who also serves as the ECP’s application development lead for chemistry and materials science. “In particular, we were able to execute a kernel that utilizes OpenMP offloading to the GPU and verify that correctness and performance is consistent with the fraction of peak FLOPS (floating operations per second) on Gen9 that we anticipate. While this was accomplished for a kernel and not the entire application, the results are very encouraging.”

Argonne physicist Walter Hopkins is leading an ESP project that will use Aurora to run simulations of particle interactions for the ATLAS experiment at CERN’s Large Hadron Collider (LHC). The project, which was originally led by the now-retired Argonne Distinguished Fellow Jimmy Proudfoot, is working to develop exascale workflows, algorithms, and machine learning capabilities to advance the search for new physics discoveries at the LHC.

“The amount of simulation needed for the High-Luminosity LHC (HL-LHC) upgrade will require the power of Aurora and future exascale systems,” Hopkins said. “Accurate and sufficiently large simulation samples will be essential in searches for new physics at the HL-LHC. We also want to prepare ATLAS data to be used for deep learning and hyperparameter optimizations on Aurora.”

Hopkins attended the workshop to learn about the deep learning libraries that will be available on Aurora and to work on porting their fast calorimeter simulation code, FastCaloSim, to Intel GPUs using Data Parallel C++ (DPC++), an extension of C++ that incorporates the SYCL programming model and other new features. He and his colleagues were able to make progress on both fronts.

“We had some valuable discussions on the capabilities of DPC++,” Hopkins said. “It was also very helpful to get prompt feedback on bugs and advice on structuring current and future aspects of our code.”

Deslippe also appreciated the opportunity to work directly with the Argonne and Intel experts and to hear about the progress that other ESP and ECP teams were making.

“Getting questions answered and bugs fixed in real-time leads to a really productive experience,” Deslippe said. “There is a psychological effect of being surrounded by a group of people with common goals and motivations that inspires you to get as much out of your code as you can.”

Looking ahead, the ALCF will continue its Aurora training efforts with some web-based events in the coming months, including the next installment of its quarterly ESP webinar series in June. The Argonne-Intel Center of Excellence also continues to host Aurora hackathons, with some virtual sessions being planned for 2020. These multi-day collaborative events pair individual ESP teams with Argonne and Intel staff members to further advance efforts to port and optimize applications for Aurora.

The Argonne Leadership Computing Facility (ALCF), a U.S. Department of Energy (DOE) Office of Science User Facility located at Argonne National Laboratory, enables breakthroughs in science and engineering by providing supercomputing resources and expertise to the research community. Supported by DOE’s Advanced Scientific Computing Research (ASCR) program, the ALCF is one of two DOE Leadership Computing Facilities in the nation dedicated to open science.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation's first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America's scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy's Office of Science.

The U.S. Department of Energy's Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science