This article originally appeared in AltEnergyMag.

How do you identify the ideal molecular structure for a solar absorber or, perhaps, for a drug supplied as a powder of molecular crystals? The computing power of one of the world’s most advanced supercomputers can help.

To accomplish this task, a collaborative research team from Duke University, University of North Carolina at Chapel Hill, and Carnegie Mellon University conducted innovative computational research using the Theta supercomputer at the Argonne Leadership Computing Facility (ALCF) to run predictive simulations of materials and speed up the discovery of new atomic structures of organic and organic-inorganic hybrid materials.

According to Dr. Volker Blum, Associate Professor in the Department of Mechanical Engineering and Materials Science at Duke University, “It is important to have an affordable way to manufacture materials with very specific properties, controlled at the atomic scale, that can be used in practical devices such as solar cells or other semiconductor applications. For instance, the advantage of organic-inorganic hybrid solar cells is that they are easier to manufacture because some of them can be solution processed at a much lower temperature range than is possible in silicon technology. Using supercomputers allows us to simulate organic-inorganic hybrid materials with highly tailored functionality and look at their electronic structure in a way that would not be possible otherwise. Predictive simulations of the structure and electronic properties of these materials or interfaces - so-called 'electronic structure theory' - greatly enhance our ability to identify the most promising organic-inorganic combinations faster, reducing the need for time consuming experimental synthesis attempts while navigating a complex materials space.”

The research was part of the Theta Early Science Program (ESP) to get the Theta HPC system ready for production. The universities worked with the ALCF on this project using FHI-aims, GAtor and ELSI software. These software packages allow efficient, large-scale predictions of the structure and properties of organic and organic-inorganic materials and interfaces [NM1] [VB1].

“We used the Theta supercomputer to simulate crystalline organic-inorganic materials with 400 atoms per unit cell or more with high accuracy, predicting their electronic structure and identifying promising materials for further study by our experimental collaborators. Using our research, we can tell our collaborators which molecules to make to realize a promising new semiconductor material, something that cannot currently be done without high-performance computing (HPC) simulation. Having such a resource is a huge advantage for designing a new material such as a potential new solar cell absorber or other semiconductor,” states Blum.

Predicting the Crystal Structure(s) of Molecular Materials using HPC Research

Organic photovoltaic (PV) research done by co-principal investigator Dr. Noa Marom (Assistant Professor of Materials Science and Engineering at Carnegie Mellon University) uses the GAtor massively parallel genetic algorithm to predict the structure of molecular crystals, which often have semiconducting properties. Using current electronic structure theory, one can predict electronic properties with reasonable success and high accuracy if one knows the underlying structure (the arrangement of atoms and molecules in a crystal). Knowing the right structure can also be central in more basic applications, such as figuring out in which form to best deliver a new molecular drug. This requires computations on many possible structures. No currently feasible quantum-mechanical method can exactly compute the energy of a given molecular crystal - the quantity that determines which structure is most stable. Generally, the more precise an approximation is, the more computer time is required. An approach building on hierarchies of different approximations with increasing accuracy is often the most practical approach.

Figure 1 shows the tricyano-1,4-dithiino[c]-isothiazole molecule arranged in different ways in a crystalline structure [NM1]. Running the simulation with GAtor resulted in identification of a set of 100 low-energy crystal structures. In panel (a), the left column shows examples of the possible structures, initially identified using a relatively cost-effective computational approximation. The middle column shows the energetic re-ranking of these structures by a more accurate approach. The right column shows an even higher (and more costly) level of theory. The experimentally observed crystal structure, highlighted in red, is ranked as the most stable only by the most accurate method. It is important to know not just the "lowest" energy structure, but which additional low-energy structures (so-called polymorphs of a molecular crystal) might exist. A lower energy structure means greater stability, but it may not have ideal physical properties for a given application. For example, certain polymorphs of pharmaceutical drugs may have unpleasant side effects, or certain polymorphs of a molecular semiconductor might transport current better than others.

Running Research on Photovoltaic Materials on the Theta Supercomputer

The Theta Cray X40 supercomputer is a massively parallel, many-core 11.69-petaflops system based on Intel Xeon Phi processors, all integrated by Cray’s HPC software stack. Theta has features that allow scientific codes to achieve higher performance, including high-bandwidth MCDRAM (300 - 450 GB/s depending on memory and cluster mode), improved single thread performance, potentially much better vectorization with Intel AVX-512 and large total memory per node.

Using FHI-aims, GAtor and the ELSI Software to Solve Scientific Processing Needs 

Blum began developing FHI-aims ("Fritz Haber Institute ab initio molecular simulations") over a decade ago. FHI-aims is a now a highly accurate, scalable all-electron electronic structure package with an active, globally distributed developer community. The team used FHI-aims to perform production simulations of materials with many hundreds of atoms in their unit cells and demonstrated scalability for model sizes up to several thousand atoms.

GAtor is a massively parallel, first principles genetic algorithm (GA) specifically designed for structure prediction of (semi)-rigid molecular crystals. GAtor has a highly modular structure that allows users to easily switch between or modify core GA routines for specialized purposes. GAtor can fully utilize high performance computing (HPC) architectures, by spawning several parallel GA replicas that read from and write to a common population of structures. For energy evaluations and local optimization of trial structures, GAtor uses dispersion-inclusive density functional theory (DFT) by interfacing with FHI-aims.

Large-scale simulations in FHI-aims on advanced architectures such as Theta are also facilitated by the ELectronic Structure Infrastructure (ELSI) software stack, developed in Blum's group and supported by NSF's "Software Infrastructure for Sustainable Innovation" (SI2) program. ELSI provides and enhances scalable, open-source software library solutions, focusing on a computationally critical step, known as an eigenvalue problem. “Using ELSI's interface to the ELPA (Eigenvalue SoLvers for Petaflop-Applications) software library helped speed simulation performance by reducing the bottleneck of large-scale eigenvalue problems. We were able to demonstrate simulations on a one million by one million dense matrix in about three hours,” states Dr. Álvaro Vázquez-Mayagoitia, Computational Scientist, ALCF.

Optimizations and Performance of Simulations on the Theta System

The teams found that the simulations ran quite well on Theta and that porting the code was relatively simple without extensive code changes. The key algorithms in FHI-aims scaled efficiently on massively parallel HPC architectures, such as the Intel Xeon Phi processor, relying almost exclusively on message passing interface (MPI) communications using internally shared memory in the Intel library. They found that performance results also depended on the choice of compilers and memory usage.

The Importance of Computational Materials Research in the Future

“Theta and computers like it currently place us in the position to predict the electronic properties of semiconductor materials, their interfaces and their nanostructures correctly, with sufficient throughput, and with the full predictive power of quantum mechanics for the right reasons. Computational materials science is virtually certain to see impressive advances thanks to Theta and similar machines,” states Blum.

Vázquez-Mayagoitia indicates “A number of changes are needed in the future. It is estimated that human activities consume globally 15 Terawatts per year (TW-year) of energy. In the next thirty years, we will need much greater energy up 50 TW-year to power all our devices. The renewable sources of energy such as biomass, wind or geothermal will not be enough to provide this kind of increase in power. It is vital that we harvest solar energy to provide the increase in energy needed. But the problem is the cost of solar cells. Our project showed the advantage of using supercomputers to help locate the most efficient crystals, like hybrid organic-inorganic ones, to use in solar cell production.”

According to Blum, “In computational material science, we currently have reasonably good simulation capabilities to predict the electronic structure of materials, or other functional properties, such as electronically excited states. In trying to predict the properties of a quantum condensed system that consists of electrons as well as nuclei, we often try to predict phenomena such as an unusual life time of a carrier. For research on large or condensed structures, such simulations can be too computationally intensive—it is critical to do more in our simulations than is currently possible. Additionally, we need to be able to run simulations that include bringing the nuclei back into the computational simulation, which is currently very computationally expensive.

“I believe supercomputers will develop to meet these needs in the future,” Blum continued. “For simulating real materials in our scientific work, we have not reached the limits of what we need to do. There is still a long way to go in catching up with real, experimental insights on a broader scale. That is good news because there are still important, interesting and undiscovered phenomena to research and develop in the field of materials. I am personally very excited about the future work in this field that provides a great playground for finding relevant materials that can be used in the real world.”

[NM1] Farren Curtis, Xiayue Li, Timothy Rose, Álvaro Vázquez-Mayagoitia, Saswata Bhattacharya, Luca M. Ghiringhelli, and Noa Marom, GAtor: A First-Principles Genetic Algorithm for Molecular Crystal Structure Prediction. JCTC (Journal of Chemical Theory and Computation), DOI: 10.1021/acs.jctc.7b01152, February 28. 2018.
https://pubs.acs.org/doi/abs/10.1021/acs.jctc.7b01152

[VB1] Chi Liu, William Huhn, Ke-Zhao Du, Alvaro Vazquez-Mayagoitia, David Dirkes, Wei You, Yosuke Kanai, David B. Mitzi, and Volker Blum, Tunable Semiconductors: Control over Carrier States and Excitations in Layered Hybrid Organic-Inorganic Perovskites, arXiv:1803.07230 [cond-mat.mtrl-sci] (2018). https://arxiv.org/abs/1803.07230