High-Performance Computing in the Year of COVID-19

By early March, the SARS-CoV-2 virus had begun spreading in Germany, and previously unimaginable measures were soon implemented to control a deadly new pandemic. On March 13, the state of Baden-Württemberg closed all schools to on-site learning, and on March 16, the University of Stuttgart recommended that all employees not essential for the operation of campus facilities work from home. Although case numbers in Germany fell over the summer, the absence of an effective vaccine and arrival of an even more dangerous second wave of disease in the fall meant that many restrictions remained in effect for the rest of the year. Even at the end of 2020, just a few employees were onsite at HLRS daily, with the majority working remotely from home, networked by email and online videoconferencing software.

The unprecedented disruption caused by the COVID-19 pandemic affected HLRS in many ways, forcing the center to quickly adapt to a new reality. At the same time, however, it highlighted the critical roles that high-performance computing (HPC) and other advanced computing technologies now play in addressing public health emergencies. For HLRS, 2020 was a year of rising to meet many new kinds of challenges, and foreshadowed how supercomputing could evolve in the coming years to prepare for other kinds of crises.

The computational sciences have come to play an essential role in disciplines such as physiology, immunology, virology, and drug discovery. Scientists now routinely use simulation, machine learning, and high-performance data analytics to study many kinds of pathogens and how they cause infection, gaining essential information for identifying potentially effective drugs, vaccines, and other strategies for fighting disease. Using algorithms designed to run on supercomputers, scientists generate testable predictions of biological activity that occurs at scales that are too small or that is too complex to study using experimental approaches alone. This, in turn, accelerates research, as it helps investigators to focus their wet lab experiments on specific hypotheses that have the highest likelihood of success.

In 2020 the international scientific community deployed computational approaches in many different ways to address the COVID-19 pandemic. Some, for example, used such approaches to gain a better understanding of molecular characteristics of the virus. Others conducted virtual drug screenings, using large data sets to predict which existing drug compounds might be repurposed to incapacitate the virus. Clinical scientists began using computational approaches to analyze electronic patient records, looking for important features in the data that could reveal more personalized treatment strategies. Epidemiologists also used data to track and predict how disease spreads in populations, providing insights that public health officials can use to prepare for and contain an outbreak. An unprecedented global scientific response using these and other approaches rapidly produced a massive knowledge base that helped guide the search for vaccines, cures, and effective public health interventions.

At the same time that HLRS was going on lockdown in March, the center stepped forward to provide computing muscle for COVID-19 research. Stuttgart quickly joined its partners in the Gauss Centre for Supercomputing — the alliance of Germany's three national supercomputing centers — in announcing expedited access to its systems for scientists investigating the virus and its accompanying disease. Although HLRS is perhaps better known for its support of computational engineering, its HPC infrastructure is also suitable for health-related research, and in the following weeks and months, scientists came forward and began using Hawk to contribute to the fight against the novel coronavirus. Eight COVID-related projects made use of the HLRS system over the course of the year, while the center's invitation for additional users remains open.

In several of the approved projects, scientists focused on gaining a better understanding of the virus's physical structure, including features that are involved in the infection of human cells. This included research aimed at modeling the COVID-19 spike protein, which initiates infection when it interacts with the human ACE2 receptor on the surfaces of cells. Others used an approach called molecular dynamics to simulate at extremely high resolution how proteins on the surface of the virus change shape. Understanding these phenomena better could help identify drugs that could inhibit the interaction between the virus and its human host as it gains entry to cells. Another user of HLRS's systems received an allocation to investigate specific factors in the clinic that lead to negative outcomes in some patients. Because of the complexity of these kinds of problems, HLRS's computing systems provided essential tools for this research.

Beginning with the arrival of the COVID-19 pandemic, Germany's national strategy focused on containing the spread of the virus to prevent overwhelming burdens on hospital capacity. This involved implementing, and at times relaxing, a wide range of interventions, including closing businesses, prohibiting public events, issuing stay-at-home orders, and requiring the wearing of protective masks in public places. Throughout 2020 this strategy was largely successful in preventing unmanageable demand for intensive care units, although the arrival of the pandemic's second wave once again forced the country to implement new restrictions to prevent a flood of critical cases. Until vaccines are effectively deployed, it is likely that such cycles will continue.

In an effort to provide tools that can help public health agencies and governments to determine what protective interventions are needed, HLRS researcher Ralf Schneider collaborated with investigators at the German Federal Institute for Population Research (Bundesinstitut für Bevölkerungsforschung, BiB) to develop software for predicting spikes in demand for intensive care units (ICUs) across Germany up to eight weeks into the future. This COVID "weather report" could be used to help strike a sustainable balance between protecting public health and minimizing other social, economic, and personal impacts of the pandemic. A preprint publication describing the tool was published in December on the medRxiv preprint server for the health sciences.

The software was developed on HLRS's Hawk supercomputer, but is designed to be usable by data scientists anywhere. As Schneider explained, "The model is based on epidemiological parameters that correspond to available data in Germany. However, the code is structured so that other countries could potentially adapt it to their own situations, assuming the relevant data is available."

Also important to the success of this effort was EXCELLERAT, a European project led by HLRS that was created to support the engineering industry in the movement toward exascale supercomputing technologies. HLRS staff, together with EXCELLERAT partner SSC-Services, showed that technologies developed for the engineering branch could also be useful in addressing public health challenges. The collaborators implemented a secure data transfer tool that enabled the BiB researchers to quickly move data between Hawk and systems at their offices in Wuppertal, giving them access to HLRS's capabilities for large-scale data analysis without needing to be present onsite.

In addition to providing high-performance computing time to individual scientists for their research, HLRS also leads and contributes to dozens of funded research projects addressing technical challenges facing high-performance computing. In one new project approved in the summer by the European Commission's Horizon 2020 program, HLRS will help develop an important tool for managing public health during pandemics.

Since March 2020, medical centers in Europe and around the world have accumulated masses of COVID-19 patient data documenting patient characteristics, treatment plans, and disease progression. Initially, however, these data were largely gathered and saved by different healthcare providers in their own local databases, with few connections between them. Because modern biomedical and epidemiological studies gain statistical power by using larger datasets, bringing this information together would provide investigators with a valuable resource.

The new project, called ORCHESTRA is being led by Prof. Evelina Tacconelli at the University of Verona, Italy, and includes 27 partner institutions from 15 countries in Europe, Africa, South America, and Asia. It will develop a multilayered data infrastructure for collecting and sharing COVID-related patient data from across Europe and other parts of the world. As part of this effort, HLRS will work together with scientists at the high-performance computing centers CINECA (Italy) and CINES (France), to develop the data management infrastructure.

According to the ORCHESTRA plan, national data hubs in participating countries will collate collections of data, while a centralized, cloud-based portal will enable researchers to access, share, and link together data stored in the various national hubs. The resulting virtual cohort, potentially including hundreds of thousands of patients, will enable a range of multidisciplinary studies investigating COVID-19 genetics, epigenetics, immunology, and epidemiology, among other topics.

In addition to helping design the infrastructure, HLRS will also provide expert support in standardizing data collection, storage, and curation across all of the national data hubs. Such harmonization is essential for ensuring that data sets originating in many different places can be seamlessly integrated in ways that maximize their utility and impact.

The ORCHESTRA team anticipates that this new platform will support studies to improve public health and vaccine strategies. It could also provide a model for collecting and sharing data in the event of future pandemics.

More locally, the COVID crisis affected HLRS in other practical ways. Because of "social distancing" regulations that prohibited large gatherings of people, it became impossible for the center to offer its busy calendar of HPC training courses in a normal classroom setting. Instead, within a few weeks of the beginning of the lockdown, instructors began revising courses for an online format, holding them using the videoconferencing platform Zoom.

Despite the challenge of preserving the hands-on, interactive nature of the courses, this digitalization of HLRS's training program had unique benefits. Not only did the center avoid a significant decrease in the total number of course participants it sees in a normal year, but the courses also reached an audience across a broader geographical range than normal, with 38% of all participants logging in from outside Germany.

"Although we obviously look forward to the time when we can once again offer courses in person, the experience over the last several months has shown how online learning could enhance our abilities to increase expertise in the use of high-performance computing," said Dr. Rolf Rabenseifner, who leads HLRS's training program. "At this point, we anticipate that HLRS could potentially continue to offer online courses in parallel with its core training program even after the current pandemic has passed." In this case, the necessity to adapt to COVID-related restrictions could have unforeseen future benefits.

Similarly, HLRS's user support team was forced to quickly adapt its April code porting and scaling workshop to an online format. Using videoconferencing software, scientific computing experts at HLRS worked one-on-one with individual teams of scientists in dedicated breakout rooms to address problems they encountered with their codes.

"In optimization workshops, we do not just hold a lecture or show a set of slides," said Björn Dick, a member of the HLRS user support team. "This new approach enables us to work collaboratively online, to really dig into the code together and make changes. By using remote control functionality, I can briefly take over control of a user's screen, write in some code to explain what I mean, and the person on the other side immediately understands it and can continue on his own."

The success of the event also inspired the HLRS team to begin using videoconferencing tools in other settings where in-person meetings weren't practical. "In one sense COVID forced us to change the way we work," said PUMA Department Leader Dr. Thomas Bönisch, "but it also enabled us to develop a user support solution we hadn't considered previously. This has improved our ability to help HLRS's system users even outside the context of the performance optimization workshop."

Unfortunately, COVID-19 is unlikely to be the last pandemic that humanity will face. Moreover, considering global atmospheric and social instabilities likely to created by climate change, ongoing risks of chemical or nuclear accidents, vulnerabilities to natural disasters, and other global threats, simulation and related fields such as artificial intelligence will likely be important in quickly responding to new kinds of problems.

To bolster HLRS's ability to meet these challenges, computer hardware manufacturer AMD, whose processors make up the core of Hawk, announced in September 2020 a donation of 10 new computing nodes to HLRS as part of its AMD COVID-19 High Performance Computing Fund. The additional computing power, consisting of AMD Radeon Instinct graphic processing units and due to be delivered in early 2021, will be reserved for pandemic-related research. In the future, the new resources will also expand HLRS's capacity to address urgent computing needs when sudden crises arise.

As HLRS Director Prof. Michael Resch explains, "The COVID-19 crisis has been a big wakeup call in Germany and across Europe, showing that new challenges can develop very suddenly and have widespread impacts across societies. Because high-performance computing provides a backbone for tools that accelerate scientists' ability to address these challenges, it is clear that investments and preparation are needed now to be ready the next time a similar challenge comes along. At HLRS and at other HPC centers across Europe we anticipate that implementing new resources and strategies to be ready to handle emergency situations where simulation and data analysis can help will be a big focus in the coming years."

Supercomputers alone will never be the cure for threats like the COVID-19 pandemic, but they will continue have important roles to play in preparing for and successfully addressing them.

— Christopher Williams