Stuttgart, Germany, October 11, 2023 — Scientists at the High-Performance Computing Center Stuttgart (HLRS), working in collaboration with software developers at WIKKI GmbH as part of the EuroHPC Joint Undertaking-funded project exaFOAM, have successfully tested a new method that dramatically improves the scalability of OpenFOAM applications on high-performance computing (HPC) systems. Based on a newly developed concept that the researchers call the coherent file format, the approach simplifies data management for HPC file systems during simulations, and eliminates time-intensive steps such as data pre- and post-processing that slow down ordinary OpenFOAM simulation workflows.
Oct 11, 2023
See all press releases
Using HLRS’s supercomputer, Hawk, members of the exaFOAM team successfully completed a benchmark computational fluid dynamics (CFD) simulation in which OpenFOAM scaled efficiently with respect to input/output (I/O) performance to 4,096 nodes (524,288 CPU cores). This number doubles the size of a test run that HLRS completed in August 2023, and is more than four times the size of the previous scaling record for OpenFOAM.
The coherent file format was initially conceived by Dr. Henrik Rusche, CEO of WIKKI GmbH and one of OpenFOAM’s lead developers. It uses a strict sorting approach in laying out the computational mesh in a CFD simulation to create a global mesh comprehension. Processes involved in a large-scale simulation are subdivided into groups, each of which contains a single data-aggregating process that both has a fixed relationship to the data of the other processes within the group and is responsible for their input/output operations. In this way, the coherent file format transforms a large, fragmented collection of data distributed across processors into a smaller, more consecutive, and more meaningful set of “chunks” that are easier for the file system to manage.
For users of OpenFOAM — a free, open source software framework for CFD simulations that is widely used in fields such as automotive engineering, manufacturing, and the energy industry — the coherent file format promises to make simulation workflows run much faster, including on pre-exascale and exascale systems. By simplifying the file management required for typical OpenFOAM workflows, it will also make simulation more accessible to users in academia and industry who rely on more limited computing resources.
Whereas data pre-processing might take as much as a week in a typical large-scale simulation, researchers using this new method could have their results in just a couple of hours. This approach also avoids the time-consuming step of reconstructing the results of individual domains after a simulation is complete, as the structure of the mesh is already reflected in the coherent data structure in a global file layout. For high-performance computing centers, this approach will help to optimize the allocation of computing resources.
The developers of the coherent data method have made the code available on a free, open source basis to the global OpenFOAM community at the following link: https://code.hlrs.de/exaFOAM. Integration of the method into future releases of production branches of OpenFOAM is ongoing and is being supported by the developers of the coherent file format. They are also planning scientific publications describing the logic behind their methods, technical details about the benchmarking process, and best practices for users.
For more information, please visit: https://www.hlrs.de/news/detail/coherent-file-format-accelerates-time-to-solution-with-openfoam.
Weiß RG, Rusche H, Lesnik S, Galeazzo FCC, Ruopp A. Preprint. Coherent mesh representation for parallel I/O of unstructured polyhedral meshes. J Supercomputing.
The High-Performance Computing Center Stuttgart (HLRS) was established in 1996 as Germany’s first national high-performance computing center. As a research institution affiliated with the University of Stuttgart and a founding member of the Gauss Centre for Supercomputing, HLRS provides computing resources for academic users and industry. HLRS operates state of the art high-performance computing systems and provides advanced training in HPC programming and simulation. The center also conducts research to address key problems facing the future of supercomputing. Among HLRS's areas of expertise are parallel programming, numerical methods for HPC, visualization, cloud computing, high-performance data analytics, and artificial intelligence. Users of HLRS computing systems are active across a wide range of disciplines, with an emphasis on computational engineering and applied science.
The exaFOAM project aims at overcoming the current limitations of computational fluid dynamics (CFD) technology, especially in the exploitation of massively parallel HPC architectures. It is developing and validating a range of algorithmic improvements across the entire CFD process chain (preprocessing, simulation, I/O, post-processing). Effectiveness will be demonstrated via a suite of HPC Grand Challenge and Industrial Application Challenge cases, focusing on sectors in which engineering design through CFD has contributed strongly to industrial competitiveness and sustainability. Learn more at https://exafoam.eu.
Sophia Honisch, High-Performance Computing Center Stuttgart. Tel.: +49 711 685-68038. sophia.honisch@hlrs.de
ExaFOAM has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956416. The JU receives support from the European Union’s Horizon 2020 research and innovation program and France, Germany, Spain, Italy, Croatia, Greece, and Portugal.
Funding for Hawk was provided by Baden-Württemberg Ministry for Science, Research, and the Arts and the German Federal Ministry of Education and Research through the Gauss Centre for Supercomputing (GCS).
