Giant US supercomputer uses 50,000 GPUs to model 13.4 trillion dark particles

Researchers at the US Department of Energy’s Argonne National Laboratory completed two of the largest cosmological simulations ever, modeling 13.4 trillion dark matter particles across a 28.6-billion-light-year region of the universe.

The simulations ran on Aurora, Argonne’s exascale supercomputer, using 8,100 computing nodes and about 50,000 graphics processing units (GPUs) per calculation. The team completed both runs and much of the data analysis in about four days, creating enormous datasets designed to help scientists investigate the universe’s earliest moments and interpret observations from major astronomical surveys.

Led by Argonne computational scientist Nicholas Frontiere, the project focuses on understanding cosmic inflation, a proposed period of extremely rapid expansion shortly after the Big Bang. Although inflation is a central part of modern cosmology, the physics that drove it remains uncertain.

Simulating the universe to investigate cosmic inflation

According to the researchers, inflation stretched tiny quantum fluctuations across vast distances in the early universe. Over billions of years, gravity amplified these small density variations, helping produce the large-scale cosmic web of galaxies and dark matter observed today.

Dark matter cannot be observed directly through ordinary light, but its gravitational influence helps shape the distribution of galaxies. The simulations use particles to trace dark matter’s density and velocity, allowing scientists to model how cosmic structures evolved.

The team is particularly interested in primordial non-Gaussianity: subtle departures from the simple statistical distribution predicted by standard inflationary models. Detecting such deviations could help distinguish between competing explanations for inflation and reveal more about the universe’s earliest conditions.

However, any potential signal is expected to be faint and can be obscured by cosmic variance, the statistical uncertainty that arises when observations cover only a finite portion of the universe. Researchers therefore need simulations that combine enormous volumes with enough resolution to capture dark matter halos, the structures in which galaxies form.

Why Aurora’s computing power matters

Aurora’s architecture helped the researchers tackle both requirements. The machine has 10,624 nodes and 63,744 GPUs, alongside high-capacity memory and a storage system designed to handle enormous data flows.

The simulations generated more than 85 petabytes of data through Aurora’s Distributed Asynchronous Object Storage system, primarily for checkpointing. These checkpoints allow a calculation to resume after an interruption instead of restarting from scratch. The team retained approximately 8 petabytes for scientific analysis.

The team used Argonne’s Hardware/Hybrid Accelerated Cosmology Code (HACC), developed to run large-scale cosmological simulations across different supercomputing architectures. Unlike an earlier project on Oak Ridge National Laboratory’s Frontier system, which modeled gravity alongside gas, stars, and other processes, the Aurora simulations focused on gravity alone. That approach allowed the researchers to cover a larger cosmic volume with more particles.

Preparing for the next generation of sky surveys

The two simulations use the same cosmological model and non-Gaussian signal, but their initial phases are flipped. Analyzing them together helps cancel some random variations and isolate the underlying physics. The signal strength is also hidden, allowing researchers to test analysis methods without knowing the intended answer beforehand.

The resulting synthetic observations will help scientists validate workflows and identify potential biases before applying them to real data from projects such as the US Department of Energy-supported Dark Energy Spectroscopic Instrument (DESI) and NASA’s SPHEREx mission.

By generating realistic mock observations across the sky, the simulations could help researchers extract more reliable measurements from these surveys. Their scale also gives scientists a powerful testbed for examining subtle signatures of the early universe.

The project received computing time through the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment program. As DESI, SPHEREx, and other surveys gather increasingly detailed observations, simulations of this scale could help bridge the gap between theoretical models and the universe astronomers actually see.

Source: https://interestingengineering.com/space/aurora-supercomputer-massive-universe-simulations