The NASA Center for Climate Simulation (NCCS) has almost tripled the peak performance of its supercomputer Discovery, reaching over 3.3 petaflops, or 3.361 trillion floating point operations per second.

This unprecedented upgrade is necessary to meet the explosion of modeling applications of Earth science at NASA. First step was to choose the best equipment able to run the GEOS-5 (Goddard Earth Observing System Model) and NU-WRF (NASA Unified Weather Forecasting and Research) applications.

More cores for better performance
The adopted SGI Rackable cluster-based solution is intended to replace parts of the Discover supercomputer built in 2011 to establish three Scalable Compute Units (SCU) exploiting the current generation of Intel Xeon Hawsell E5-2697 v3 processor with 14 cores each, for a cumulative total of 64,512 processor cores. Teams from the NCCS constantly keep in mind the needs of its user community during the technical definition of the new components of Discovery. “Gradually, as the number of cores increases, memory requirements also grow forcing us to review the source code of our applications to take advantage” said Daniel Duffy, the director of the unit HPC NCCS. Within each of the three SCUs, each processor core is flanked by 4 gigabytes of memory. The total being 138 Terabytes.

An earth model that is gaining in precision
The results calculated by the overall calculation model GEOS-5 at 12 km resolution are carried on to the new clusters to calculate regional models in better definition of 24, 12 and 4 km. The researchers then compare the accuracy gain on weather phenomena such as northwestern US snow storms, summer storms and the western states rivers in winter. Another architectural improvement is the presence of a full speed interconnect fabric. On newer servers, each node consists of 28 cores which communicate with other nodes via an FDR (Fourteen Data Rate) Infiniband link, permitting speeds of 56 gigabits per second.

“GEOS-5 calculation models become very large and require extremely fast processors and communication buses. If they are insufficiently dimensioned, this may become a handicap to the smooth running of the simulations to be calculated” insists Daniel Duffy. This data explosion due to the accuracy improvement of the calculated models also impacts Discovery’s storage requirements, which were increased to 33 Peta-bytes. The normal time of installation of an UCS averages a year. “When we’re done, we will have installed three SCU in 7 months, “said Mike Donovan’s Discovery system administrator.