In a truly democratic HPC world, what would you vote for?

It’s not a facetious question. Democratization’s been the metaphor of choice as the HPC market has matured and expanded, extending its reach beyond elite research labs to diverse project teams and end users who simply want to buy compute time to run their analytics.

As in any democracy, all participants in the computing ecosystem have a say in what happens. And many in HPC have been speaking up about their need for a platform where they can install, manage, and maintain HPC systems with ease.

Figen Ulgen, general manager of Intel’s HPC platform software group, announces the Intel HPC Orchestrator Advanced at SC16.

HPC has proliferated quickly, and traditional approaches to stack development have proven too fragmented to be workable. IDC has observed that initiatives by OEMs and other vendors to turn software components into HPC stacks have led to “considerable duplication of effort that may no longer be tolerable for keeping pace with escalating requirements.”

When anyone can change the hardware ingredients and software components of an underlying platform, developers are basically walking in the dark. Many ISVs have found themselves unable to replicate issues yet being charged with solving them, debugging them, and producing updates. Many have been constrained to delivering niche services.

A call for consistency

Intel, one of the key players in HPC, listened to the frustrations of developers and heard a common complaint: inefficiencies were costing everyone dearly in time, money, and lost opportunities. In conversation after conversation with people in diverse environments around the globe, complaints pointed to a root cause, one as old as the computing industry itself: lack of consistency.

Over several decades Intel, like many organizations and businesses, addressed the problem by crafting standards, backing initiatives, and leading consortiums, all of which can reach solutions, given the time and effort to bring disagreeing parties to agreement. Clearly, the HPC community, already in tacit concord, needed something different – and without delay. Intel answered the call by instigating the creation of an Open Source Community project called OpenHPC.

Voting with their feet

Launched under the Linux Foundation umbrella in November 2015, OpenHPC exists to create, through collaboration, a strong, stable, consistent software platform – specifically, a software management stack – that enables innovation.

The Foundation made good sense as a sponsoring organization. OpenHPC is driven by a desire to aggregate a number of common ingredients required to deploy and manage High Performance Computing (HPC) Linux clusters. This, in turn, makes deploying an HPC system easier. The stack is architected for scalability all the way to exascale computing.

While the benefit to the software community may be apparent, the boost for the hardware community isn’t hard to discern. Software innovation leverages advances from hardware, and hardware vendors have an interest in seeing HPC systems achieve exascale performance.

To help resolve the consistency issue, Intel invited software vendors, research institutions, equipment manufacturers, supercomputing teams, and end users from around the world to come together, extend the conversation they’d already been having, and form an initial group. The response was remarkable.

While it took some convincing that Intel was stepping forward as an open source community participant, not an initiative driver, 32 acceptances arrived the day the kickoff meeting invitation went out, rising to 42 in just a few months. Today, 30 participants, including ARM, are official (financially contributing) members of OpenHPC, with more applying to join. It’s also attracted positive attention from U.S. government projects that want to see it flourish as an all-inclusive community.
To kick-start the work, Intel seeded the Community with an integrated stack that OpenHPC demoed at recent supercomputing events. It’s being embraced, downloaded at the rate of hundreds per month.

Access for everyone

The appeal of OpenHPC is its modular approach to middleware. The Community pulls relevant, reliable, validated versions of system management components from upstream communities and acts as a midstream liaison to integrate a stack that becomes the base for HPC system management software. The OpenHPC software stack includes provisioning tools, I/O clients, development tools, and a variety of scientific libraries.

In fact, what distinguishes the OpenHPC stack is that it allows system vendors and highly sophisticated end users to incorporate their preferences for particular components – SLURM instead of Altair PBS Pro as a scheduler component, for example. Because no one is subjected to a lowest common denominator, users across the spectrum can take advantage of the stack.

The OpenHPC Community is very deliberately integrating all the components necessary to install, manage, and maintain an HPC system. As of today, the system is being tested against the latest versions of the RedHat, CentOS, and SUSE operating systems.

Power to the people

Having access to a consistent, modular stack available is a boon. But how do you take advantage of its power?

If you’re responsible for an enterprise system you’re more likely to want support with SLAs and a big-name company behind it than take a chance on implementing an open source product, even one as impressive as OpenHPC, on your own.

Intel recognized that users would need a product to make OpenHPC’s innovation available across the ecosystem and fully democratize HPC. In the near term, the company is addressing the practical issue of taking OpenHPC into production-ready environments with the launch of a product it calls Intel® HPC Orchestrator.

Intel HPC Orchestrator modular view

HPC Orchestrator is the Intel distribution of the OpenHPC stack, optimized for Intel hardware. It’s a modular toolset that improves the setup and management of systems, and enables testing at scale as well as simplified deployment.

Intel backs its HPC Orchestrator with professional services and support – what you need whether you’re a manufacturer who can’t lose a day to breakdowns or a shop too small to find quick help with homegrown applications.

The first of three levels of product, and available through OEM partners Dell and Fujitsu, is Intel HPC Orchestrator Advanced. Designed for Top500 technical and commercial users with vertical HPC applications, it addresses medium to large HPC systems that require application compatibility along with a high measure of configurability or customization.

The two upcoming levels of Intel HPC Orchestrator, Custom and Turnkey, aim at the lower and higher tiers bracketing the Advanced level.
Intel HPC Orchestrator Turnkey, as the name implies, will support new HPC users looking for turnkey management and deployment. Its software stack will make it easy to develop and deliver small- to medium-scale HPC systems compatible with a broad range of ISV applications.

Intel HPC Orchestrator Custom, for Top50 and high-end HPC users, will implement a highly scalable hierarchical system architecture to address the leading-edge requirements of the world’s most scalable systems.

Intel HPC Orchestrator, at its various levels, puts the benefits of HPC in the hands of people across the ecosystem. IT administrators can reduce the R&D needed to build and maintain a fully integrated stack and more easily manage clusters from multiple vendors. ISVs can spend less time retesting applications as new components are released. OEMs can focus on providing differentiation in the top of the stack instead of building and maintaining one that’s fully integrated. End users, likewise, can direct their energy toward work rather than testing.

As a key software component of the Intel® Scalable System Framework, Intel HPC Orchestrator serves as the glue that connects the goodness in the hardware to the software. That means hardware innovations reflected in software can put everyone on a faster path to exascale.

Power from the people

The word democracy traces back to the Greek words demos, meaning “people,” and kratia, meaning “power.” It’s what we think of – rightly – as all people having equal power. Within HPC, democratization has an even richer meaning: working together, the people in the community can actually create greater levels of power, all the way to exascale.

Intel conceived of Intel HPC Orchestrator as a hub for innovation. Pre-integrated, pre-tested, and pre-validated, it acts as a virtual operating system for HPC software. By relieving the burden of testing and making applications work on myriad systems, Intel HPC Orchestrator frees developers to do what they do best: develop.

It’s easy to imagine the advances that could come from a university student with a brilliant approach to code. From a small company ready to try its breakthrough idea. Or a lab discovering the missing piece to a puzzling problem in a new module in the OpenHPC stack.

In the longer term, code modernized on a consistent platform could yield higher compute performance. Intel HPC Orchestrator provides the foundation for additional functionality that can take advantage of parallel computing. Scientific visualization tools such as Embree and Osprey have already been demonstrated on Intel HPC Orchestrator.

Reaching exascale performance or accommodating highly complex workloads like machine learning requires a strong, stable software platform where everyone can develop together, simultaneously and in collaboration, flowing in all their innovations.

Enabling power to – and from – the people in the HPC ecosystem is a lofty goal. It takes a deep commitment to achieving exascale and a conviction that equal access to OpenHPC enriches everyone.

Through its Intel HPC Orchestrator program Intel has brought the ecosystem closer to that vote-worthy, democratized goal.