HPC Cloud Adoption Trends
Currently we are observing several important trends and initiatives fostering HPC in the Cloud:

• Benefits: The benefits for design & development engineers and their companies from using computing in the Cloud are undoubtedly huge (as explained above) and will drive adoption as soon as this is recognized by engineers and scientists.
• Software Vendors: There is a growing cloud acceptance by ISVs, particularly ANSYS announcing ANSYS Enterprise Cloud on Amazon AWS. There are other ISVs like CD-adapco with token-based cloud-licensing, or most elegantly COMSOL allowing free cloud usage for customers who have a network license.
• Consumer & Enterprise Clouds: Because cloud services are widely accepted in the consumer and enterprise communities (e.g. for ERP, CRM, administration), the acceptance of Cloud services in companies’ R&D departments seems to be a natural next step and will happen with a certain time delay.
• Supply Chain Pressure: Large manufacturing companies more and more expect their supply chain partners to perform high-quality end-to-end simulations on HPC systems.
• Big Data Analytics: Big data is a big candidate for clouds. Collecting data en-mass is one thing, but extracting insight and knowledge out of tera- or even peta-bytes of data delivers much higher value. This analytics process is a natural cloud candidate.
• 3D Printing: Experts expect 30 million people to enter the 3D printing field by 2025. Many of them will work with CAD and CAE, but not everybody would buy their own HPC server then.
• Digital Natives: All those kids born after 1995 are called ‘digital natives’ and they are growing up with very different understanding for privacy and security. Most of their digital activities already happen in the cloud, of course.
• Cloud-related Initiatives: And, last but not least, US and international initiatives like the Missing Middle strongly supported by companies like Intel, the National Center for Manufacturing Sciences (NCMS), the UberCloud HPC Experiment, Cloud service providers like Amazon AWS, Microsoft Azure, Nimbix, Sabalcore and many others, and electronic magazines like HPCwire, Desktop Engineering, Bio-IT World, and HPC Today are currently creating strong awareness for the benefits of in-house HPC and HPC as a Service in the wider SME market.

In summary what we foresee is the HPC in the Cloud market growing especially towards the long tail [8], with tens of millions of very small to medium size enterprises (VSMEs, including the one-man show) which can’t (or which won’t) afford to buy expensive HPC servers for their different simulations. They all are or will become potential candidates for HPC Cloud services.

And for existing service providers – be it ISVs or Cloud resource providers – these individual businesses are simply too small to be able to feed their expensive sales people, in contrast to a fully automated online marketplace which doesn’t have these high overhead cost. Therefore, naturally, software and cloud service providers are concentrating on the larger accounts, while the only way the tens of millions of VSME are and will be served is an online marketplace for engineering and scientific applications in the Cloud.

The HPC Cloud Marketplace – Field of Dreams ?
When we started UberCloud in 2012 it was the time in Silicon Valley when every entrepreneur and every venture capitalist was talking about start-ups like Uber and Airbnb. Although there were many online marketplaces before, these two shooting stars made it to every front page of every relevant news magazine because of their ingenious and disruptive approaches shaking the long-established taxi and hotel businesses. Influenced by this enormous hype it was quite obvious for an HPC veteran and a Cloud practitioner (that’s us) to give an online marketplace for HPC as a Service some thoughts. But then we found that there was no solid information about end-user experience nor reliable market analyses in HPC Cloud; nothing to build on. So we decided to abandon the Build It He Will Come approach which worked in the wonderful movie Field of Dreams but won’t work in HPC; engineers and scientists tend to be more realists …

Instead, from the very beginning, we selected a crowd-sourcing approach and invited our whole engineering and scientific community to participate in our experiments, using our experimentation platform and technology, and providing feedback. That way, successively, we developed the UberCloud community platform for HPC Cloud information, the HPC experiment (sponsored by Intel) with 175 free voluntary cloud experiments, the HPC container technology based on Docker, and finally the UberCloud Marketplace with currently 30 service stores and about 100 cloud service products.

Behind the Scenes of the UberCloud
In articles and market research in mid-2012 we mostly found educated guesses about cloud adoption being so slow in the scientific and engineering community while in many enterprises cloud applications were widely in use. So we decided to start a few real-life experiments together with engineers taking their applications to the cloud. The first Call for Participation in June 2012 attracted an amazing 160 companies from over 30 countries, and we were able to perform 25 UberCloud experiments.

The UberCloud Experiment provides an online platform for engineers, scientists, and their service providers to discover, explore, and understand the end-to-end process of accessing and using HPC in the Clouds, and to identify and resolve the roadblocks. End-users, software providers, resource providers, and HPC experts are collaborating in teams, jointly solving the end-user’s application in the cloud. That way we were able to conduct 13 successful experiments. The other 12 teams failed, because of different reasons, such as: the ISV was not able to support the experiment; the end-user didn’t get approval from management; the end-user dropped out because of a sudden in-house project; and more.

Until today (July 2015), the UberCloud Experiment has attracted more than 3000 organizations from 72 countries, allowing us to build 175 teams so far. Recently, the UberCloud Marketplace [9] has been added, and Intel sponsored 2 Compendiums (and a third one is in preparation) which contain more than 60 cloud case studies. SMBs who are aiming at developing better products faster are invited to join the free voluntary UberCloud Experiment to explore HPC as a Service, in the cloud [10].

UberCloud’s Holy Grail – Software Containers
In early 2013, after about 50 cloud experiments, and an in-depth analysis of the 22 experiments which stumbled across the cloud roadblocks and failed, we believe we found the solution: Linux containers. But instead of starting from scratch we turned to Docker in March 2013 and attended one of their very first developer workshops in San Francisco. And timing was perfect. Since then we are developing more and more HPC features into Docker which after two years recently resulted in our first UberCloud application containers for engineering and scientific applications.

UberCloud Containers are ready-to-execute packages of software. These packages are designed to deliver the tools that an engineer needs to complete his task in hand. The ISV or Open Source tools are pre-installed, configured, and tested, and are running on bare metal, without loss of performance. They are ready to execute, literally in an instant with no need to install software, deal with complex OS commands, or configure. The UberCloud Container technology allows wide variety and selection for the engineers because they are portable from server to server, Cloud to Cloud. The Cloud operators or IT departments no longer need to limit the variety, since they no longer have to install, tune and maintain the underlying software. They can rely on the UberCloud Containers to cut through this complexity. This technology also provides hardware abstraction, because the container is not tightly coupled with the server (the container and the software inside isn’t installed on the server in the traditional sense). Abstraction between the hardware and software stacks provides the ease of use and agility that bare metal environments lack.

The Cloud Challenges
However, HPC in the Cloud comes with challenges too: it’s a new business and working paradigm; security, privacy, and trust in service providers can be an issue; conventional software licensing is now slowly including the pay-per-use or subscription model; Internet bandwidth often depends on the ‘last mile’ unable to accommodate heavy data transfer; unpredictable costs of cloud computing can be a problem in securing a budget for a given project; and there is often a lack of easy, intuitive self-service access and use of cloud resources. But in the following we will describe how we were able to reduce or even remove most of these cloud challenges.