When it comes to moving tightly coupled high performance computing (HPC) workloads to the cloud, many organizations have been skeptical about its benefits. After all, messaging bottlenecks and scalability challenges have long overshadowed cost and flexibility related benefits of cloud deployments. Some challenges of moving HPC workloads to the cloud that have made organizations slow to migrate in the past have included:
- Slow, high-latency interconnects
- Long job queues
- High virtualization overhead and poor isolation from other workloads
- Data staging challenges and costs (getting large data sets into and out of the cloud)
- Moving from always-on, multi-user batch environments to on-demand cloud resources
- On-prem clusters becoming obsolete or poorly utilized
In the past couple of years, however, cloud providers like Amazon Web Services (AWS) have made rapid progress on improving their environments for HPC workloads. For example, AWS now offers a fully integrated suite of services for building and managing HPC clusters that includes features such as the Elastic Fabric Adapter (EFA), a network interface designed to run workloads that require high levels of internode communications at scale on Amazon EC2 instances. Every workload can run on its own on-demand cluster using an optimal set of services for their unique application. Individuals and teams can rapidly scale up or scale down these resources as needed, commissioning or decommissioning HPC clusters in minutes, instead of days or weeks.
Given ongoing improvements to cloud environments, along with the successes of many organizations now running critical HPC workloads in the cloud, it’s worth taking a look at how your organization might benefit from a cloud migration. Especially if you frequently struggle with long wait times or have to maintain an expensive infrastructure for running large but intermittent HPC workloads.
Migrating a weather modeling application to AWS
Organizations across an array of industries are seeing success with running a variety of workloads and applications - from computational fluid dynamics (CFD) and DNA sequencing applications to genome processing and reservoir modeling on AWS. Six Nines recently helped an innovative parametric insurance provider, Global Parametrics, successfully migrate a customized version of the National Center for Atmospheric Research Model for Prediction Across Scales (MPAS) . Given that MPAS is a complex, non-traditional model for AWS, a combination of HPC and AWS expertise were essential to successfully completing the migration and pilot project.
The configuration relies on Amazon Elastic File System (EFS) along with Amazon FSx for Lustre for high-performance shared storage and Amazon Elastic Block Storage (EBS) for standard data storage space output files with the application running on Amazon Linux 2. The entire infrastructure was created as code using Terraform, giving Global Parametrics the ability to rapidly create environments as needed.
Blazing a trail for difficult cloud HPC use cases
Ultimately, the success with moving the MPAS workload to AWS was an important win for Global Parametrics as well as other organizations with HPC use cases that have not been considered for the cloud. It demonstrates that the previous limitations of HPC workloads in the cloud can be overcome. And that in many cases there is now a far more cost-effective and flexible alternative to building and maintaining expensive on-prem systems. Six Nines and AWS can help with migrating HPC workloads - These workloads span the traditional HPC applications, like genomics, computational chemistry, financial risk modeling, computer aided engineering, weather prediction, and seismic imaging, as well as emerging applications, like machine learning, deep learning, and autonomous driving.
Learn more about the Global Parametrics MPAS migration:
Talk with one of our AWS and HPC experts about whether your HPC application may be a good candidate for a cloud migration. CONTACT US TODAY