Deep Learning at Scale

Cray is synonymous with large-scale computing. While this is by no means all we do, the research that leverages these large systems is interesting to examine as leading indicators of things to come in the broader community. This has been true for a long time in modeling and simulation, and we’re now beginning to see the same advances in the area of deep learning.

For example, the Cray® XC™ “Cori” supercomputer at the National Energy Research Scientific Computing Center (NERSC) is an amazing system, sitting at #8 on the most recent Top500 list of the world’s fastest supercomputers. Of course, it’s the organization that surrounds Cori that enables the use of such technology. To this end and to help further the use of Cori for emerging big data and artificial intelligence workloads, NERSC established the Big Data Center to solve the DOE’s leading data-intensive science problems at scale. We’re proud to say that Cray has joined the NERSC Big Data Center program.

Our interest in the NERSC Big Data Center is simple: We want to focus on areas that are key to furthering and expanding scientific research and artificial intelligence on supercomputer systems.

Advancing Deep Learning at the Big Data Center

The Cori system has already been used in advancing the state of the art in scalable deep learning training algorithms by breaking the 10-petaflop (PF) barrier on two deep learning applications. Earlier this year, a team of scientists and engineers from Lawrence Berkeley National Laboratory, Stanford University and Intel achieved peak throughput of 15 PF with deep learning software. Their accomplishment, according to a recently published research paper, made it the world’s most scalable deep learning implementation and “the first 15-petaflop deep learning system for solving scientific pattern classification problems on contemporary HPC architectures.”

Finding Novel Particles in HEP Data

One application used a supervised convolutional neural network for discriminating among signals in high-energy physics (HEP) data. Classification of such data is used to find unique events — such as previously undiscovered particles — from new physics. The HEP classification used training data based on 10 million images generated by physics simulations, and it ran on 9,600 Intel® Xeon Phi™ processors, achieving peak throughput of 11.73 PF and sustained throughput of 11.41 PF.

HEP is important because it seeks rare signals of new particles produced by collisions in accelerators like the Large Hadron Collider. Improvements in classifying these collisions and the resulting particles “could aid discoveries that would overturn our understanding of the universe at the most fundamental level,” according to the project’s researchers.

The HEP model scaled impressively, seeing a 6,173-fold speedup when going from one processor to the full 9,600 processors.

Identifying Extreme Weather Events

The second application, which used a semi-supervised architecture for localizing and classifying extreme weather in climate simulations, saw even better scaling results. The researchers attained a 7,205-fold speedup when scaling from one to 9,622 Xeon Phi processors on Cori.

Climate change is a hot topic and one of today’s most critical scientific challenges. Climate simulations can help us understand weather patterns as well as carbon emission scenarios and intervention strategies, but they produce massive datasets. In this project, researchers used a 15 TB dataset from climate simulations. Their model resulted in peak throughput of 15.07 PF and sustained throughput of 13.27 PF.
One significant technological development in this project was the use of a hybrid training technique that combined synchronous stochastic gradient with asynchronous stochastic gradient, each tunable by the team. This technique allowed training to be scaled to the full system.

Deep learning in climate simulation and similar scenarios is expected to become increasingly rigorous and capable of identifying ever-more-subtle phenomena as it continues to develop, and HPC platforms are becoming a new standard for high-performance deep learning applications.

Unlocking Capabilities Required for the Most Challenging Workflows

Cray is committed to furthering the use of deep learning for the most challenging scientific and enterprise workflows, as evidenced by the extreme weather event work. This week, the 2017 Conference on Neural Information Processing Systems (NIPS) is taking place in Long Beach, California. Two workshops are of particular interest, as they support our belief that deep learning is a high-performance computing application. The first workshop, on December 8, is “Deep Learning for Physical Sciences.” At the second workshop on December 9, “Deep Learning at Supercomputer Scale,” some of the 15 PF performance work described above will be discussed, including some additional performance information obtained by Intel when using our recently announced Cray Programming Environment Machine Learning Library.

We look forward to working with our partners in the Big Data Center and are excited to be part of this dynamic community.

Learn more about deep learning and XC computing technology on Cray’s website.

Speak Your Mind

Your email address will not be published. Required fields are marked *