Why Artificial Intelligence Needs HPC

As we work with our customers on their artificial intelligence (AI) journeys, we are finding that a range of approaches are being taken by distinct organizations to implement AI. We recently had an opportunity to speak with Brian Dolan, chief scientist and co-founder of an AI startup — Deep 6 AI — to understand their interest in the Chapel parallel programming language, as well as why they view AI as a high-performance computing (HPC) problem.

First a bit of background on Deep 6 AI and Chapel.

Deep 6 AI is an artificial intelligence company whose primary mission is to find more patients faster for clinical trials — an important part of drug development and drug discovery. In 2016, Deep 6 AI was chosen by the Cedars-Sinai Accelerator Powered by Techstars, and since that time has signed several hospitals and contract research organizations. As Brian tells it, the Deep 6 AI software will help accelerate the time it takes to run trials, cutting costs significantly in the drug development process and getting life-saving cures to people faster.

Along the way, Brian found out about the Chapel parallel programming language that Cray has pioneered. Chapel is designed to improve the programmability of large-scale parallel computer systems. An open-source effort led by Cray and supported by a rich community of collaborators, Chapel’s primary goal is to increase the productivity of high-end computer users by making it easier to write, read, modify, port, tune and maintain parallel programs. You can find out more about Chapel by reading our Why Chapel?, Why Chapel? Part 2 and Why Chapel? Part 3 blog posts written by Brad Chamberlain, principal engineer at Cray on the Chapel project.

We sent Brad down to Deep 6 AI’s offices for a sit-down conversation on a range of topics related to AI, Chapel, Deep 6 AI’s reasons for looking at Chapel as a tool in their development chain, and finally to talk about AI in general. We’ll be posting videos over the coming days with highlights of the conversation between Brian and Brad.

One of the areas we wanted to get Brian’s insight on was how he views AI and high-performance computing. At Cray, we strongly believe that as more and more organizations adopt AI technologies, the systems and programming languages best suited to accommodate AI applications utilizing machine and deep learning algorithms will be high-performance computing.

So, we asked Brian straight out: What is the role of HPC in the AI space?

I’ve chosen a few highlights below:

“We’re starting with some overlap between artificial intelligence methodologies and HPC. I was brought to HPC because we are dealing specifically with very compute-intense problems. I think in general the AI community doesn’t know how much of an HPC problem they have yet. There is again certainly an intersection between these: the high-performance computing world and the artificial intelligence world. And I think it’s only now coming into focus.”

“Semi supervised learning on graphs and graph-based methods are becoming more popular and those are fundamentally different beast[s] in terms of what kind of compute has to be done. In a linear regression or convolutional neural net, you can take your massive amount of data, cut it up into little pieces, and send it off to your local workers. And again it’s embarrassingly parallel. In a graph structure, if you have a massive graph you never know if this guy’s going to have to talk to that guy…that poses a lot more problems in terms of how you’re going to manage that communication. How are you going to manage the underlying data structures.”

“In commercial AI right now it’s lots of data on fairly simple models. You run it against an embarrassingly parallel algorithm and then you’re done for the day. And that’s a very data intensive process. What it’s not is a very communication intensive process. For instance, having to talk to multiple parts of the graph. Once you have that you’re beyond … the promise of the NoSQL movement where everything can be distributed and parallelized, and you’re into a true HPC problem where you have huge amount of compute, operating objects that have to talk constantly to each other and the new wrinkle for HPC is — over huge amounts of data.”

Here’s the full video of Brian’s and Brad’s conversation. We’ll have more videos and excerpts in the weeks to come.


  1. 1

    Prem Kumar Gutty says

    Very true. AL and DL are just rediscovering the need for higher compute and parallelization of large tasks to multiple sets of smaller interconnected tasks.

Speak Your Mind

Your email address will not be published. Required fields are marked *