This is the continuation of my previous blog article describing why we are pursuing the development of the Chapel parallel programming language at Cray Inc. In that article, I argued for why the HPC community should consider pursuing productive new languages like Chapel. In this article, I’ll tackle some of the skepticism that typically arises in such conversations.
With so many languages trying and failing . . . is this an intractable problem?
Personally, I don’t believe that there is any inherent technical reason that we, as a community, can’t create a decent and productive language for large-scale computing. I believe that our failure to do so thus far has been due less to technical challenges (though they do exist) and more to insufficient long-term efforts, resources, community willpower, co-design and patience. Furthermore, I think these constraints have caused us to pursue conservative language designs that, by nature, have not been sufficiently general, powerful or extensible to gain the broad user community they need to take off. The growing Chapel community is striving to address these challenges, learning from the failures of the past while anticipating the challenges of the future.
But aren’t languages notoriously hard to get adopted?
Yes, they certainly are, and anyone who claims otherwise is being naïve. On top of the sustained effort and technical challenges required to get a new parallel language up and running, language developers must also wrestle with social factors like programmers’ tastes, community inertia, “marketing” and politics. These are concerns to be aware of and to strategize for, but not reasons to give up without trying. I think of it this way: If I were able to jump ahead 5, 10, or 40 years and found that we still had no viable language-level alternatives to today’s parallel programming notations (or more likely, the continued evolution of today’s notations in the face of ongoing hardware mutation), I’d be surprised and sorely disappointed in our community. And in conversations with developers of parallel applications, most say they feel the same way. That tells me we need to keep striving for something better — wrestling with the social challenges, learning from our mistakes and improving upon them.
Beyond simply “What choice do we have?” there are also reasons to be optimistic about the adoption of new languages. University professors report that today’s students tend to be increasingly polylingual in their programming, fluidly moving between various languages as warranted by the context. This suggests to me that social barriers against considering and adopting new languages may be waning. The level of programmer interest in nascent languages like Go, Julia and Rust seems to confirm this impression.
But even if I’m being overly optimistic in that assumption, I believe that the ongoing need for a language that supports scalable and parallel programming will cause us to eventually reach a tipping point and adopt such a language. Programmers wanted a productive, scalable parallel programming language 20 years ago (and probably earlier than that), they wanted one 10 years ago and they want one today. If 10 years from now we still haven’t developed one, they will almost certainly continue wanting one, particularly given the anticipated increase in machine scale and complexity within HPC. Meanwhile, in the mainstream, parallelism at scale continues to be used more pervasively and programmers and students are increasingly accustomed to modern, productive languages. For these reasons, the adoption of a productive and scalable parallel programming language seems inevitable to me. To that end, nothing is gained by simply giving up now. It might delay the inevitable, but it won’t stop it.
Chapel bears some resemblance to HPF, a high-profile language that failed to be adopted. How can Chapel hope to succeed?
It’s true that there are aspects of Chapel that resemble HPF — most notably, its support for global-view distributed arrays. In actuality, Chapel’s array features were more closely modeled after those of ZPL, another language from the ‘90s that was arguably more successful than HPF from a performance portability perspective. Yet, similarities and precursors aside, the Chapel team took the opportunity to learn from the missteps of both languages and to improve upon them. For instance, we support multidimensional distributions, user-defined distributions and a richer notion of arrays and their index sets.
Arguably, one reason HPF and ZPL failed to achieve broader adoption is that they were too narrow in scope, supporting only a single level of regular, global-view data parallelism. In contrast, Chapel was designed to support a much richer and more extensible definition of parallelism and locality, while also incorporating productive features from mainstream languages. To that end, while one facet of Chapel may resemble HPF/ZPL, there are numerous other feature areas that resemble successful languages like Fortran, C, C++, C#, Java, Python, Ruby and MATLAB, as well as important, yet less-pervasive, languages like Modula, NESL, CLU and the Cray MTA™ extensions to C and Fortran.
But the more important point is this: The failure of one parallel language — even a high-profile, well-funded, government-backed one — does not dictate the failure of all future attempts any more than early failures in flight or space travel implied that those milestones were impossible. As I’ve written elsewhere, I believe that there are a multitude of reasons that HPF failed to be broadly adopted. In designing Chapel, we’ve forged our strategy with those factors in mind, along with lessons learned from other successful and unsuccessful languages. Past failures are not a reason to give up; rather, they provide us with a wealth of experience to learn from and improve upon.
But why create a new language? Why not extend C, C++, Fortran . . . ?
Another good question, but once again I’m up against my word-count limit. In the next and final article in this series, I’ll tackle this question and wrap up. In the meantime, if you have questions about Chapel that you’d like to see addressed in future posts, please send them to firstname.lastname@example.org.
Brad Chamberlain, Principal Engineer