Big or Big Fast Data?

cray-elephant-500X500 (2)

You say HPC & Hadoop®, I say Big Fast Data While the “Big” part of the Big Data nomenclature is a characteristic that garnered so many headlines, organizations using Big Data technologies like Hadoop, in meaningful ways are coming to discover that size alone isn’t enough. Though I’ll use the term Hadoop in this discussion, I’m really referring to anything in the MapReduce ecosystem, as well as a host of alternative “Big Data” technologies. There are a few reasons why organizations need to think about being fast, before they get big. Reason 1: Getting big is the easy part Go big, or stay relational… or even flat, for that matter. Technologies like Hadoop aren’t interesting for their ease of use, as they’re arguably prohibitively … [Read more...]

Cray Clusters Feed Growing Market

Fotosearch_20480530_csh0181

It’s well known that the high performance computing (HPC) market has grown quickly over the last couple of decades. In fact, the HPC market has tripled in size from $3.7 billion in the mid-1990s to $11.1 billion in 2012 according to IDC. But what may surprise some industry observers is the furious growth of cluster supercomputers. In 2003, these systems accounted for only 20 percent of all HPC system revenue; but from 2007 onward, clustered systems have grown to represent roughly 65 percent of total HPC system revenue, according to IDC. In addition, we’re likely to see the demand for cluster supercomputers continue to climb over the coming years as Big Data increases the demand for systems that can handle data-intensive workloads. A … [Read more...]

Computing Models Advancing as Oil and Gas Industry Demands Change

Oil and Gas_2

The oil and gas industry has long been among the leading practitioners of supercomputing technology, as 3D seismic data processing has been a staple of the sector for years. Recent developments in the industry have added additional layers of complexity and opportunity that promise to improve exploration and production operations - Permanent Reservoir Monitoring, or PRM. In the past, the development plan for a newly discovered field was based entirely on the original seismic data, observations made by the discovery and subsequent wells. In the previous two decades, it had been demonstrated at more than twenty producing fields around the world that when 3D seismic surveys were repeated two or three times a year it was possible to detect … [Read more...]

Open & Trusted Storage Solutions for Cluster Systems

CC3

I’ve worked in the data management and storage business since 1999.  One of the things that struck me early on was how supply-side and proprietary the industry was.  Storage vendors often pushed extraneous hardware or over-provisioned configurations.  Systems were proprietary and didn’t interop.   You could not replicate data from vendor A’s system to vendor B’s system and the worst part was that customers often were burdened carrying the tab for years, integrating and making due with supply-side configurations. For example, one customer I worked with described how a storage vendor dropped the gear off at the loading dock and walked away.  The customer was stuck figuring out how to put the pieces together. In HPC, especially for … [Read more...]

Chapel Parallel Iterators: Giving Programmers Productivity with Control

CHAPEL-LOGO-FINAL

As described in my previous blog article, Chapel is an emerging parallel programming language that we’re developing at Cray to dramatically simplify parallel programming, from your multicore laptop to the world’s most powerful Cray systems.  Chapel is being developed in an open-source manner and is designed and implemented with portability as a key concern. While the previous article gave a high-level overview of Chapel's concepts, implementation, and next steps, in this article I'll provide a deeper dive into one specific Chapel feature area—its support for user-defined parallel iterators.  This feature is designed to give programmers full control over the parallelism and scheduling used to implement parallel loops, yet in a very … [Read more...]