Autonomous Vehicle Development Drives Extreme Storage Requirements

We can probably all agree that driving a car today is a radically different experience than it was just a few years ago. The next generation of vehicles – self-driving cars – are poised to take modern automotive features a quantum leap forward and provide totally different ways to get from point A to point B.

One of the reasons fully autonomous vehicles are still a few years off is that designing an AI model that can safely and accurately drive a car is turning out to be an incredibly complicated task. The two main challenges are simulating every possible driving scenario that could occur so the model can be trained to identify the appropriate reaction, and handling the vast volume and variety of data generated by fleets of training cars.

These challenges are driving (no pun intended) some pretty extreme compute and storage requirements, which is where Cray comes in. We’re committed to supporting autonomous driving projects by providing highly scalable, efficient, and cost-effective technologies that enable customers to arrive at faster results with larger datasets.

Our customers all reported a set of similar challenges when attempting to use enterprise storage technologies for their autonomous driving projects:

1. Gaining enough storage capacity to handle multiple petabytes of data (scalability)

2. Ingesting and moving data from storage to the compute engine in a timely manner to avoid starving the multi-GPU compute architecture (throughput)

3. Providing a cost-effective way to achieve results in a shorter training window without blowing the storage budget (affordability)

To fully understand their requirements, we examined the typical AI workflow for autonomous driving, which is an extremely data- and compute-intensive process generally falling into five stages:

Figure 1. The AI Workflow for Autonomous Driving (Source: Cray, 2019)

We quickly realized that each stage in the workflow has its own technical considerations and therefore requires a unique set of technologies, tools, and capabilities. And so our approach was born: we would examine this complex AI problem in the context of the workflow to help the relevant teams focus on implementing the most appropriate technology for the dominant task in that stage.

For extremely data-intensive storage applications such as autonomous driving, Cray® ClusterStor® high-performance storage is the only storage solution on the market today that can seamlessly address these extreme requirements. ClusterStor is HPC storage that was purpose-built to support applications that rely on an increase in data availability to generate better, faster, and more accurate results (which happens to be most supercomputing applications today). Providing this data availability means the storage architecture must be able to both provide higher aggregated throughput for streaming data and deliver more storage input/output per second (IOPS) for random access to smaller amounts of data.

ClusterStor’s unmatched ability to address these requirements has quickly catapulted it to the No. 1 position for storage systems used by the top 100 supercomputer sites in the world.

Here at Cray, our approach to AI has always been rooted in the belief that techniques like machine learning and deep learning are quickly becoming core capabilities for users of supercomputing systems. As traditional supercomputing applications increasingly converge with AI methods, these fundamentally different approaches are being performed on the same hardware and demand similar requirements in terms of sustained performance and efficiency. Not only do we provide the computing components that power today’s most complex supercomputing applications, but our customer-centric solutions approach means we look at every AI problem in terms of the entire workflow. This helps us to hand-select and fine-tune each technology component until we’ve fully addressed each customer’s requirements.

Some of my colleagues fully explore these challenges and more in our new ebook, “How to Streamline AI System Design for Autonomous Vehicles.”

Let us help you fully understand and evaluate your technical requirements for training AI systems for autonomous driving. Contact us today to learn more about ClusterStor or schedule an on-premises briefing with our experts. You can also join the conversation on Twitter at @cray_inc.

Speak Your Mind

Your email address will not be published. Required fields are marked *