Last week at Super Computing 2015 (SC15) show in Austin, TX, the University of San Diego (UCSD) and San Diego Super Computer Center (SDSC) presented the “Pacific Research Platform” (PRP). This is a virtual highway, capable of delivering on the promise of Big Data by leveraging fiber optic networks to transmit data at speeds of 10 gigabits to 100 gigabits per second.
The challenge PRP faced was to have an endpoint capable of terminating this 1,000x times faster optical network. One of the innovations PRP leveraged was the Flash I/O Network Appliance (FIONA) which used commodity PCs and turned them into Big Data appliances using Mellanox adapters talking to a 100G Arista TOR switch and SanDisk® InfiniFlash as the flash storage system. This enabled PRP to participate successfully in the “Petabyte Challenge” where a Lustre-based data oasis was transferred to the remotely mounted FIONA running at the SC2015 booth.
1PB Flash Storage, 1400 Watts
The stacked FIONA rack with 1PB flash storage (2 InfiniFlash units) consumed only 1400 watts. Just imagine designing the same rack with conventional flash arrays or SAS HDDs. We’d need to fill up the hall with storage systems!
High Performance Data Analytics (HPDA)
The Big Data Superhighway is a great example of High Performance Data Analytics (HPDA) a new market category established by IDC which brings High Performance Computing and Big Data Analytics together —a market that is predicted to grow rapidly.
HPDA is enabled by new trends and hardware technology advancements, notably:
InfiniBand and 100 Gbit/s Ethernet fabrics
High performance POSIX-compliant distributed parallel file systems, with Hadoop connectors such as
Lustre, IBM Spectrum Scale (GPFS), Red Hat Gluster
Disaggregated, denser flash storage
Compute-driven HPC architectures and HDFS-driven Big Data architectures, leveraging the best of both worlds
The need to process both batch-oriented large capacity historical data and live streaming data, which, for example, is popularizing Lambda Architecture.
Enabling HPDA with Big Data Flash
HPDA needs real-time ingest and performance while providing fab economics for lowering CapEx and OpEx. The Big Data Superhighway is just one of the use cases demonstrating this new Big Data convergence. Customers are increasingly using HPDA for emerging use cases like Machine Learning, Natural Language Processing (NLP), Graph Analytics, Video, Sound, and Image Analysis.
HPDA is already contributing to solving science, healthcare, security and data challenges and will continue to have a major role in finding new solutions and opportunities. InfiniFlash , which IDC defined as “Big Data Flash,” is an essential building block to enable the HPDA market, delivering massive capacity with extreme performance and breakthrough economics.
Shailesh has over 20 years experience in the industry earning significant experience with Internet of Things/Everything (IoT/IoE), Healthcare and Genome Sequencing, Financial Services Industry (FSI) and Enterprise Application verticalsAt SanDisk he is tasked with strategy, product planning and execution for HPC, Big Data and 3rd Platform Application Portfolio.Shailesh has worked for companies like EMC, NetApp, Force10 Networks, Adaptec, Brocade, HP and Digital. He has worked in product and solutions marketing, business development and presales roles.
Shailesh earned his bachelor’s degree in Electrical Eng. from the University of Mumbai, followed by a M.S and MBA in Finance and General Management from San Jose State University, including course work from UC Berkley.