In the Cloud

How to Supercharge your Search Engine with Fusion ioMemory

Search engines are powerful tools. They provide real-time access to a variety of different data types with a relatively small level of effort. But with enterprise data sets growing larger, meeting the performance demands of today’s users can be a challenge. Larger indexes can quickly overwhelm traditional search platforms, forcing users to invest in expensive scale-out architectures in order to meet their performance SLAs.

Performance Considerations: Running Out of Memory

Most search engines use locally attached disks to store their indexes. Traversing an index that is stored on disk can be time consuming, especially when that index is multiple terabytes in size. So to improve performance, these applications store as much of the index in memory as possible. However, large indexes can quickly exceed the amount of available memory in the server, resulting in degraded performance as the workload spills over onto disk.

To overcome this problem, most organizations adopt scale-out architectures, distributing the index across multiple servers in a cloud configuration. While this technique works, it requires a massive investment in infrastructure, increasing IT costs astronomically.

Fusion ioMemory to the Rescue

Fusion ioMemory PCIe Application Accelerators provide a persistent, high-performance, high-capacity memory tier that delivers terabytes of memory within a single server. By storing your index on ioMemory rather than traditional memory, it is possible to support these larger indexes without having to scale for DRAM. That means your indexes have room to grow without the need for additional servers, memory, or the added operations and maintenance costs of running a large cluster.

Recent testing with Apache Lucene has shown performance gains of an order of magnitude or more over disk, with latency reductions of more than 50x! In addition, operations such as bulk indexing and index optimization can be performed with no noticeable impact on search performance.

Bringing Enterprise Search Within Reach

Enterprise data sets are continuing to grow, and in order to meet the performance demands of these workloads, our search platforms will need to adapt to grow with them. As IT departments everywhere are struggling to meet workload demands, management challenges, and budget constraints, Fusion ioMemory provides a cost effective alternative to scaling with DRAM, bringing enterprise search capabilities within reach.

To learn more and see our Apache Lucene testing results, download our latest white paper Accelerating Enterprise Search with Fusion ioMemory PCIe Application Accelerators here.

Subscribe to this blog

Enter your email address to subscribe to this blog.

Mike McWhorter has more than 15 years of experience in the IT industry. He specializes in performance tuning and application acceleration using flash storage technology. At SanDisk, Mike is the principal solutions architect for the federal sales team, where he is responsible for designing and implementing large-scale distributed systems for the federal government. In addition to his sales role, he is also involved in testing and benchmarking new applications, and optimizing them for solid state storage. Previously, Mike worked as a developer for the MITRE Corporation, writing custom software for government sponsors for a variety of different applications. During this time, he worked in a rapid prototyping lab, building mission-focused solutions for situations where commercial software was not available. Mike received a Bachelor’s degree in Computer Science from Longwood University in Virginia.

bring your data to life

Today’s digital economy with mobile, IoT and cloud is based on the value of data. How do you unlock it? Download the infographic to learn more: