Search engines are powerful tools. They provide real-time access to a variety of different data types with a relatively small level of effort. But with enterprise data sets growing larger, meeting the performance demands of today’s users can be a challenge. Larger indexes can quickly overwhelm traditional search platforms, forcing users to invest in expensive scale-out architectures in order to meet their performance SLAs.
Performance Considerations: Running Out of Memory
Most search engines use locally attached disks to store their indexes. Traversing an index that is stored on disk can be time consuming, especially when that index is multiple terabytes in size. So to improve performance, these applications store as much of the index in memory as possible. However, large indexes can quickly exceed the amount of available memory in the server, resulting in degraded performance as the workload spills over onto disk.
To overcome this problem, most organizations adopt scale-out architectures, distributing the index across multiple servers in a cloud configuration. While this technique works, it requires a massive investment in infrastructure, increasing IT costs astronomically.
Fusion ioMemory to the Rescue
Fusion ioMemory PCIe Application Accelerators provide a persistent, high-performance, high-capacity memory tier that delivers terabytes of memory within a single server. By storing your index on ioMemory rather than traditional memory, it is possible to support these larger indexes without having to scale for DRAM. That means your indexes have room to grow without the need for additional servers, memory, or the added operations and maintenance costs of running a large cluster.
Recent testing with Apache Lucene has shown performance gains of an order of magnitude or more over disk, with latency reductions of more than 50x! In addition, operations such as bulk indexing and index optimization can be performed with no noticeable impact on search performance.
Bringing Enterprise Search Within Reach
Enterprise data sets are continuing to grow, and in order to meet the performance demands of these workloads, our search platforms will need to adapt to grow with them. As IT departments everywhere are struggling to meet workload demands, management challenges, and budget constraints, Fusion ioMemory provides a cost effective alternative to scaling with DRAM, bringing enterprise search capabilities within reach.
To learn more and see our Apache Lucene testing results, download our latest white paper Accelerating Enterprise Search with Fusion ioMemory PCIe Application Accelerators here.
