Technical Deep Dive
The Redis Secondary Indexing Module (redis-modules-secondary) was built on a deceptively simple premise: allow Redis to index arbitrary fields within hash data structures, then query those indexes with range conditions, equality checks, and basic aggregations like `COUNT`, `SUM`, and `AVG`. Under the hood, the module implemented a skip list data structure for each indexed field, maintaining sorted order to support `ZRANGEBYSCORE`-style queries. This was a natural choice given Redis's existing use of skip lists for sorted sets.
Architecture Highlights:
- Index Creation: Using `FT.CREATE idx ON HASH PREFIX 1 "user:" SCHEMA age NUMERIC SORTABLE`, the module would parse the schema, create an in-memory skip list for `age`, and register a hook on all `HSET` commands targeting keys with the given prefix.
- Write Path: On every `HSET`, the module would extract the indexed field, update the skip list (insert/delete), and maintain a reverse mapping from the Redis key to its index entries. This introduced a non-trivial write amplification: each indexed field update triggered an O(log n) operation in the skip list.
- Query Execution: A query like `FT.SEARCH idx "@age:[18 65]"` would traverse the skip list, collect matching keys, and return them. Range queries were efficient (O(log n + k) where k is result count), but multi-field queries required intersection of multiple skip lists, which could degrade to O(n) in worst cases.
- Memory Overhead: Each indexed field stored a copy of the field value in the skip list, plus pointers. For a dataset of 1 million users with 3 indexed fields, overhead could exceed 200 MB—significant for memory-bound Redis deployments.
Comparison with RediSearch:
| Feature | Secondary Indexing Module | RediSearch (current) |
|---|---|---|
| Index Type | Skip list (sorted) | Inverted index + compressed bitmaps |
| Full-Text Search | No | Yes (stemming, fuzzy, phonetic) |
| Aggregation | Basic (COUNT, SUM, AVG) | Advanced (GROUP BY, SORTBY, APPLY, REDUCE) |
| Multi-field Queries | Intersection of skip lists | Union/intersection of bitmaps (O(1) per term) |
| Memory Efficiency | ~200 bytes per indexed value | ~40 bytes per term (compressed) |
| Query Latency (1M docs) | 5-15 ms for range | 1-3 ms for full-text |
| Write Throughput | ~10K ops/sec (single field) | ~50K ops/sec (single field) |
Data Takeaway: The Secondary Indexing Module's skip-list approach was elegant but fundamentally limited for search workloads. RediSearch's inverted index and bitmap compression reduced memory overhead by 5x and improved query latency by 3-5x, making it the clear successor.
Open-Source Reference: The original module code remains on GitHub under `RediSearch/RediSearch` (now 12.5k stars). The `src/module-secondary/` directory contains the legacy implementation. Developers can still compile it for educational purposes, but it is not recommended for production.
Key Players & Case Studies
Redis Labs (now Redis Inc.) was the primary developer. The Secondary Indexing Module was part of a broader push to modularize Redis, alongside modules like `RedisJSON`, `RedisGraph`, and `RedisTimeSeries`. The team included Dvir Volk (lead engineer on RediSearch) and Itamar Haber (chief architect), who later pivoted entirely to RediSearch after recognizing the module's limitations.
Case Study: E-commerce Real-Time Inventory
A mid-sized e-commerce platform attempted to use the Secondary Indexing Module to power a real-time inventory filter: "Show me all products in category 'electronics' with price between $50 and $200 and stock > 0." They indexed `category`, `price`, and `stock` fields on a hash per product. Initially, queries returned in 2-3 ms for 500K products. However, as they scaled to 5 million products, write latency became problematic: each inventory update (price change, stock decrement) triggered three skip-list updates, causing write throughput to drop from 20K/sec to 4K/sec. They eventually migrated to RediSearch, which handled 50K writes/sec with the same hardware.
Competing Solutions at the Time:
| Solution | Indexing Approach | Latency (p99) | Max Dataset | Cost (per 1M docs) |
|---|---|---|---|---|
| Redis Secondary Indexing | Skip list | 10 ms | 10M | $0.50 (memory) |
| RediSearch | Inverted bitmap | 2 ms | 100M+ | $0.80 (memory) |
| Elasticsearch | BKD tree + inverted index | 15 ms | 1B+ | $2.00 (disk + memory) |
| MongoDB (2dsphere) | B-tree | 20 ms | 500M | $1.50 (disk) |
Data Takeaway: The Secondary Indexing Module offered the lowest cost but at the expense of scalability and feature depth. RediSearch struck the best balance for Redis-native workloads, while Elasticsearch remained the gold standard for complex search at scale.
Industry Impact & Market Dynamics
The Secondary Indexing Module's failure was a critical learning moment for the Redis ecosystem. It demonstrated that a simple key-value store could not be retrofitted into a search engine without fundamental architectural changes. This realization drove Redis Inc. to invest heavily in RediSearch, which now powers over 60% of Redis Enterprise deployments with search requirements.
Market Data:
- Redis Inc. raised $245M in Series G (2021) at a $2B valuation, partly on the strength of RediSearch's adoption.
- RediSearch is used by Uber (real-time driver matching), GitHub (code search), and Netflix (content discovery).
- The global in-memory database market is projected to grow from $4.5B (2023) to $12.8B (2028), with search capabilities being a key differentiator.
- Redis's modular architecture inspired competitors: Dragonfly (a Redis-compatible in-memory store) and Garnet (Microsoft's fork) both offer similar indexing modules.
Business Model Shift: The Secondary Indexing Module was open-source but required a commercial license for clustering. RediSearch is free in the community edition but restricted in the Enterprise version (e.g., no cross-cluster search). This freemium model has been highly effective: Redis Enterprise now accounts for 70% of the company's revenue.
Data Takeaway: The module's suspension was not a failure but a strategic pivot. Redis Inc. learned that search is a first-class workload, not an add-on, and that charging for advanced search features is a sustainable business model.
Risks, Limitations & Open Questions
1. Memory Bloat: The skip-list approach stored every indexed value twice (once in the hash, once in the index). For high-cardinality fields like user IDs or timestamps, this could double memory usage. RediSearch's compressed bitmaps mitigate this, but the trade-off is higher CPU usage during writes.
2. Write Amplification: Every `HSET` that modifies an indexed field triggers an index update. In high-write scenarios (e.g., IoT sensor data), this can become a bottleneck. The Secondary Indexing Module had no batching mechanism; RediSearch added a write-behind buffer, but it's still not ideal for write-heavy workloads.
3. No Full-Text Search: The module could only handle exact matches and numeric ranges. This severely limited its utility for search applications. RediSearch solved this, but at the cost of increased complexity.
4. Operational Complexity: Managing multiple Redis modules (JSON, Search, TimeSeries) can lead to version conflicts and memory fragmentation. The Secondary Indexing Module was particularly prone to crashes under memory pressure because its skip lists were not designed for graceful degradation.
Open Question: Could a modernized version of the Secondary Indexing Module, using Rust or Zig for memory safety and SIMD for faster skip-list traversal, outperform RediSearch for specific use cases? The answer is likely no, given RediSearch's decade of optimization, but the question remains open for niche workloads like real-time analytics on numeric-only data.
AINews Verdict & Predictions
The Redis Secondary Indexing Module is a textbook example of a failed experiment that nonetheless advanced the state of the art. Its core insight—that in-memory databases need native indexing for anything beyond key lookups—is now table stakes. RediSearch is the direct beneficiary, but the module's legacy extends beyond Redis: it influenced the design of MongoDB's Atlas Search, Aerospike's secondary indexes, and even PostgreSQL's in-memory extensions.
Our Predictions:
1. Redis will never revive the Secondary Indexing Module. RediSearch is too entrenched, and the engineering cost of maintaining two search modules is prohibitive. The code will remain as a historical curiosity.
2. The modular architecture Redis pioneered will become the norm. By 2028, every major database will offer a pluggable indexing layer, allowing users to choose between B-trees, inverted indexes, and vector indexes without changing the core engine.
3. The biggest risk to Redis's search dominance is not a better search module, but a better database. Dragonfly and Garnet are closing the gap on performance and modularity. If they add search capabilities that match RediSearch while offering 2x throughput, Redis could lose its edge.
4. Watch for a new generation of lightweight indexing modules that target edge devices and serverless environments. The Secondary Indexing Module's small footprint (under 1 MB) makes it ideal for IoT, where RediSearch's 50 MB minimum is too heavy.
Final Verdict: The Secondary Indexing Module was a noble failure. It tried to do too much with too little, but its ambition paved the way for RediSearch. Developers should study its code to understand the trade-offs of in-memory indexing, but should never deploy it in production. The future of Redis search is RediSearch, and the future of database search is modular, extensible, and deeply integrated.