I built a search engine before I reached for embeddings

Search is one of those problems where the modern answer arrives before the question does. You need search, so you reach for Elasticsearch, or you embed everything and query a vector store, and it works, and you learn nothing about why it works.

I wanted to know what was underneath, so I built a small search engine on Redis with TF-IDF ranking and a four-command CLI on top: index, search, remove, stats. It indexes over ten thousand documents and answers queries in under a millisecond, which surprised me more than it should have.

The data structure does the ranking

The trick that makes this work is that a Redis sorted set already is a ranked list. Every member has a score, and Redis keeps them ordered by it.

So at index time, for each term, I store the documents containing that term in a sorted set keyed by the term, with the term frequency as the score. Term frequency is half of TF-IDF and it is now sitting exactly where a ranking needs it.

The other half is inverse document frequency, which cannot be computed at index time because it depends on the whole corpus. That gets computed per query.

def _compute_scores(self, terms, n_docs):
    """Compute TF-IDF scores for search terms."""
    weights = {}
    for term in terms:
        df = self.redis.scard(f"idx:{term}")
        if df == 0:
            continue
        # IDF: log(N / df), so rare terms score higher
        idf = math.log(n_docs / df)
        weights[f"idx:{term}"] = idf

    if not weights:
        return []

    # ZUNIONSTORE combines the per-term sets, multiplying
    # each set's scores (TF) by its weight (IDF)
    dest = f"tmp:search:{uuid4().hex}"
    self.redis.zunionstore(dest, weights)
    results = self.redis.zrevrange(dest, 0, -1, withscores=True)
    self.redis.delete(dest)
    return results

ZUNIONSTORE takes a mapping of set names to weights, multiplies each set's scores by its weight, and sums across sets. That is the entire TF-IDF calculation, expressed as one Redis command. The scoring happens inside the database in a single operation rather than in a loop in my code.

Negative terms fall out of the same idea. Searching python -flask means finding documents matching python and subtracting the set containing flask, which is ZDIFFSTORE. No iteration, no filtering pass.

documents indexed

10k+

milliseconds per query

<1

CLI commands

4

Redis command doing the ranking

1

What the ranking is actually saying

TF-IDF has a plain-language meaning that got clearer once I had implemented it. A word scores high for a document when it appears often in that document and rarely across the corpus.

That second half is what makes results feel relevant. Without it, every document matching "the" ranks well. The logarithm keeps common words from being penalized into irrelevance while still pushing distinctive words to the top.

Watching the scores move as I tuned things taught me more about relevance than reading about it did. When a result looks wrong, you can trace exactly which term contributed what.

Where this approach stops

I am not arguing that this replaces a vector database, and I want to be clear about where it runs out:

• It matches words, so a query for "car" will not find a document about "automobile". Embeddings solve exactly this and TF-IDF cannot.
• Tokenization decides your quality ceiling. Stop word removal and stemming choices matter more than the ranking math.
• The whole index lives in memory, which is why it is fast and also what limits how far it scales.

Knowing those limits is the actual value of having built it. When a project genuinely needs semantic matching, I now know what I am adding and what problem it solves, rather than reaching for the vector store because that is what people do.

There is one more thing I did not expect. Almost all of the speed came from batching writes into Redis pipelines during indexing rather than from anything clever in the scoring. Most of the time was going into round trips, and collapsing those was worth more than any change I made to the ranking math.