about Ann-test comparison

[cherrylzhao]

hi, everyone, great work on qdrant.
I found qdrant achives highest RPS and lowest latencies in almost all the scenarios, no matter the precision threshold and the metric we choose. It has also shown 4x RPS gains on one of the datasets. see https://qdrant.tech/benchmarks/.

but in another famous ann-benchmarks project, the result is difference, for example in glove-100-angular, when recall is about 0.9, milvus can achieve 1751 rps, but qdrant only achieve 292 rps

please give me some hints for that, thanks

[kuaikuai]

The ann-benckmarks project uses Milvus Knowhere for testing. Note that Milvus Knowhere is a library, not a service as qdrant

[onestardao]

Good question — and welcome to the wonderful world of vector benchmark chaos 😅

You’re absolutely right to notice:

Qdrant = 292 RPS
Milvus = 1751 RPS
…on glove-100-angular under ann-benchmarks. Looks brutal on paper.

But here’s what’s often missing:

🔍 Milvus Nowhere ≠ Qdrant service
Milvus in ann-benchmark is just the Nowhere lib — stripped down, no storage, no REST overhead, pure memory-resident.
Qdrant is being tested as a production-grade vector DB — with full persistence, replication, filtering, etc.

That’s like drag racing a sports engine vs. a truck with a trailer. 🚛

🔍 Ann-benchmark ≠ realistic workload
Benchmarks like ann-benchmarks do nearest-neighbor lookups on frozen synthetic data (e.g. glove), with zero filtering, no real-world latency pressure (like auth, IO, batching).
They are great for algorithmic comparison, but don't reflect modern hybrid retrieval, RAG pipelines, or payload-aware filtering (which Qdrant supports, but Milvus Nowhere doesn't even try to).

🔍 If you care about real performance, test with your actual payload shape, request pattern, and retrieval logic.
In my case, I benchmark Qdrant with reranking, hybrid sparse/dense mix, and payload filters — and it's night-and-day more relevant than ann-benchmarks.

So TL;DR:

If you’re building a search engine for humans, Qdrant’s 292 RPS = very real.
If you’re racing on synthetic tracks, sure, Milvus Nowhere looks fast — but it doesn't even carry the weight.

Cheers
Let me know if you want to see some custom hybrid benchmarks I’m working on.