RAG with embeddings + rerank
Embed → vector-search → rerank → chat. End-to-end EU-only pipeline.
A two-stage retrieval pipeline that scales: fast embedding search narrows the candidate set, rerank picks the best few, and a chat model answers from them. Every step runs in the EU.
At a glance
| Stage | Endpoint | Model |
|---|---|---|
| 1. Embed corpus + query | /v1/embeddings | cohere/embed-v4 |
| 2. ANN search (your DB) | pgvector / Qdrant | — |
| 3. Rerank top candidates | /v1/rerank | cohere/rerank-v3-5 |
| 4. Generate answer | /v1/chat/completions | mistral/mistral-small-3.2 |
The full pipeline
import { createCleverRouter } from '@cleverrouter/sdk';
const cr = createCleverRouter({ apiKey: process.env.CLEVERROUTER_API_KEY! });
// 1. Embed at ingest time
async function embedDocs(docs: string[]) {
const res = await cr.embed(docs, { model: 'cohere/embed-v4' });
return res.data.map((d) => d.embedding as number[]);
}
// 2. Vector search (pseudo — replace with your store)
async function vectorSearch(query: string, k = 50): Promise<{ id: string; text: string }[]> {
const [queryVec] = await embedDocs([query]);
return db.search(queryVec, k); // your pgvector / Qdrant call
}
// 3. Rerank to top-N
async function pickTop(query: string, candidates: { id: string; text: string }[], n = 5) {
const res = await cr.rerank({
query,
documents: candidates.map((c) => c.text),
top_n: n,
});
return res.results.map((r) => candidates[r.index]!);
}
// 4. Answer from the top-N
async function answer(query: string, context: string[]) {
const res = await cr.chat({
model: 'mistral/mistral-small-3.2',
messages: [
{
role: 'system',
content:
'Answer using only the context below. ' +
'If the answer is not there, say so. Cite the doc number.',
},
{
role: 'user',
content:
`Context:\n${context.map((c, i) => `[${i + 1}] ${c}`).join('\n\n')}\n\n` +
`Question: ${query}`,
},
],
});
return res.choices[0]?.message.content;
}
// Orchestrate
async function ask(query: string) {
const candidates = await vectorSearch(query, 50);
const top = await pickTop(query, candidates, 5);
const text = await answer(query, top.map((d) => d.text));
return { text, sources: top.map((d) => d.id) };
}Why two stages
Pure vector search has high recall but mediocre precision. Rerank is the opposite — high precision, but expensive to run on big sets. Pipeline them: ANN narrows 1M → 50, rerank narrows 50 → 5, chat reads 5.
Cost ballpark for a 1M-document corpus, 1k queries/day:
| Stage | Cost driver | Per call |
|---|---|---|
| Embed query | 1 short embedding | < €0.0001 |
| Vector search | Your DB | local / DB cost |
| Rerank top 50 | 1 search unit (50 docs) | ~€0.002 |
| Chat answer | ~2k tokens | ~€0.003 |
That's ~€5/day fully loaded for 1k Q&A turns at this size.
Provider routing per stage
You can pin per call if compliance dictates:
await cr.embed(docs, { model: 'cohere/embed-v4', provider: 'bedrock' });
await cr.rerank({ query, documents, provider: 'bedrock' });
await cr.chat({ model: 'mistral/mistral-small-3.2', messages, provider: 'scaleway' });Common gotchas
- Mismatched vector spaces. Embed and query with the same model + dimensions. Re-embed the whole corpus if you change either.
- Don't skip rerank for short corpora. Under ~10 candidates, rerank is overkill. Send them all to chat directly.
- Cite. Always. Without citations, hallucinations are
invisible. Format the user-visible answer with
[1]/[2]markers tied to source IDs.
Where does the data live?
Your vector store is yours — CleverRouter never sees it. We see the query string at search time and the top-N candidate strings at rerank time. Both pass through under ZDR.