Voice bot with Whisper
Transcribe → reason → respond. Whisper + Mistral on the same EU gateway.
A minimal voice-in / text-out bot: upload an audio clip, run Whisper on it, hand the transcript to a chat model, return the answer. All EU-resident.
At a glance
| Stage | Endpoint | Model |
|---|---|---|
| 1. Transcribe | /v1/audio/transcriptions | systran/faster-whisper-large-v3 |
| 2. Reason | /v1/chat/completions | mistral/mistral-small-3.2 |
For text-out you can stop there. For voice-out, plug a TTS provider (out of scope today — TTS isn't in the CleverRouter catalog yet).
The pipeline
import { createCleverRouter } from '@cleverrouter/sdk';
const cr = createCleverRouter({ apiKey: process.env.CLEVERROUTER_API_KEY! });
export async function answerVoice(audio: File, history: ChatMessage[] = []) {
// 1. Transcribe
const tr = await cr.transcribe(audio, {
model: 'systran/faster-whisper-large-v3',
language: 'de',
response_format: 'json',
});
const userText = (tr as { text: string }).text;
// 2. Reason
const messages = [
{
role: 'system' as const,
content: 'You are a helpful assistant. Reply in two sentences max.',
},
...history,
{ role: 'user' as const, content: userText },
];
const res = await cr.chat({
model: 'mistral/mistral-small-3.2',
messages,
max_tokens: 200,
});
const reply = res.choices[0]?.message.content ?? '';
return { transcript: userText, reply };
}
interface ChatMessage {
role: 'system' | 'user' | 'assistant';
content: string;
}In a Next.js route
import { NextResponse } from 'next/server';
import { answerVoice } from '@/lib/voice-bot';
export async function POST(req: Request) {
const form = await req.formData();
const audio = form.get('audio') as File;
if (!audio) return NextResponse.json({ error: 'missing audio' }, { status: 400 });
const { transcript, reply } = await answerVoice(audio);
return NextResponse.json({ transcript, reply });
}Client side:
'use client';
import { useState } from 'react';
export function Recorder() {
const [reply, setReply] = useState<string | null>(null);
async function upload(file: File) {
const form = new FormData();
form.append('audio', file);
const r = await fetch('/api/voice', { method: 'POST', body: form });
const { reply } = await r.json();
setReply(reply);
}
return (
<div>
<input type="file" accept="audio/*" onChange={(e) => e.target.files && upload(e.target.files[0])} />
{reply && <p>{reply}</p>}
</div>
);
}Streaming the reply
For a snappier UX, stream the chat response while the user is reading the transcript:
const stream = cr.stream({
model: 'mistral/mistral-small-3.2',
messages,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}Wire that through your Edge route — see Edge runtime.
Provider notes
| Step | Default provider | Alternatives |
|---|---|---|
| Whisper | Tensorix | Scaleway (openai/whisper-large-v3) |
| Chat | Scaleway | Tensorix, Bedrock |
Pin per call when needed:
await cr.transcribe(audio, { provider: 'scaleway' });
await cr.chat({ model: '...', messages, provider: 'tensorix' });Audio size + timeouts
Whisper-large processes faster than realtime, but a 10-minute
upload can still hit the SDK's default 60s timeout. For long
audio, split client-side and merge transcripts — or bump
timeoutMs to 120_000.
Common gotchas
- Set
languageif you know it. Auto-detection is good but cuts hallucinations on noisy clips. - Use
promptfor vocab. Names, jargon and brand terms go in thepromptfield, not the system message. - Echo cancellation is upstream. The transcription quality is set by the microphone and the recording — there's nothing Whisper can fix at the model layer.