Benchmarks
What we measured, how, and what it does not tell you.
Benchmark · Wolof
Speech-to-text on the Wolof test split of Google FLEURS. Lower is better.
| Model | WER | CER | Speed | License |
|---|---|---|---|---|
Meta MMS-1B-all (wol adapter)read-speech option wav2vec2 MMS · 1B | 40.0% | 12.2% | 0.91 | CC-BY-NC-4.0 non-commercial license |
HuBERT base · 95M | 46.9% | 18.1% | 0.17 | AGPL-3.0 |
wav2vec2 MMS · 1B | 47.3% | 15.7% | 0.49 | MIT (base CC-BY-NC) non-commercial license |
w2v-BERT 2.0 · 600M · trained on FLEURS train split (same domain as the test set) | 48.5% | 18.5% | 0.58 | MIT |
wav2vec2 XLS-R · 300M · trained on FLEURS train split (same domain as the test set) | 49.2% | 18.7% | 0.15 | Apache-2.0 |
Test set: Google FLEURS wo_sn (test), 239 utterances, 69 min of audio · Hardware: GitHub Actions runner, 2 vCPU x86_64, int8 dynamic quantisation · Measured on 29 Oct 2026
Text-to-speech intelligibility
We have no human ratings yet, so we use a proxy: synthesise test sentences, transcribe them with our best speech-to-text model, and measure how much text survives the round trip. It says the speech is intelligible to a model, not that it sounds natural.
| Model | Round-trip CER | Speed | License |
|---|---|---|---|
SpeechT5 + HiFi-GAN | 15.0% | 0.77 | MIT |
SpeechT5 + HiFi-GAN | 29.7% | 0.76 | MIT |
SpeechT5 + HiFi-GAN | 53.3% | 0.99 | MIT |
60 × FLEURS wo_sn · ASR: soynade-research/Wolof-HuBERT-CTC
Wolof mixed with French
Real Senegalese speech switches between Wolof and French mid-sentence. On Kallaama (spontaneous farm-radio speech, CC-BY 4.0) we compare clips in Wolof only with clips that contain French words, as marked by the Kallaama transcribers. Radio speech is much harder than FLEURS read speech, so all numbers are higher.
| Model | WER, Wolof only | WER, with French | Wolof words missed | French words missed |
|---|---|---|---|---|
| Soynade Wolof-HuBERT-CTC training data not documented (may include Kallaama) | 28.5% | 30.8% | 26.4% | 33.8% |
| asr-africa XLS-R Wolof (Kallaama+FLEURS, 51 h)Served trained on Kallaama | 36.6% | 38.4% | 32.6% | 46.8% |
| Meta MMS-1B-all (wol adapter) | 70.6% | 75.6% | 69.2% | 95.1% |
250 Wolof-only clips and 250 clips with French (1314 French words), Kallaama test split.
Translation
Wolof ↔ French on the FLORES-101 devtest split: 300 sentences per direction, scored with chrF++ (higher is better) against professional translations.
| Model | Direction | chrF++ | BLEU | s / sentence |
|---|---|---|---|---|
| facebook/nllb-200-distilled-600MServed | fra → wol | 24.8 | 5.1 | 1.08 |
| bilalfaye/nllb-200-distilled-600M-wolof-french 1012/1012 test sentences are in its training data: score not valid | fra → wol | 48.1 | 26.5 | 2.26 |
| Lahad/nllb200-francais-wolof 1012/1012 test sentences are in its training data: score not valid | fra → wol | 33.1 | 9.6 | 1.62 |
| MaroneAI/nllb-French-to-Wolof-615M 635/1012 test sentences are in its training data: score not valid | fra → wol | 32.7 | 9.2 | 1.83 |
| facebook/nllb-200-distilled-600MServed | wol → fra | 36.0 | 14.0 | 1.24 |
| bilalfaye/nllb-200-distilled-600M-wolof-french 1012/1012 test sentences are in its training data: score not valid | wol → fra | 65.2 | 50.2 | 1.00 |
| galsenai/wolofToFrenchTranslator_nllb training data not documented: score cannot be trusted | wol → fra | 57.3 | 39.9 | 1.17 |
| MaroneAI/nllb-Wolof-to-French-615M 635/1012 test sentences are in its training data: score not valid | wol → fra | 40.0 | 17.5 | 1.65 |
Every published Wolof↔French fine-tune we tested either trained on these exact test sentences or does not say what it trained on. We serve base NLLB-200, the only model we can measure cleanly, until we have a clean test set of our own.
FLORES-101 devtest (CC-BY-SA 4.0)
Methodology
- Test set: Google FLEURS, Wolof (wo_sn), test split. Read speech of Wikipedia sentences, recorded by native speakers, licensed CC-BY 4.0.
- Text normalisation before scoring: lowercase, punctuation removed, whitespace collapsed. Letters such as ë, é, ñ and ŋ are kept: getting them right is part of the job.
- Each model decodes greedily (no language model) with the same int8-quantised CPU code the API runs, so the numbers match what you get.
- Runs are reproducible: a GitHub Actions workflow downloads the test split and the models, and commits the raw per-utterance results. Speed is measured on that 2-vCPU runner.
- Text-to-speech has no human ratings yet. We synthesise 60 test sentences per voice and transcribe them back with a separate Wolof model (Wolof-HuBERT-CTC) so the judge is not the served transcriber.
What these numbers do not say
- FLEURS is clean read speech. Conversations, phone audio, noise and Wolof mixed with French are harder and will score worse.
- Some models were trained on the FLEURS train split, which is the same domain as the test set; their numbers are optimistic. We flag them.
- Wolof spelling varies a lot in practice. Some errors are spelling differences a human would accept.
- Numbers include orthography choices: FLEURS writes some words differently from the models (melni vs mélni, 78 vs juróom ñaari fukk…).