Oído: open-vocabulary speech recognition on a $5 microcontroller

Lokutor · ESP32-S3, 240 MHz, 8 MB PSRAM, no neural accelerator; measured on one core

Oído runs NVIDIA's 13M-parameter Conformer-CTC Small speech recognizer entirely on an ESP32-S3, with an int8 engine written for the chip's vector instructions. It transcribes any English sentence, not a list of commands. On LibriSpeech it scores 3.7% / 8.2% word error rate (test-clean / test-other), or 3.3% / 7.2% with the on-chip language model used on this page.

How to read this page. Every clip below was picked at random with a fixed seed, not chosen for looking good, and every error is shown: wrong or extra words in red, missed words struck through. Transcripts come from the released firmware (int8 model + on-chip language model) running on a physical ESP32-S3 board. The laptop build of the same C code gives an identical transcript on 22 of these 27 clips; on the others the laptop's math library rounds the last bit differently from the chip's and a word or two changes, always on the hard (noisy) clips. Compute on the board is the time the chip took, measured with its 64-bit timer, on one of its two cores (RTF is compute time divided by the length of the audio). This v1 demo ran at RTF about 2. The v2 models run faster than real time (RTF 0.54–0.72 on one core; see the README). The same firmware in the emulator produced the same transcripts. Text appears after a 0.8 s pause plus this compute time.

Same sentence, different rooms

Two LibriSpeech test-clean sentences mixed with real noise recordings (DEMAND), other talkers, and simulated room reverberation, as in the robustness benchmark of the repository.

Sentence 1

AmbienceAudioReferenceOído on the ESP32-S3WERCompute on the board
Quiet room
the original recording
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.4 s
RTF 1.91
1606 M instructions
Car, 5 dB SNR
real car-interior noise (DEMAND)
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.3 s
RTF 1.91
1603 M instructions
Kitchen, 5 dB SNR
real kitchen noise (DEMAND)
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.4 s
RTF 1.91
1601 M instructions
Living room, 5 dB SNR
real living-room noise (DEMAND)
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.4 s
RTF 1.91
1607 M instructions
Cafeteria, 5 dB SNR
real cafeteria noise with background talk (DEMAND)
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.4 s
RTF 1.92
1607 M instructions
Babble, 10 dB SNR
four other talkers
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation wood6%13.5 s
RTF 1.93
1612 M instructions
Babble, 5 dB SNR
four other talkers, louder (hardest case)
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationdon't it is such a noble ambition between that it is a pity it has usually such a shallow proination of the italian39%13.7 s
RTF 1.96
1626 M instructions
Reverberant room, RT60 0.4 s
5 x 4 x 3 m room, talker 2 m from the microphone
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow foundation0%13.4 s
RTF 1.91
1600 M instructions
Very reverberant room, RT60 0.8 s
same room, harder walls
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such an open ambition that it is a pity it has usually such a shallow foundation11%13.4 s
RTF 1.91
1599 M instructions
Far field + kitchen noise
0.6 s reverberation and kitchen noise at 10 dB
7.0 s
it is such a noble ambition that it is a pity it has usually such a shallow foundationit is such a noble ambition that it is a pity it has usually such a shallow combination6%13.5 s
RTF 1.93
1610 M instructions

Sentence 2

AmbienceAudioReferenceOído on the ESP32-S3WERCompute on the board
Quiet room
the original recording
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.3 s
RTF 2.01
2569 M instructions
Car, 5 dB SNR
real car-interior noise (DEMAND)
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.4 s
RTF 2.02
2575 M instructions
Kitchen, 5 dB SNR
real kitchen noise (DEMAND)
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.3 s
RTF 2.01
2571 M instructions
Living room, 5 dB SNR
real living-room noise (DEMAND)
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.4 s
RTF 2.01
2572 M instructions
Cafeteria, 5 dB SNR
real cafeteria noise with background talk (DEMAND)
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness than some against event12%21.2 s
RTF 2.00
2558 M instructions
Babble, 10 dB SNR
four other talkers
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling a greater or less remoteness and some member of21%21.2 s
RTF 2.00
2570 M instructions
Babble, 5 dB SNR
four other talkers, louder (hardest case)
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur by giving a spar or of greater or less remote and some remembered event38%21.2 s
RTF 2.00
2566 M instructions
Reverberant room, RT60 0.4 s
5 x 4 x 3 m room, talker 2 m from the microphone
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.4 s
RTF 2.02
2575 M instructions
Very reverberant room, RT60 0.8 s
same room, harder walls
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered event0%21.4 s
RTF 2.02
2574 M instructions
Far field + kitchen noise
0.6 s reverberation and kitchen noise at 10 dB
10.6 s
in actual fact there are doubtless various factors that concur in giving us the feeling of greater or less remoteness in some remembered eventin actual fact there are doubtless various factors that occur in giving us the feeling a greater or less remoteness and some remembered event12%21.4 s
RTF 2.02
2576 M instructions

Real-world recordings

Unmodified recordings from public corpora: home microphones (Common Voice), parliament (VoxPopuli) and meetings (AMI).

SourceAudioReferenceOído on the ESP32-S3WERCompute on the board
Common Voice
volunteers' own microphones, at home
6.2 s
no such authority was given for the court of appeal whose registries remained closedno such authority was given for the course of peel whose registries remained closed14%12.0 s
RTF 1.93
1425 M instructions
Common Voice
volunteers' own microphones, at home
7.7 s
the performance was recorded and later released on a live album titled late orchestrationthe performance was recorded and later released on a live album titled late orchestration0%14.7 s
RTF 1.92
1777 M instructions
Common Voice
volunteers' own microphones, at home
5.8 s
he has been locked up so long he has been mostly forgotten abouthe has been locked up so long he has been mostly forgotten about0%11.1 s
RTF 1.92
1302 M instructions
VoxPopuli
European Parliament microphones
3.8 s
the qatari government and the football authorities must take noticethe qatari government and the football authorities must take notice0%7.4 s
RTF 1.96
844 M instructions
VoxPopuli
European Parliament microphones
7.9 s
the court still sees scope for further improving the quality of spending by simply applying the current system to its full potentialthe court still sees scope for further improving the quality of spending by simply applying the current system to its full potential0%15.9 s
RTF 2.01
1883 M instructions
AMI meeting
spontaneous meeting speech, headset microphone
3.6 s
so that indicates that there's an menu under that menuso that indicates a t an menu under that menu30%6.9 s
RTF 1.94
797 M instructions
AMI meeting
spontaneous meeting speech, headset microphone
3.8 s
yeah i think of it in terms of you know not even maybe to wellyeah i think of it in terms of you know not even maybe to what7%7.3 s
RTF 1.91
856 M instructions

Over the whole benchmark

Word error rate (%) over 300 sentences per condition. Oído on the chip, with and without the language model; Whisper tiny.en and Moonshine tiny in full precision on a laptop.

AmbienceOído + LM (ESP32-S3)Oído, no LM (ESP32-S3) Whisper tiny.en (laptop)Moonshine tiny (laptop)
Quiet room3.33.55.64.3
Car, 5 dB SNR3.23.65.94.4
Kitchen, 5 dB SNR4.24.77.56.1
Living room, 5 dB SNR4.85.77.97.0
Cafeteria, 5 dB SNR8.69.512.312.2
Babble, 10 dB SNR9.710.914.514.1
Babble, 5 dB SNR24.727.635.937.3
Reverberant room, RT60 0.4 s4.95.810.29.8
Very reverberant room, RT60 0.8 s14.116.022.528.6
Far field + kitchen noise11.413.519.824.3

Credits

Model: Oído, adapted from NVIDIA stt_en_conformer_ctc_small (CC-BY-4.0); the Oído model files are CC BY-NC-SA 4.0 with Lokutor's terms (see models/LICENSE-WEIGHTS). Speech: LibriSpeech (CC-BY-4.0), Common Voice 17 (CC0), VoxPopuli (CC0), AMI Meeting Corpus (CC-BY-4.0). Noise: DEMAND (CC-BY-4.0). Reverberation: pyroomacoustics. Page generated by eval/make_demo_page.py.