What does a speech model actually hear?

The clip is 9.72 seconds of speech. The model sees it as air pressure, 16,000 numbers each second.

The ears read the whole clip once. They give one row of ear frames, one for each 80 ms of sound.

The writer reads the frames and writes text. It runs once per piece of text, not once per clip.

Blue is the ears. Grey is the writer and the tables. In Model B, the ears hold 66% of the values.

Sound is air pressure, measured 16,000 times each second. This clip has 155,520 of these numbers.

80 ms is 1,280 of those numbers. Each ear frame is centred on one 80 ms slice. After the layers, it also draws on the frames around it.

The ears give 122 frames. Each frame is a list of 1,024 numbers, drawn here as one column.

155,520 numbers go in and 124,928 come out. The ears keep roughly the same amount of numbers.

It turns sound into a picture. It has no learned values, so its numbers are fixed.

Three halving stages cut 973 windows down to 122 frames. Each frame then has 1,024 numbers.

Each layer refines all 122 frames. Every layer has the same shape, so they stack up.

One layer has five sub-parts, run in this order. Each part adds its result to the running value.

The running value is the list the layer keeps adding to.

The 32 layers hold 99.5% of the ears' values. The shrinker holds 0.5%.

The clip is one man reading one Hindi sentence, 9.72 seconds long. We follow window 194, centred on 1.94 seconds.

A window holds 400 samples. Each sample is one air pressure number, taken 16,000 times a second.

Each sample minus 0.97 times the one before. The line changes faster, so high pitches stand out more.

A fixed bell curve multiplies every sample. It is 0 at both ends and 1 in the middle.

The next window starts 10 milliseconds later. So each pair of windows shares 15 milliseconds of sound.

This is window 194 after the bell curve. Its edges are 0, so the cut is clean.

The model measures how strong each pitch is, from 0 Hz to 8,000 Hz. In this window the strongest pitch is 6,094 Hz. Pitches below 2,000 Hz are weak here.

Each band is a fixed triangle of weights over nearby pitches. Band 0 covers one pitch step. Band 127 is 344 Hz wide.

Each band gives one number for this window. So the 128 numbers stack into one column.

The human ear tells low pitches apart finely and high pitches coarsely. Bands copy that: narrow at low pitch, wide at high pitch.

Each window gives one column of the picture. The yellow column is window 194. 973 columns side by side make the whole picture.

Pitches are merged into 128 bands. The picture gets shorter: 257 rows become 128.

A sound 100 times stronger becomes one fixed step up, not 100 times taller. So quiet details stay visible next to loud ones.

Each band gets its own mean and spread for this clip. The mean is taken away, and the rest is divided by the spread. So a loud clip and a quiet clip give almost the same picture.

We rebuilt this picture on a laptop, using the model's two fixed tables. It matches the model's own picture: correlation 1.000000, largest difference 0.002.

The clip is 9.72 seconds of sound. The front end turns it into a picture. Across: 973 time steps. Up: 128 bands, low pitch at the bottom.

A 3 x 3 patch is a small detector. It slides over the picture and keeps every second position.

There are 256 different patches. Each one makes its own small picture.

Stages 2 and 3 do the same halving, with a mixing step after each. The picture shrinks to 244 x 32, then 122 x 16.

The channel count stays at 256.

Each of the 122 time steps now holds 256 x 16 = 4,096 numbers.

One linear map turns them into 1,024 numbers. The output is 122 x 1,024.

Attention compares every time step with every other one.

Before the shrink: 973 x 973 pairs. After it: 122 x 122 pairs.

Each step gives one list of numbers per 80 ms of sound.

How finer timing is held, we did not test.

It is 122 frames, each a list of 1,024 numbers.

The layer adds to it. It never replaces it.

A side branch reads each frame alone and makes a new answer.

The answer is halved, then added to the running value.

Every frame looks at every other frame. The result is added at full strength.

The old numbers stay in the sum.

Each frame reads a window of 9 frames: itself, 4 before, 4 after. That is 720 ms.

Its result is added to the running value.

The second side branch is halved and added, as in step 2.

Then one normalise keeps the numbers in range.

This bar chart shows where one layer's values sit.

The two feed-forward parts hold most of them.

Frame 24 makes a question. Every frame makes a label and a message.

Each one is a list of 1,024 numbers.

One score says how well the question fits each label. A second score uses how far apart the two frames are, with the question.

Thicker arcs mean higher scores. This drawing is a schematic.

The shares always add up to 1. Frame 24 takes the share-weighted mix of the messages.

It adds that mix to its running value.

The 1,024 numbers are cut into 8 groups of 128. Each group makes its own shares.

The score has two parts. One uses the two frames' content.

The other uses how far apart they are, read with frame i's own question. The model never uses the frame number itself.

Frame 24 also looks at frames 25 to 121, which come after it.

So the whole clip must be there first. That is why these ears cannot run live as they are.

Frame 24 is yellow. Each frame covers 80 ms of sound. The whole clip has 122 frames.

The window is frame 24, with 4 frames before it and 4 after. Together they cover 720 ms of sound.

The layer first widens the numbers, then one half gates the other half. A gate multiplies by a value from 0 to 1. Then a 9-frame slide does the neighbour work.

The same weights are used at frame 25, then frame 26, and so on. Only the window moves.

Attention reads any frame in the clip. The convolution reads only 9 frames: itself, 4 before and 4 after. A layer with both is called a Conformer layer.

The convolution part holds 3,163,137 values. Only 9,216 of them make the 9-frame slide: 1,024 positions times 9 weights.

Only frame 24 goes in. It is a list of 1,024 numbers. No neighbour is read.

A set of weights turns the 1,024 numbers into 4,096 numbers. Each new number mixes the old 1,024.

Each of the 4,096 numbers goes through a smooth on/off curve. The curve is flat near zero on the left and rises on the right. Then the 4,096 numbers become 1,024 again.

The result is multiplied by 0.5. It is then added to the running value. The running value is what the layer keeps adding to.

There are two of these parts in each layer, A at the start and B at the end. Together they hold most of the layer's values.

Attention and convolution move information between frames. Feed-forward works on what one frame holds now. This is the standard reading of the design. We did not test it.

Each dot shows how alike two frames are. The diagonal line runs from top left to bottom right. Each frame is exactly alike itself.

Frames are alike only with close neighbours. Frames far apart are not alike.

Layer 1 changes the pattern only a little. Close neighbours are still the most alike.

At layer 16, frames are alike over a wider span. Of the four depths we saved, its mean likeness is the highest: 0.320.

At layer 32, frames are apart again. The mean likeness drops to 0.122. Close neighbours stay alike.

These pictures show how the frames relate to each other. They do not show what each layer knows. We did not test that.

A caller speaks for 9.72 seconds. At any moment the model has only the sound up to now.

The yellow line is now. Everything to its right has not arrived yet.

Model B does nothing until the clip ends. Then it computes all 122 frames and writes all the text.

Three of its parts need the whole clip. So it cannot start early.

The live model computes each small piece of sound when it arrives. Here a piece is 4 frames, 320 ms.

Its writer looks at each new frame and writes a word or stays silent. So the text grows while the caller speaks.

Model B levels each band of the picture with numbers from the whole clip. So the picture is ready only at the end.

The live design skips that step. Each column is used when it arrives.

In Model B a frame looks at every frame of the clip, also the later ones. That needs the whole clip.

In the live model a frame looks back 56 frames and ahead only a few. Here it is 3.

The Model B convolution reads 9 frames: 4 before, itself and 4 after. Four of them are in the future.

The live convolution also reads 9 frames, but all from the past: 8 before and itself.

The Model B writer reads all 122 frames again for every word. So it cannot write before the last frame exists.

The live writer is tiny. For each new frame it writes a piece of text or stays silent.

Every part of Model B that needed the future has a live version that does not.

The layers themselves keep the same four sub-parts and almost the same size.

Model B looks at every frame, before and after each one. So it must wait until the clip has ended.

The blue bar shows 56 frames, which is 4.48 seconds of sound. The green bar shows the few frames it may look ahead.

The model card lists 0, 1, 3, 6 or 13 frames. Each pick sets the piece of sound.

The paper behind this design drops per-band normalising. We did not check this model’s front end.

The streaming model has 24 layers, Model B has 32. One layer has almost the same count of values.

In this example a piece is 320 ms, which is 4 frames. The 4 new frames go through all the layers.

When the next piece arrives, the old frames run again with the new ones. That work is done twice.

Attention and each 9-frame convolution keep a store of past frames. The store sits beside the layer.

Piece 2 brings 4 new frames. They read the store, and then the store is updated.

Attention and convolution look at past frames. Feed-forward and normalisation work on one frame alone, so they need none.

The card says the model keeps caches in these layers. Each frame is computed once.

A setting is the length of one piece of sound. The model card gives the average error at all five settings.

With the language given, Hindi falls from 8.13 to 6.81 over the same five settings.

1,120 ms down to 80 ms is 14 times less waiting. The average error rises by 1.54 points: 10.38 minus 8.84.

Ears trained only with the whole clip in view collapse when the future is hidden. At 160 ms the error is 94.05.

Ears trained for both ways at once reach 10.51 at 160 ms. Offline they lose almost nothing: 6.63 against 6.47.

We did not run this model. Every number here is the authors' number, from the model card or the paper.

One public leaderboard, dated 9 October 2026. Its score is the average word error on English tests. Lower is better.

We skip closed systems, because their ears are not public. Eleven of them rank above the tenth open model.

Four models use Conformer ears, four use Whisper ears, and two use Qwen ears. Cohere's family comes from its model card.

Ranks 6, 7 and 10 use ears of the same shape as Whisper large-v3. Their values were trained further.

Ranks 2 and 3 have ears of the same shape and size.

This ranking covers English. It does not show how these models hear Indian languages.

A Conformer ear takes a sound picture and shrinks it in time.

Model B and Phi-4, read in their code, run four parts in this order.

Both use 128 bands and shrink time 8 times.

Model B attention sees the whole clip. The streaming model looks back, with a short look-ahead.

It uses 80 bands and shrinks time 8 times. Its convolution reads itself and 2 frames before.

Its attention sees the whole clip.

It glues 2 picture columns together, so time shrinks only 2 times.

Its frames last 20 ms. Its attention works in blocks of 4 seconds.

Each member's layer has its own count of values. The bars show them, counted from the weight files.

As released, Whisper pads every clip to 30 seconds. Our clip is 9.72 seconds.

Of the four models here with this shape, only MOSS does the same.

They halve time once: 3,000 columns become 1,500 frames.

One frame now covers 20 ms.

Each frame has its own stored position.

In Whisper as released, nothing longer than 30 seconds fits.

Attention, then feed-forward. There is no convolution inside.

Each part adds its result to the running value.

Attention compares every frame with every other frame.

Whisper as released compares 1,500 frames with 1,500 frames.

Whisper as released lets every frame look at the whole window.

TheWhisper ships 10 second windows and a streaming pipeline, per its card. Higgs and ARK pad only to the longest clip.

Qwen's shrinker is similar to the Conformer one. It cuts time 8 times.

It runs 3 halving stages with 480 channels. It makes 12.5 frames per second, one per 80 ms.

Each layer has attention and feed-forward. It has no convolution.

Qwen3-ASR has 24 of these layers.

Attention looks only inside a window of 1 to 8 seconds. The authors set this in their paper.

The authors say this lets the same ears run live and on whole clips.

Our clip has 973 picture columns. 973 divided by 8 gives about 122 frames.

Qwen works in pieces of 100 columns, so the count may be a few more. We did not run it.

In Qwen3-ASR, the ears are 15.4% of the model.

In Hojo, they are 12.5%. We did not read Hojo's own settings.

The coloured part is the ears. The grey part is everything else, mostly the writer. The counts come from the weight files.

In this chart, five models have big ears: 66% to 97% of their weights.

Eight of the ten use a language model as the writer. Seven of them have ears of 5% to 34%.

Every language-model writer model here is 0.9 billion or more. The three under 0.9 billion all have big ears.

Across these models the ears are 0.31 to 0.81 billion values. What differs most is the writer.

Meta wav2vec 2.0 reads the raw wave. Seven conv layers shrink it to 50 frames per second.

Each layer looks back 16 frames and ahead 4 or 0. So Moonshine can run live.

A separate codec turns sound into codes, 32 per step. One model reads them and writes the text.

Voxtral Mini 4B Realtime uses 32 layers. Attention looks only inside a sliding window.

We read only the public settings of these four. We did not count their weights or run them.

One man reads one Hindi sentence. It lasts 9.72 seconds. Every row uses the same time scale.

Model B, Phi-4 and the NVIDIA streaming model make one frame per 80 ms. That is 122 frames. Qwen gives about 122 by our arithmetic; we did not run it.

Granite makes one frame per 20 ms. The raw-wave design does too. Both give about 4 times more frames than the 8x designs.

OpenAI's Whisper, as released, pads every clip to 30 seconds. That gives 1,500 frames. Only 486 hold sound. Of the four models here with ears of this shape, only MOSS is shown to do the same.

Fewer frames mean less work for attention. More frames mean finer time detail.

Does the design run live as shipped? Is the whole model under 0.7 billion? Each design lands in one box.

Two small live examples appear here: the NVIDIA streaming model and Moonshine. NVIDIA streaming keeps a cache and a look-ahead dial. Qwen and Voxtral say they run live.

Parakeet has whole-clip ears and is not live as shipped. OpenAI's Whisper, as released, uses a fixed 30-second window. Model B looks at the whole clip.

Both counted small models have big ears. Parakeet's are 97.1% of its weights; NVIDIA streaming's are 95.5%.

Which ears keep tone of voice, pace and background best? Nobody here publishes that. We did not test it either.

We froze three sets of ears and fed them the same clips. A tiny reader on each layer had to answer six questions.

It was tested on speakers it never heard. The red line shows what guessing scores.

Blue is the NVIDIA streaming model. Its first layers answer background, line damage, loudness and gender almost perfectly.

Emotion peaks near layer 9. Pace is best near layer 20.

Orange is Whisper large-v3. It shows the same pattern: surroundings low, emotion in the upper middle.

Its best emotion score is 78%, a little above the live ears.

Green dashes are a codec built for speech generation. Grey dots are the bare sound picture.

The codec matches the ears on loudness and gender. It is far behind on emotion: 49% against 73%.

The shaded band is the top of the live ears, the part the writer reads. Background falls from 93% to 33% there.

Line damage falls from 100% to 49%. So a reader on the last layer cannot say why it cannot hear.

The clips are English. The emotion is acted. The noise and line damage are simulated, and the reader is very simple.

One run, about 300 test clips for four of the six questions. It shows what the ears keep, not how a finished reader will do on real calls.

Correction, same day: the background score used the same noise recordings to learn and to test. On a new recording it is about 40% (page I6).

The 9.72 second clip becomes a picture of 128 bands. It is padded to 1,000 columns, which is 10 seconds.

Then the picture is cut into ten pieces of one second each.

Each piece goes through three convolutions, alone. Each one halves the bands and the time.

After the third, a piece is 16 bands by 13 time steps, in 480 versions.

Each time step's 7,680 numbers become 1,024. A position from 0 to 12 is added.

The padding is dropped and the pieces are joined. Our clip is now 126 frames.

All 126 frames go through 24 layers. The shape never changes.

Attention works inside windows of 104 frames, which is 8 seconds.

A last normalise, then two maps make each frame 2,048 numbers long. That is the size the writer reads.

The ears took 12 ms for this clip, in our run.

Qwen makes the same kind of picture as Whisper: 128 bands, one column every 10 ms.

Our clip gives 972 columns. Qwen pads it with 28 empty columns to reach 1,000.

The picture is cut every 100 columns. That gives ten pieces of one second each.

Every piece goes through the three convolutions by itself. A convolution never sees across a cut.

At the edge of a piece it sees zeros, not the neighbouring second.

A full piece comes out as 13 frames. The last piece has only 72 real columns, so it gives 9.

Nine pieces of 13 and one of 9 make 126 frames.

A second can be shrunk the moment it has arrived. Nothing later is needed for it.

That is our reading of the code. The model's authors do not explain the choice in the code.

One second of picture: 128 bands by 100 columns. It is a single sheet of numbers.

A small 3 by 3 patch slides over the sheet and stops at every second position. So both sides halve.

There are 480 different patches. Each makes its own sheet of 64 by 50.

These are six real sheets from convolution 1. Each detector lights up for a different pattern in the sound.

After every convolution a smooth curve removes weak negative values.

The same step two more times. The sheets shrink to 32 by 25 and then 16 by 13.

These two hold most of the shrinker's patch values: 2,074,080 each.

After convolution 3 there are 480 small sheets. Each sheet has 16 bands and 13 time steps.

Take time step 11 from every sheet. That gives 480 x 16 = 7,680 numbers. The wide picture shows them.

A fixed map mixes the 7,680 numbers into 1,024 numbers. Each of the 1,024 is a weighted sum of all 7,680.

This list of 1,024 numbers is one frame. It is frame 24, the sound at 1.9 seconds.

Do the same for all 13 time steps of this second. You get a table of 13 rows, each with 1,024 numbers.

Each second of the clip gives one such table. Our clip gives 10 tables.

The last page ended with a table for each second: 13 rows, 1,024 numbers in each row.

Attention compares rows with each other. It does not know which row came first, so order must be written into the rows.

The model holds a table of marks: 13 rows, 1,024 numbers in each row. Row 0 is the mark for place 0, and so on.

The marks come from a formula of waves. Nothing in them is trained, and there are only these 13 rows.

Row 11 of the sound table is frame 24. Row 11 of the mark table is added to it.

The first number is 0.534 + (-1.000) = -0.466. The same is done for all 1,024 numbers. The shape does not change.

The mark table is added to the table of each second. So the marks start again at 0 in every second.

Frames 11, 24, 37 and so on get the identical mark: all are at place 11.

The 10 tables are put end to end. Rows made only of padding are dropped.

The result is one list of 126 frames. Frame number = 13 x the second + the place. For frame 24 that is 13 + 11.

A mark says: I am at place 11 in my second. It does not say which second.

You can see it in layer 1. Frame 24 takes 1.7 times more from frames with its own mark.

For this first picture the ears got the whole clip at once. All 9.7 seconds were already recorded.

Frame 24 is the sound at 1.9 seconds. Each bar shows how much it takes from one frame.

Frame 24 takes 63% from frames after it. That is sound after 1.9 seconds.

This sound was on the recording already. The model does not guess sound that has not come.

After frame 103 every bar is zero. A window is 8 seconds, and each second gives 13 frames.

So one window is 8 x 13 = 104 frames: frames 0 to 103. A frame takes nothing from another window.

Stack the strips of all 126 frames and you get a square. Row 24 is the strip you just read.

Each window makes one square. Our clip is 9.7 seconds long, so it has a full window and a short one.

In a call the sound arrives bit by bit. Here the ears ran when frame 24 was the newest frame.

It has no frame after it. It takes 70% from earlier frames and 30% from itself. Its answer is only 0.27 alike to the final one.

Qwen's code runs the ears every 2 seconds, on all sound so far. At 2 seconds the answer is 0.35 alike. At 4 seconds it is 0.91.

At 8 seconds the window is full. From then on the answer is final.

Red and yellow cells are answers that will still change. They are always the newest frames.

This is why Qwen writes its last words again in every run. The ears also do the old sound again each time.

Because it changes the answer a lot. With no later sound, frame 24 is 0.27 alike to its final answer.

One frame is 77 ms, a part of one spoken sound. The sound after it helps to tell what it was.

The fence does not mean that later sound is useless. We ran one test with the fence removed.

The frames next to the fence changed the most. They are the frames that the fence cuts off from their neighbours.

The layer keeps one list of 1,024 numbers for each frame. Each sub-part adds to it. Nothing is replaced.

Each frame looks at other frames and mixes in what it needs. Qwen uses 16 heads of 64 numbers each.

This is how information moves between frames.

Then each frame is worked on alone. It grows from 1,024 to 4,096 numbers and comes back to 1,024.

Two thirds of the layer's values are here.

A Model B layer has four sub-parts: two feed-forwards, attention and a convolution. A Qwen layer has two.

So one Qwen layer has 12.6 million values. One Model B layer has 25.2 million.

There is no convolution, so a Qwen layer has no built-in local view. Only attention moves information between frames.

We did not measure what that costs or saves.

The sound arrives all the time. Every 2 seconds the model does one run.

For our 9.72 second clip that is five runs. The last one is at the end.

In each run the ears get all the sound from the start up to now. Not only the new 2 seconds.

So in the run at 6 seconds the ears process 6 seconds of sound.

The writer does not start from nothing. It is handed the text it wrote in the run before.

The last 5 pieces of that text are taken away first.

The writer writes those 5 pieces again and then goes on with the new words.

So the newest words can still change. Older text stays as it is.

Right after a run the text covers all the sound so far. Then it waits up to 2 seconds for the next run.

So at second 10 you have the text of at least the first 8 seconds.

The ears do old sound again in every run. For this clip they process 29.7 seconds of sound in total.

Ears with a cache process each second once: 9.72 seconds.

As shipped, attention uses windows of 104 frames. A frame at the start of a window takes from sound up to 8 seconds later.

Until that sound arrives, its answer is a draft. Chapter G5 shows this.

The code lets the caller pass its own window limits. We passed windows of one second: 13 frames each.

Then the ears can answer after every second of sound.

We compared the output frames of the two runs. On average they are 0.78 alike. The worst frame is 0.24 alike.

So the ears give a clearly different answer when they see less.

We did not run the writer. So we do not know how much the text changes.

The authors say they trained with windows from 1 to 8 seconds. That may make the writer cope.

The NVIDIA live ears use a view that moves with each frame and looks a few frames ahead.

Qwen uses fixed fences. A frame at the end of a short window sees almost nothing after it.

The report presents three models. Two of them write text from speech.

The third finds the time of every word in a recording.

Older recognisers match sound patterns to text, from the bottom up.

This kind first forms an understanding of the sound, then writes. That is the authors' own description.

The 1.7B model is the best open recogniser, they say. It matches the strongest closed services.

The 0.6B model gives the best accuracy for its size.

The authors say public tests are worn out. Scores have reached the level of labelling mistakes.

So they built private tests for noise, dialects, old people and children.

The values of all three models are open. So is the code to run them and to fine-tune them.

The ears were trained on their own first. They cut time by 8 and give 12.5 frames a second.

The paper gives them 300 million values. We counted 314 million.

Two small maps change each frame from 1,024 to 2,048 numbers. That is the size the writer reads.

The writer is Qwen3 with 1.7 billion values. It reads the frames like words in a sentence.

Quantizers come up often. This model has none. A frame is never turned into a code.

So nothing in it removes noise by choosing a nearest code.

Qwen3-ASR 0.6B has ears with 180 million values and 896 numbers per frame. Its writer has 0.6 billion.

The ears use attention windows from 1 to 8 seconds. The authors say this serves live use and whole clips.

The ears were first trained as a recogniser of their own. They used about 40 million hours of sound.

The text for it was written by other models, not people. Most of it is Chinese and English.

Ears and language model were trained together on sound, pictures and text. The amount is 3 trillion text pieces.

This stage is the same as for the Qwen3-Omni model.

Then the model learned one job in one fixed format. This data is much smaller and was not used before.

It also holds sound with no speech, live-style cuts and hint words.

Last, a short stage rewards better transcripts. It uses about 50,000 clips.

The authors say it plays an essential role in noise, stability and hard cases.

35% of the clips are Chinese and English. 35% are other languages.

30% are chosen for hard conditions. The authors call them functional data.

The paper does not say how much noise or music the 40 million hours held.

It gives no score before and after any stage.

The model always writes the same three things: the word language, the name of the language, then the text.

It names the language before it writes any text. That is how it tells languages apart.

There is no separate language detector.

For sound without speech it writes: language None, and no text at all.

It was trained on such sound on purpose, so it does not invent words.

You can put background words in the system prompt: names, product terms. The model uses them when it writes.

The paper gives no score for this feature.

If a speaker says a command, the model writes the command down. It does not follow it.

The authors trained this to stop injected instructions.

The two recognisers support 30 languages. Hindi is the only Indian one.

They also support 22 Chinese dialects, and English from many countries.

So the paper counts 52 languages and dialects in all.

One clip can be up to 1,200 seconds, which is 20 minutes. The same model works live and on whole clips.

It reads speech, singing voice and whole songs with background music.

The aligner supports 11 languages and clips up to 300 seconds. It gives all time marks in one pass.

They served many callers at once. Each clip was about 2 minutes long.

They used a fast serving program and 16-bit numbers.

With one caller the small model handles 106 seconds of sound each second. The big one handles 67.

With 128 callers the small model reaches 2,000 seconds each second. The big one reaches 1,220.

With one caller the first text comes after 92 ms for the small model. The big one needs 102 ms.

With 128 callers it is over 3 seconds. One in twenty callers waits over 6 seconds.

The aligner handles up to 962 seconds of sound each second. It needs no step-by-step writing.

Three are closed services: GPT-4o Transcribe, Gemini 2.5 Pro and Doubao ASR.

Three are open models: Whisper large-v3, Fun-ASR MLT Nano and GLM-ASR Nano.

The authors ran the closed services and Whisper themselves. The Fun-ASR numbers come from its own report.

Public English and Chinese tests. The authors' own hard-case tests. Many-language tests. Singing and songs.

The score is the share of words written wrong. For Chinese, Cantonese and Korean it counts characters.

Lower is better. An average over languages gives each language the same weight.

The hard-case tests are private, so nobody can repeat them.

The closed services were run by the team whose own model is in the comparison.

Seven systems on eight tests. A green cell is the best in its row.

The 1.7B model is best on four tests: LibriSpeech other, GigaSpeech, Common Voice and VoxPopuli.

These hold messy speech from the web and from crowds.

On Tedlium the 0.6B model is best with 3.85. It beats its bigger sister there.

GPT-4o Transcribe is best on LibriSpeech clean and on Fleurs. Gemini is best on MLS.

These are cleaner, read-aloud tests.

GPT-4o Transcribe scores 25.50 on GigaSpeech. Every other system is under 10 there.

The authors note that closed services vary a lot between tests.

The 1.7B model is best on all six Mandarin rows.

On WenetSpeech meetings it scores 5.88. Gemini scores 13.47 and GPT-4o Transcribe 32.27.

Doubao gave no usable result on this test, the authors say.

Qwen is best on six of eight dialect rows. That covers Cantonese and the mixed-dialect KeSpeech.

On the two Sichuan rows the Chinese service Doubao is best. Qwen is close behind.

Chinese and English are most of the training data in every stage. The paper says so directly.

Qwen3-ASR 1.7B is best on all seven rows.

On very noisy Mandarin it scores 16.17. Whisper scores 63.17 and GPT-4o Transcribe 36.11.

Doubao scores 17.04, less than one point behind.

The 0.6B model scores 17.88 in extreme noise. Its writer is three times smaller.

It is best on talk in 16 English accents and in 22 Chinese dialects.

On the dialect mix the gap to closed Western services is very large.

It also leads on old people and children and on tongue twisters. Doubao is close on both.

All seven tests were made by the authors and are private. The noise test is Mandarin only.

On singing alone, Qwen is best on three of four tests. On the fourth it is second by a hair.

Whole songs with music are much harder. Whisper and Fun-ASR could not do them at all in this test.

On whole Chinese songs Qwen is best: 13.91 against Gemini's 18.68.

On whole English songs Gemini is better: 12.18 against Qwen's 14.60.

The table has no column for the 0.6B model. The paper gives no song results for it.

Five open models on seven rows. Qwen3-ASR 1.7B is best on six.

On tests with 8 to 20 common languages it beats Whisper large-v3, by a small step.

On all 30 languages of Fleurs, Whisper is better: 8.16 against 12.60.

The authors say it needs work on less common languages.

The 0.6B model scores 21.80 on the 30 languages. That is almost twice the big model.

The 20-language set adds Hindi and seven more. The 30-language set adds ten rarer ones.

Each language has two bars: the big model and the small one.

Twelve languages are under 6 for the big model. They include Chinese, English, Korean, Italian and Japanese.

Hindi scores 17.15 for the big model and 19.12 for the small one. It is in the weaker half.

Ten languages are above 19 for the big model. Hungarian is worst at 34.22.

In the weak languages the small model is about twice as bad. Hungarian reaches 59.47.

The same languages score very differently from test to test. Each test has its own kind of speech.

German, Spanish and French are between 4.6 and 8.6 on MLS and Common Voice for the big model.

MLC-SLM holds real conversations. There German is 17.19 and French 20.75.

That is three to four times worse than on read speech.

Arabic on Common Voice is 37.97 for the big model. It is by far the weakest row.

The authors also sell a closed version called Flash. It is better than the open 1.7B on most rows.

So the open model is not their best model.

The model writes the language name before the text. No separate detector is used.

On four tests the 1.7B model is right 97.9% of the time on average. Whisper is right 94.1%.

The 0.6B model is right 96.8% of the time. It is also above Whisper.

Most remaining mistakes are between Malay and Indonesian. The two languages sound very alike.

The paper tests full clips of one language. It shows no result for speech that switches language midway.

The authors fed the sound in pieces of 2 seconds. The last four pieces of text may still change.

On average the 1.7B model goes from 2.69 to 3.33 in live use. That is 24% more errors.

The 0.6B model goes from 3.48 to 4.40. That is 26% more errors.

Noisy English is hit most. LibriSpeech other goes from 3.38 to 4.51 for the big model.

Only 2-second pieces were tested, and only on three clean tests. No shorter piece is reported.

There is no live result in noise or for other languages.

You give the sound and its correct text. The model gives the start and end time of each word.

After each word, two empty time slots are put into the text: one for the start, one for the end.

Ears turn the sound into frames. A 0.6 billion language model reads frames and text together.

Then it fills every slot at once. It does not write step by step.

One frame is 80 ms. A slot holds a frame number from 0 to 3,749.

So the longest clip is 300 seconds. Time = frame number times 80 ms.

No one labelled the times by hand.

The loss is counted only in the slots. Slots are put in at random places during training.

On hand-labelled tests the model has 67% to 77% less time error.

On short clips the Qwen aligner is off by 42.9 ms on average. The other tools are off by 130 to 161 ms.

The other tools have no result for Japanese, Korean or Portuguese. Each needs its own model per language.

When clips are joined to 300 seconds, the others get far worse. WhisperX is off by 2.7 seconds on average.

The Qwen aligner stays at 52.9 ms.

On marks made by people it is off by 32.4 ms on average. The two other tools are off by 101 and 141 ms.

With noise added its error rises from 27.8 to 41.8 ms. It is still the best of the three.

The authors give reasons in words. They never switch one cause off and measure again.

The model has no code-books. So they cannot be the cause.

The small model has a writer three times smaller. In extreme noise it is only 1.7 points worse.

GPT-4o and Gemini also use big language models. They score 36.11 and 29.06.

The ears heard 40 million hours. The authors call the result general and stable.

The paper has no number that tests this alone.

The authors say the reward stage plays an essential role in noise. 30% of its clips were hard cases.

They show no score with and without it.

In our test, ears of this kind kept background sound in their low layers.

So the ears do not clean the sound. The model learns to read through the noise.

A strong sound-and-text model, trained in four stages, beats models of its size and larger. It matches closed services.

30 languages with language naming. Hard sound, accents and dialects. Singing. An open time-mark model.

On all 30 languages it is behind Whisper. They call this room for improvement.

On a few dialect rows a closed service is better.

No stage and no part is tested alone. So the cause of the noise strength stays open.

The hard-case tests are private. The training data is not described beyond its size.

Live use is tested only at 2 seconds, on clean speech. And a better closed version exists as a paid service.

Each model heard a clip and got one question with the answers listed. It had to reply with one letter.

There are eight kinds of question. Two of them use Hindi, Tamil and Telugu speech. A model that gives one answer every time scores the red line.

Blue is Step-Audio 2 mini. It tells woman from man and it names the acted emotion.

Its emotion score is as good as our best small reader on frozen ears.

Orange is MOSS-Audio, green is Gemma 4, grey is Phi-4. MOSS gets woman or man mostly right.

Gemma 4 is the only one that knows which Indian language it hears: 63% of 12 choices.

Look at the shaded boxes: background place, line damage and loudness. All four models sit on or near the red line.

We measured the clips to be sure. The noise, the gaps and the low level are really in the sound.

The black marks are from our layer probe. A one-layer reader on small frozen ears answered the same kinds of question.

It gets line damage and loudness almost perfectly. The big models miss both.

One prompt wording, one run, letters only. The emotion is acted English. The damage is made by code.

The next two pages ask the same models in two other ways.

Seven choices may be too many. So we asked each thing as a plain yes or no, on the same clips.

Half of the clips have the answer yes. A model that always says one thing lands on the red line.

The two blue rows are about the voice. Step-Audio 2 mini is right on 96% of the angry-or-calm clips and 89% of the woman-or-man clips.

MOSS-Audio is right on 98% for woman or man.

The six white rows ask about noise, gaps, distortion and a very quiet voice. The dots stay on the red line.

The best single score is 64%. Most are within a few points of 50%.

The boxes on the right give how often each model said yes. The true share is 50% in every row.

Step-Audio 2 mini said no to every distortion clip. MOSS-Audio said yes to 98% of them, and yes to every background clip.

It is not the wording and not the number of choices. These models were not trained to judge the recording itself.

Our tiny reader on frozen ears gets line damage and loudness almost perfectly. So the information is in the sound.

There are 35 clips: clean ones, noisy ones, damaged lines, quiet ones, acted emotions and Indian languages.

The yellow line is the truth. We know it because our code made the clip.

It names woman or man and the mood well. It sometimes hears a real noise, for example wind in the park clip.

For all six damaged lines it wrote that the audio quality is good. It named the Indian language correctly on 2 of 15 clips.

It gives the richest answer: age, voice colour, pace and accent. It is the only one that hears a bad recording.

It called the three damaged English lines low quality and the clean ones clean. It does not say what is wrong: gaps and a phone line are both "muffled and distorted".

It named the language correctly on all 15 Indian-language clips. For everything else it gives one stock sentence.

"Neutral tone, no background noise, audio quality is clear", whatever the clip holds.

It answers with a short caption like "A man is speaking". It does not follow the request.

It is built for speech to text and translation, and it shows.

Inside Qwen's ears a frame looks at other frames in a window. The released setting is 8 seconds.

We set it to 1, 2, 4 and 8 seconds and to the whole clip. The weights stayed the same.

The green lines are clean recordings. They are almost flat.

English goes from 1.6% wrong words to 1.7% when the window shrinks to 1 second.

The orange lines have a real place mixed in, speech 5 dB louder than the noise.

English stays near 2.4%. Hindi rises only for the 1-second window.

The red lines have noise as loud as the speech. Now 1-second windows hurt.

English goes from 4.1% to 6.4%. Hindi goes from 20% to 28%.

Look at the shaded band. From 4 seconds up, every line is as low as with the whole clip.

So the ears use about 4 seconds of sound around a frame, and more only helps in loud noise.

200 clips per language, one run. The Hindi score counts spelling differences as errors, so read its shape, not its height.

The window cuts both sides of a frame. This is not yet live use; the next page is.

Drag the run slider. After each new piece of sound the model runs once and the text on the screen grows.

Black text is kept from the run before. Blue text is new, or written again.

After every run the text covers the sound that has arrived. Nothing waits for the end of the clip.

With a run every 0.5 seconds, the first words show after half a second of sound.

In some runs, words already on the screen change. The yellow note marks such a run. A change of only a comma or a capital letter does not count.

It happens in about one run in ten for English and one in four for Hindi. Runs that change more than three words are about 1% or less.

The left chart compares the final text with the text from the whole clip at once. English stays at 2.1% wrong words, however often it runs.

Press Hindi. It goes from 13.3% to 14.7% with a run every 2 seconds. With a run every 0.5 seconds it is 21.3%. We did not find the cause.

Every run hears all the sound again from the start. The right chart counts that work.

With a run every 2 seconds the ears process about 3 times the clip. With a run every 0.5 seconds it is 9 times for English and 11 times for Hindi.

One caller at a time, so the times are rough.

60 clips per language, each 4 to 20 seconds. On longer talk the repeated work grows further.

A tiny reader looks at one layer of frozen ears and answers a question about the clip.

New here: the ears of Qwen3-ASR in two sizes, and a background test on a place never heard.

Blue is the live 0.6B ears and grey is Whisper. This repeats page F1 with the same result.

The live ears know the line and the loudness low down and lose them at the top.

Orange is the 1.7B Qwen ears and green is the 0.6B. Low down they are as good as the others.

Line damage 100%, loudness 99%, background 95%, emotion 78%.

Read the orange text under each chart. The last layer of Qwen's ears still answers most questions.

The live ears do not. Their last layer falls to 47% on line damage and 68% on woman or man.

The shaded chart is the new question. The reader learned each kind of place from one recording and was tested on another.

All ears score about 40%. So the 93% on page F1 was partly the reader knowing that one recording.

English clips, acted emotion, damage made by code, one run. About 300 test clips for most questions.

One recording per kind of place is very little to learn from. More places may lift that score.

No open audio model could say what is wrong with a recording. So do not wait for a judge.

Measure the damage with code. The answer is exact and costs nothing.

Step-Audio 2 mini named acted emotion at 75% and angry or calm at 96%. That is good enough to read tone from speech.

It was tested on English only. Test it on Indian speech before you trust it there.

Qwen-style ears keep the voice and the line up to their last layer. The live ears we tested lose them.

So a small model built on Qwen's ears can feed the writer and the decision reader from one place.

Four seconds of sound around a frame gave Qwen's full quality. The whole clip is not needed.

That fits a live small model with a short window and a cache.

Qwen's own live recipe redoes old sound in every run and Hindi lost quality when it ran often.

Build the small model to hear each second once.

One test did not run: a reader trained on Hindi only, tried on Tamil, Telugu and Bengali. The method is ready to run.

Also open: real phone calls, real emotion, and many more recorded places.

Each clip is cut after 0.5, 1, 2, 3 and 5 seconds. The ears hear only that much, as in a live call.

A small reader then answers one question from what the ears made. The ears are never changed.

Four questions about the sound: noise behind the voice, a bad line, too quiet, echo.

After half a second the reader is right 88% to 93% of the time. After 2 seconds it is at 94% to 98%.

Man or woman is 64% after half a second and 97% after 2 seconds. A voice needs some time to show itself.

Which of 12 languages is spoken needs the whole clip: 15% at half a second, 86% at the end.

Which kind of place the noise comes from: 46%, where guessing gives 17%. The test places were never heard in training.

So the reader can say "there is noise". It cannot say "you are in a car".

After 1 second it misses three cases: a second voice, an overdriven line, and faint noise. The first two get better with the whole clip.

It also flags 4% to 6% of clean clips by mistake. For echo it is 10%.

The best layer for these questions is a low or middle one, not the last. The last layers keep words and drop the rest.

In the live Conformer the last layer drops "too quiet" from 97% to 73%. Man or woman drops from 98% to 73%.

The damage was added by code to read speech. Real phone calls can differ.

One run. Each reader is one small layer. A real reader can do better.

Each row is one set of ears. The number after the name is its size, in millions of numbers.

After one second, even the smallest ears flag a bad line 94% of the time.

With the whole clip, all nine sets are close for noise, bad line and echo. Size helps for the last column only.

Which language is spoken goes from 75% with the smallest ears to 88% with ears ten times larger.

The smallest ears are 38 times smaller than Qwen's large ears. On bad line they lose 2 points.

So the decision part does not push the size of the model up. The writer does.

wav2vec2 and HuBERT score about 65%. Their own settings level the loudness of every clip before the ears.

So the volume is gone before the first layer. Ears for our model must get the sound at its true level.

These ears were all trained on other tasks. We only read what they already keep.

A small model trained for these tasks can keep more. This page shows the floor, not the ceiling.

The top row is a reader trained on Hindi clips only. It never heard the other eleven languages.

It still flags a bad line in most of them. A bad line sounds the same in every language.

The Hindi-only reader drops for Punjabi and Odia. It is at 73% and 66% there.

Trained on all twelve languages, the same reader is at 90% or more everywhere.

The voice question also carries over. One language is weaker in every row: Malayalam.

We did not find the cause. The test has 120 recordings for each language.

The bars show the average over the eleven languages that the Hindi reader never heard.

Adding the other languages to the training closes most of the gap.

This is the clip from the Qwen pages. At 1.9 seconds it has written four words and a part of a name.

The newest piece is often half a word. It is completed in the next run.

At 2.5 seconds the year is written as three words. At 3 seconds the same year is written as the number 1537.

So words that looked finished were changed more than a second later.

We did this for 150 Hindi recordings. For each word we took the time from first showing to its final form.

On clean sound, 55% of words are final at once and 84% within 2 seconds.

With noise, 41% of words are final at once. On a bad line it is 38%.

One word in four on the screen will still change. On clean sound it is one in seven.

The same test with Model B on 60 clean recordings of each language. On average 48% of words are final at once, and 79% within 1 second.

Tamil and Malayalam are the hardest, at 28% and 27%. A likely cause: their words are long and grow as more sound comes.

It can wait before it shows a word, or it can show drafts and mark them. Both cost something.

Qwen's own live code keeps old text fixed. That is why its Hindi got worse in our earlier test.

Chapter G5 followed one frame of one clip. Here are all frames of 360 clips.

The line is the average. The band holds 8 of every 10 frames.

With no later sound a frame is 0.41 alike to its final answer. With 1 second of later sound it is 0.93.

After that the gain is slow: 0.96 at 2 seconds and 0.98 at 4 seconds.

The same test at the middle layer: 0.82 with no later sound, 0.98 after 1 second.

This fits chapter J1. The sound decisions sit in these layers, and they need little later sound.

Qwen's 0.6B ears give almost the same lines. The effect comes from the design, not from the size.

Frame 24 of chapter G5 started at 0.27. It was a harder frame than the average.

The ears hear the sound so far, as in a call. Every half second the reader looks at the newest second of frames.

The change is at 4 seconds. A good reader is quiet before it and flags after it.

These readers were trained on steady clips only: all clean or all bad. One second after the change they flag 46% to 85% of the clips.

Before the change they flag few clips: 1% to 16%. That part works.

Now the line is bad first and then fine. With Qwen's ears the reader still flags 23% of the clips, 4 seconds after the line is fine.

It looked as if two-way ears keep the old sound in the new frames. That was the wrong reading.

We added changing clips to the training and marked each second with its own true state. Nothing else was changed.

One second after the change, 91% to 97% of the clips are flagged, across five sets of ears. One second after the line is fine again, 3% to 9% are.

For noise the reader follows the change in one second too. But it flags about 1 clean moment in 6.

One second of sound is short for noise. The clip-level reader of chapter J1 had 6% false alarms.

The delay is the length of the piece that the reader looks at. With that training, even the 8-million ears follow the change.

The training data was the limit. A reader must see changes to learn that a call can change.

Each second has its own true state.

The second that holds the change is marked as mixed. A reader should skip it in training.

Actors say the same sentences in an angry voice and in a neutral voice. The reader hears only the start.

After half a second it is right 88% of the time. After one second it is at 94%.

Choosing among six emotions reaches 77% with the whole clip. Guessing gives 17%.

Age in three bands reaches 52%, where guessing gives 33%. The ears keep little about age.

The grid shows all nine sets of ears after 1 second. Angry or calm is 89% or more for all of them.

Larger ears help for six emotions and for age.

These are actors, in English, in a quiet room. A real angry caller sounds different.

We have no Indian speech with tone labels. This page shows that the signal is fast, not how well it works on calls.

A reader of one layer on a middle layer of the ears answers the sound questions after half a second.

It reads from the ears and writes nothing into them. So it can be added or removed without any change to the writer.

Ears of 8 million numbers already carry noise, bad line, low volume and echo. So a budget under 0.5 billion is not at risk from the decision part.

One rule for the ears: do not level the loudness before them.

Use short pieces, all languages, and clips that change in the middle. Each of these three fixed a failure tonight.

Model B writes all twelve languages. Qwen writes Hindi only, of these twelve. Where they differ, hear the clip before trusting either.

About one second of later sound is the useful part for the ears. Half of the words are still drafts when they first show.

So the small model should hold words back for 1 to 2 seconds, or mark them as drafts.

Tone on Indian speech has no labelled test set yet. Real phone calls were not tested, only damage made by code.

Age and the kind of place are weak. No speech model was trained tonight. The ears stayed frozen, and only one-layer readers were trained.

Your limit is 500 million numbers. At 16 bits for each number, that is 1.0 GB of memory.

The big Qwen ears leave 0.37 GB for the writer. The small Qwen ears leave 0.63 GB.

We searched 36 labs for small writers and counted each from its weight file. The smallest Gemma 4 needs over 9 GB, so it is out.

The 1-bit and 3-level writers are large in numbers but small in memory. Three of them fit behind the small ears.

A writer puts out pieces of words, one piece for each step. Fewer pieces means fewer steps of work.

This Tamil line has six words. One writer needs 12 pieces for it. Another needs 103.

The Gemma table has pieces for Indian scripts. It needs about 2 pieces for a word.

The 1-bit writers are made from Qwen3. They keep its table: 7 pieces for an Indian word.

We let each writer guess real lines from a test set. A writer that knows the language guesses well, so its number is low.

Among the writers that fit, the small Gemma knows these languages best. The squeezed writers come next.

A 1-bit writer is a normal writer with each number cut to one bit. It loses some of what it knew.

The loss is larger in Indian languages than in English. And its maker gives no way to train it further.

We tested writers that Indian teams tuned or built. Two large ones know these languages best of all, but need 3 to 5 GB. None that fits beats the small Gemma.

The writer inside a live Indic model is not a language model. It is a large part of that model, and it works only with its own ears.

Of the 268 million numbers in the small Gemma, 168 million are its word table. Most of those rows are never needed for the languages here.

With only those rows, the writer needs 0.27 GB. Then it also fits behind the big ears.

Four stacks fit under the limit. We recommend the small Gemma as the writer, in row 1 or row 2.

The release is open and not for profit, so the licence is not a limit here.

Ears turn sound into frames. Something must then turn the frames into text.

These three models of 2026 do it in three ways: votes, a small predictor, and a language model.

We gave all three the same real clip. The words that each one wrote match the caption.

They differ in capitals, in marks, and in what they do at the end of the clip.

The model with votes took 18 ms. The one with a language model took 625 ms.

The reason is in the design. A writer works once for every piece, and votes come all at once.

The fastest one is English only, and it cannot run live. The live one has the worst English score.

The one with a language model wrote the most complete text here. It is also the largest and the slowest.

Each design answers one of our open questions: the cheapest writer, live ears, and how to join a writer.

The next three pages open each model, one step at a time.

The sound goes through 16 layers. At the end, each frame gives one score to every word piece the model knows.

No second part writes the text. The frames vote, and the votes are the text.

The sound is first turned into a picture with 80 bands. The model also gets how fast each band is changing.

Two picture columns side by side make one row of 320 numbers. One row is 20 ms of sound.

The first two layers each join two neighbour rows into one. So 492 rows become 246, then 123.

From layer 3 to layer 16 the length stays at 123. One frame is now 80 ms.

Each layer has four parts: a small network, attention, a convolution, and a second small network.

The 16 layers hold 439 of the 473 million numbers. The rest is the vote layer and the way back from votes.

The frames are cut into blocks of 128. A frame looks at the other frames of its block and no further.

Inside the block it sees later frames too. So the model must have a whole block before it can start.

After layer 8 the model votes once. The votes are turned back into numbers and added to every frame.

The last 8 layers work from this first guess. When we switch it off, most of the words are lost.

Here are all 123 frames. Most of them vote for "nothing here", because a word piece is shorter than its sound.

Only 26 frames vote for a piece. Those 26 pieces are the whole text.

Three simple rules turn the votes into text. No part of the model is run again.

All frames vote at the same time. So the time does not grow with the number of words.

It is small and very fast, and on this clip its words match the caption.

It lacks four things: marks and capitals, other languages, live use, and knowledge of words.

The model before this one needed the whole sound first. This one is built to work while the sound arrives.

It has 24 layers of ears and a very small part that writes. One frame is 80 ms.

In the model before, a frame saw all frames of its block. Here a rule limits what a frame may see.

It sees about 4.5 seconds back. Forward, it sees only to the end of its chunk.

We fed the clip in real chunks. Between chunks the model keeps a small memory of what it needs from before.

The frames came out the same as in one pass. So the frames do not change when it runs live.

One model serves about 40 language regions. A tag tells every frame which one to write.

The tag can also say "find it yourself". On our two clips that worked as well as the right tag.

The part that writes asks one question for each frame: which piece is next, or none?

It uses the frame and a small memory of the pieces so far. If a piece wins, it asks again on the same frame.

This is the real run. Most frames write nothing, and some write two or three pieces.

The text grows from left to right as the frames arrive. Nothing written is changed later.

The chunk size is a setting, from 80 ms to 1.1 seconds. The same model does all four.

On this clip the words are the same in all four settings. Only the end of the sentence differs.

The ears are 95% of the model. The part that writes is under 4%.

Each chunk takes about 19 ms of work, so the model keeps up with live sound.

The two models before had no writer, or a very small one. This one has a full language model as its writer.

The ears give 128 frames for our clip. A small bridge makes each frame as wide as one word of the writer.

The writer is given a short prompt with 128 empty slots. Each slot is filled with one frame.

So the sound does not go through the word table. The frames are put straight into the row of words.

The writer reads all 143 rows and writes one piece. Then it reads again, with that piece added, and writes the next.

It first writes which language it heard. Then a mark, then the text, then a sign for the end.

For each piece we looked where the writer was looking. The place moves forward through the clip, piece by piece.

No rule in the code forces this. The writer follows the sound from left to right by itself.

We cut the English clip short and let it write. It ends each cut with a full stop.

So a cut in the middle of a phrase gives a wrong ending. The text must be written again when more sound comes.

The same model writes Hindi. It heard the language at once, and most words are right.

It wrote the year wrong, and it needed 102 pieces for one sentence. Its word table cuts Hindi into many pieces.

The writer is 76% of the model. Each piece costs one pass through it.

So this model is the slowest of the three on the same clip. On this clip it also wrote the most complete text.

The makers of our ears trained their model in four stages.

We start from their finished ears and train further.

The ears turn sound into frames.

Two readers look at the same frames: the writer and the vote layer.

The writer must predict each next piece of the true text.

It is of the same family as the objective of the first stage of their makers.

The vote layer must name the pieces from the frames alone.

The training code needs no word timings.

A writer can guess likely words. The vote layer reads each frame by itself.

Three small tests compared the mixes.

The new parts learn fast, from the first step.

The trained parts wait, and then move slowly.

A few hundred steps.

It shows that the code learns. It does not judge the method.

The ears come from Qwen3-ASR. They turn sound into frames of 1,024 numbers each.

The writer is Gemma 3 270M. It reads one vector of 640 numbers per piece of text. The sizes do not match.

Inside Qwen3-ASR, two layers turn each frame into a vector its own language model can read. That vector has 2,048 numbers.

These two layers were trained on exactly these ears. We copy them as they are.

The vector is now in Qwen's word space, 2,048 wide. Gemma reads 640.

One new layer, called the map, turns 2,048 numbers into 640. Now Gemma can read each frame like a word.

A random map would hand Gemma nonsense at the first step.

So the map starts from a fit: for text pieces both models know, it turns Qwen's vector into Gemma's vector. Then training adjusts every part.

Sound, ears, the two copied layers, the map, then Gemma.

The frames sit in the row where words would go. After them comes a "now write" mark, and Gemma writes the text piece by piece.

Over the run, the letters wrong on the test clips fall.

One rise in the middle needs an explanation.

Most languages are near or under 10%.

One language is far behind.

The writer writes one piece at a time.

It must pick the piece "end" by itself. Nothing else tells it that the clip is over.

On this real clip the writer never picked "end".

It wrote one piece 197 times. We call that a loop.

A clip that runs on can have far more wrong letters than true letters.

So five clips held most of the wrong letters of English.

Most of these clips left the sentence early, before half of it was written.

The loop came after. The guards in the next steps limit the damage. They do not bring back the lost sentence.

The writer is stopped when the end of its text is a loop.

The clip is still wrong. The guard only stops the damage from growing.

The writer may write only so many pieces for each frame of sound.

Our first limit was too low for Odia. That was our mistake.

The run now flags texts that loop.

Fewer clips run on than at the worst point. It is better, and it is not solved.

The run is done.

Some things are still open.

All languages together are steady.

Hindi and English jump up and come back.

The writer heard the Hindi sentence.

It wrote it in the letters of another language.

It happens in 14 to 60 of 960 test clips.

The count went up and down, and it was lowest at the end of the run.

Nothing tells the writer which language the clip is in.

One stray piece of another script can also start a run-on.

We wrote the same clips again with other scripts blocked.

On the clips in another script, most wrong letters are gone, with no training.

On a public test, Gujarati is one of the best languages.

The Gujarati test texts look like the problem. We did not listen to them yet.

In six of the seven languages shown, short clips score worse than long ones.

Three changes follow from this page. None is done yet.

The speaker says "pull". Hima writes "pool".

The two words sound almost the same. So the sound reached the model, but the written word is wrong.

At that spot, the writer gave "pull" a chance of 46%.

The writer takes its top choice at each step. With "pull" under half, a close neighbour could score higher. "pool" did.

The writer builds words from small pieces of text. A rare word needs many pieces.

For the town "कणकवली", the right word had a 52% chance after the right words. But Hima had already split the word before it. From there it chose other pieces, one by one: "कण कव्वाली".

"nouns" became "now as". "outer" became "air of". "Addenbrooke's" became "a second brook's".

Each wrong text sounds like the right one. It is made of pieces of common words.

We sorted all 7,242 wrong words by the chance of the right word.

In 40% of them, the right word had a real chance, between 5% and 50%. The writer was not sure.

Words of one piece are wrong 8.9% of the time.

Words of four or more pieces are wrong 34.7% of the time. Every extra piece is one more place to choose a wrong path.

A wider search keeps several guesses at once. It helps only if the model scores the right text higher.

That was true in only 175 clips. In 2,199 clips, the model scored its own wrong text higher.

One writer writes all 12 languages.

It may use any of ten alphabets, and nobody tells it which.

Four saved models wrote the same Hindi clip.

One wrote it in Marathi words, one in Odia letters.

A Hindi reader cannot use a text in Odia letters.

At one saved model, 26 of 80 Hindi clips came out like that.

The writer gets only the sound.

Its first letter picks the alphabet, and that first letter is a guess.

Put a label in front that says the language.

The app writes the label, and the writer reads it first.

We tested a stand-in: the other alphabets were closed.

The clips in a wrong alphabet were mostly repaired.

English words inside Hindi are written in Hindi letters.

So a Hindi label should not stop them.

A fixed label would fail when the app is wrong.

It would also fail when the app does not know the language.

The writer writes the label itself, as the first piece.

The app may write it instead, when it knows the language.

A second label can say how to write English words.

Hima writes only one style today.

Three choices, side by side.

We would take the third. It is not trained or tested yet.

Normal clips are fine.

The model breaks on very short clips, on long audio, and on a second voice.

One short word is often right.

When it is wrong, it is often written in another language's letters.

Very short clips are the hardest.

Half of them come out wrong, mostly in the wrong alphabet.

Up to about 20 seconds almost nothing is lost.

At a minute the model writes a few words and stops.

Traffic and metro noise cost a few points.

Cafe noise costs most, because it is other people talking.

A telephone channel costs about 2 points on the public test.

It is a simulated channel, not real calls.

The model cannot tell which voice is the caller.

When the other voice is as loud as the caller, the caller's text is lost.

One Hindi clip came out three different ways at earlier saved models.

The final model writes it right.

One Hindi test clip in nine has letters of another alphabet.

Sometimes a whole word, often only part of a word.

Gujarati, Punjabi, Tamil and Telugu lead.

Bengali is only fifth.

The letters are not broken bytes.

The mixed letters are not a decoding error.

At the stray piece, the other alphabets hold three times more chance than Devanagari.

The model believes its wrong letters.

Right label, no label or wrong label: the result is the same.

In training, the label never told the writer anything new.

Most stray pieces continue a word.

Few start one.

Most Hindi words are one piece.

So the writer rarely writes Hindi word endings by itself, and borrows endings from other alphabets.

Strays sit on words Hima did not know.

The sound is usually wrong too.

Forbidding other alphabets gains under 1.5 words per 100 in Hindi.

It does not fix the word.

Train Hindi word endings piece by piece, and make the label count.

No hard block, because people switch languages.

Hima makes 2.5 to 7 times Pulse's errors.

It is also behind the open IndicWhisper model.

About three times Pulse's errors, in every kind of sound.

Hard sound adds the same share for both models.

The gap is already there on clean sound.

The wrong alphabet is a small slice.

Most lost words are heard wrongly.

Score the later steps tonight. Then fix the alphabet, handle the words it does not know, and add reward learning.

The same clips for all three models.

Some clips only Hima gets wrong.

Some clips all three get wrong.

Model B and Sarvam both get these right.

One of the other two is wrong as well.

No model gets these right.

Model B and Sarvam both get these right.

One of the other two is wrong as well.

No model gets these right.

Model B and Sarvam both get these right.

One of the other two is wrong as well.

No model gets these right.

When the two agree, their text is closer to the truth.

6.87% of words wrong, on average, over 8 Vietnamese tests.

3,713 times real time, from the card.

Sound becomes log-mel numbers, then flows through three model parts to text.

The ears are EmbeddingGemma 2's audio tower, about 305M weights, found by subtraction.

10 of the 12 ear layers were frozen, so they never changed.

Trained in the run: the top 2 ear layers, the 13 layer-mix weights, the reader and the head.

A writer that writes piece by piece loops once per piece. A word is often several pieces.

This model reads all frames at once: one pass per clip.

It turns an input into one point of 768 numbers.

A sound and its text land close together.

The audio tower keeps time: one frame every 40 ms.

One second of sound gives 25 frames.

An embedding model squeezes its input into one vector.

QUAN keeps every frame and passes all of them on.

The tower has 13 outputs: one after the first step, one after each layer.

A learned weight for each output sets the mix.

Lower layers keep more sound detail: the recipe's idea, not measured here.

At step 250, outputs 1 to 4 hold 0.71 of the weight; the last holds 0.11.

The reader's 768 numbers become 512, the size of Gemma's text vectors.

Each frame is scored against 14,917 units, plus blank and unk.

Its training config starts the unit rows from Gemma's text vectors.

One second of speech is 25 frames, 40 ms each.

The strip is an example, not a real recording.

Each frame guesses one unit, or blank.

Blank means: no new unit starts here.

Repeats in a row merge into one unit.

Blanks are dropped. What is left is the text.

Vietnamese has about one unit per syllable.

Assumed rate, not measured: 4 to 6 units a second, against 25 frames.

All frames are read in one pass, with no loop per piece.

Streaming uses pieces of 12 frames, 0.48 s, that look only back.

The sound part is a Google audio encoder, built like USM.

Only its top 2 of 12 layers are retrained.

It has 14,917 units, taken from Gemma's tokenizer.

That is about one unit per Vietnamese syllable.

The head starts from Gemma's own text vectors, 512 numbers each.

Its config file sets the head to start from text vectors (head_init: text).

The training data is about 2,800 hours of Vietnamese, from several sets.

The next page shows how close those sets are to the tests.

Average error over 8 tests, lower is better.

QUAN 6.87%, Qwen3-ASR-1.7B 7.01%, Whisper large-v3 18.04%.

Six of the eight test sets come from corpora on its training list.

The card does not say whether test utterances were excluded.

GigaSpeech2 is not on its training list.

QUAN has 12.34% errors; Qwen3-ASR-1.7B has 8.47%.

On the card it is the worst of the top seven.

On the training log's average of 7 test sets, error rises from 6.14% to 7.47%.

Both numbers are from step 300 of training.

It writes no capitals and no punctuation, and numbers only as words.

An optional 4-gram word model outside it scores which word sequences are likely.

A word the units cannot spell may be lost.

Units are sub-word pieces. Whether this happens: not tested.

The published log covers steps 25 to 300.

The first check (step 100) is already 6.23%, a 7-set average.

Hima's writer writes piece by piece, in a loop. A word is often several pieces.

QUAN's head and CTC read the whole clip in one pass.

Its vote layer is 25.7M, used as a helper (x0.3) in Hima's runs.

Measured: the writer 8.8% letters wrong, the vote layer 15.1%.

QUAN's reader has 6 attention layers, mixes all layers, and starts from text.

Ours is a 2-layer MLP with no attention, used as a helper (x0.3) in Hima's runs.

Vietnamese fits: about one unit per syllable.

Longer Indian words split into several pieces. Not measured yet.

We built a QUAN-style reader on Hima's own ears. A short test run has started; the full run is queued.

Then compare letters wrong on the same clips.

Each label piece is pushed up, one at a time, toward the right answer.

The example sentence is invented to show the idea.

The model writes its 4 best guesses for the same clip.

Each guess gets its word errors against the label. They differ only in है/हैं or के/की.

Guesses with fewer word errors gain probability; the others lose it.

The bars are illustrative, not measured.

The best of the 4 guesses has 0.202 word errors per word.

The top guess has 0.239. That gap is what MWER chases.

Total gap: 11.1 before, 10.5 after round 1, 9.8 after round 2.

Round 3 gave 10.1 to 10.4, so no further gain.

Half B total: 159.17 after round 2, 159.66 after round 1, 161.04 before.

Half B is every other clip of each test set.

At each box, take the piece with the bigger score and move on.

Fast, but an early small choice can lock out a better sentence.

At each step, keep the four best partial sentences.

At the end, pick the best whole sentence. In the example, that beats greedy.

Each frame chooses a piece or "nothing yet". A penalty makes it write more.

0.5 helped a little; 1.0 and 1.5 were worse.

Half A is the half used to choose. Lower bars are better.

Beam search beats greedy by about 3 points. Blank penalties barely move it.

On the full test, beam 4 scores 7.5; beam 8 with penalty scores 7.9.

Beam 4 is the final decode, with no penalty.

A separate text model helps Lahaja and Gram Vaani.

It makes Monsoon worse, so we do not use it.

Two saved models' weights are averaged, cell by cell, into one model.

No extra training is needed to make it.

Uniform: every saved model counts the same.

Greedy: add a model only if the score does not get worse. Best-of: take the best model per set, then average.

Plain average scored best. Every soup scored worse.

The axis starts at 168, so small gaps look big.

Each try added one more model to the soup.

Every addition made the score worse, so none was kept.

The dev set ranked models in the opposite order to the tests.

Its references follow the training labels' style, so it misled us.

Every other clip of each test set goes to half A, the rest to half B.

Choices use only half A. Half B is the check we report.

One caller clip goes in, with several typed questions about it.

No transcript is written in between.

Yes or no, pick one from a list, or score a level on a scale.

Each option gets one probability.

Its checksum, called a fingerprint, is the same before and after the probe.

We only add to it.

Frozen Hima stays grey. A small blue add-on is the only part that learns.

The gold spot is where the answer is read.

Each part is grey, which means it is locked and never changes.

The encoder is the largest, and the writer is close behind.

The encoder cuts the sound into frames, one list of numbers each.

The bridge turns each frame into a vector the writer can read.

One block starts the row. The sound frames follow, one vector each.

The last block says the sound is over: write the text.

A piece is a small part of a word.

Compare what it wrote with the true text.

Same three frozen parts, same row layout, a different language.

Its transcript is also in Telugu script.

The sound fills the row, and the last block says to write.

Nothing after it asks a question.

The sound stays the same. A new blue block, now decide, takes the place of now write.

The question follows, written in the writer's own style: lowercase, no punctuation.

Each option is labelled with a number: 1, 2, and so on.

The number is one piece. The option words take their own pieces.

Each question is a different text, so it has a different number of pieces.

The shortest question is about a fifth the length of the longest.

The question ends with "answer", the last spot in the row.

At that spot the writer gives a score to all 262,144 pieces.

Plan: the answer is one option number, so only their scores matter.

Their scores become chances that add to 1. Nothing is written.

Untrained, the letters A and B split the chance almost evenly.

The known answer for this clip is 0.08 for yes and 0.92 for no.

Only a small slice of the writer's pieces was ever used in its training.

The unused pieces collapsed onto one shared vector, so the letters look identical.

Numbers are clean text, which the writer already knows how to read.

Untrained, most questions already put the top chance on the right option.

Option 1 is yes and option 2 is no.

The chance of yes is simply the chance of number 1.

Each option gets its own number, one piece each.

The answer is the number with the highest chance.

Each level has a number: not urgent 1, somewhat urgent 2, very urgent 3.

The expected level adds each level times its chance.

Plan: the three question types differ only in their numbers.

One reading of the "answer" spot serves all three.

This memory is called the cache. It holds the sound it has already read.

A new question starts from that memory, not from zero.

Each question got its own full copy of the sound.

Thirteen questions meant the sound was read thirteen times.

The sound is read once. Each question is a short tail after it.

The question pieces are the same text as before.

Counted from the real pieces: the bars are drawn to scale, far fewer with the memory (plan).

The time is not measured yet, so the speed is still a claim to test.

The encoder, the bridge and the writer never change.

Their numbers stay locked for every question we ask.

A weight table is a grid of learned numbers. Each writer layer has seven.

Each one gets a thin blue pair. The thin side is the rank, a width we chose.

With the add-on off, the writer writes the transcript, as today.

With it on, the same writer answers questions. This is a plan.

The sound is read once, by the plain writer, as today.

Each question adds one block and its pieces. The transcript branch reads from the same memory.

A clip goes in, with one question about what the caller says.

The target is the answer we want. It is green on the drawing.

Loss is the score that training lowers. Lower means closer to the target.

Untrained, both losses sit well above their lowest possible value.

Every option has one known answer, a chance between 0 and 1.

Training pulls the head's bars toward these answers.

The same question is shown in two orders of the numbered options. This is a plan.

Both orders must give the same answer, so the order does not matter.

A gradient is the direction that lowers the loss. Plan: only the blue part gets one.

The grey parts stay locked, so Hima's own numbers never move.

A fingerprint is a short code made from every number in the file.

It was the same before and after our probe. Training will repeat this check.

A small memorise test also passed after the fix.

We will test 200 clips and compare the text character by character.

Plan: every clip must match today's transcript exactly.

Both bars come from the same test: unseen clips and their decisions.

The dashed bar is the writer with the add-on. We still have to measure it.

A transcript-only arm shows what the text alone can do.

Noise instead of sound must collapse. Reordered options must not change the answer.

Option A is Hima's encoder with a published router. Its smoke test passed.

Both get the same clips, the same questions and the same scoring.

Each pair of bars is agreement with the answer key, then balanced accuracy.

The green bars are the new model, the final version. It is the best of the five.

Each bar is the new model minus the first version, for one language.

Hima never learned five languages. Those five lose the most.

Balanced accuracy counts each answer type equally.

On agreement the new model is slightly behind the first version.

Each question shows the first version in grey and the new model in green.

Green means the new model scores higher on that question.

These questions need only the sound: language, turn end, tone, gender, intent.

Gender is the weak spot. The fix we plan is labelled plan on the drawing.

The new model learns only its add-on. The first version trained its whole model.

Hima itself stays unchanged. Speed is not measured yet.

We asked new questions, never seen in training.

Before, the model had learned a fixed list of questions. Now it has met far more different ones.

Real Hindi collection-call replies, each scored for its caller intent.

Right answer: the share of replies with the correct intent. Balanced: every intent counts equally.

Each row is one caller intent. Bars right of the line: the final version is better than transcript, then decide.

Transcript, then decide still wins on replies that hinge on one short word, like "bye" or "no".

Each bar is the final version on held-out clips. The thin mark is chance, the grey bar is the start model. The number on the right is the change from the middle version.

Emotion clips say the same sentence in different emotions, so only the tone tells them apart.

Agreement with the answer key and balanced accuracy, on the test set's 23,660 decisions in our 12 languages.

All the new skills came without losing the old ones.

Five changes, each measured on its own before the next.

Each one names its paper or reason on the drawing.

The encoder hears the clip once and turns the sound into a row of small frames.

The bridge turns each frame into one vector.

For each question the writer reads one row: the sound vectors, the two blue learned vectors, then the question.

The question is written in plain lowercase with numbered options.

The add-on (LoRA) is switched on over the question pieces only.

Over the sound part the frozen writer reads alone, so the same sound can still give a transcript.

At the last piece the writer scores every piece it knows.

We keep only the option digits and turn them into probabilities (softmax).

Tone, voice and speaker live in the encoder's layers.

A learned mix of those layers gives one voice vector for the clip.

These 12 clips come from a held-out test set. The model never trained on them.

Click one to hear it. Its three answers are drawn below.

8 real collection-call replies from a held-out test set, never trained on. The row "text first" writes the transcript, then decides from it.

The final version is wrong on the reply that ends the call. Text first says end call, and the final version says unclear. It also misses one refuse-to-pay reply.

Each row is one caller intent. Right of the line, the final version hears it better than transcript, then decide.

Red rows are where transcript, then decide is better, such as ends the call and refuses to pay.

A real Hindi caller says the money was already transferred.

The clip is 5.32 seconds long.

The encoder, projector and writer are frozen. They never change.

The encoder turns the clip into 69 frames, about 13 a second.

Each question is short text that the frozen writer reads.

The questions sit beside the sound, one by one.

Hima's 586.9 million numbers stay frozen.

The head is 10.2 million numbers, or 17.5 million at the wider size.

Green bars are the known answers for this clip.

They are the target: what training must learn to say.

A 307 million number trunk was too big.

196.6 million of its numbers were a word table: 256,000 pieces x 768 numbers.

The frozen encoder gives 25 views of each frame, and the writer gives 19.

The head learns one blend that adds them into one sound token.

69 frames, plus one summary token.

Each token is 384 numbers long.

A piece is one chunk of text the writer reads.

The wants-a-human question has 15 pieces.

The option text follows the words "option no" or "option yes".

Its pieces are averaged into one gold vector.

Sky bars are question pieces. Gold bars are option pieces.

An option is 2 pieces for yes or no, and 4 to 13 for call intent.

The writer has 262,144 pieces. Only 63,335 are alive.

Plan: the question text would be read once and kept, as a cache.

Each group reads one clip and one question, with its own options.

Two groups sit side by side, as the head sees them.

Block 1 is one grid. Every cell in it is green, so they all look at each other.

Each row is one reader, and each column is what it may read.

An option reads the block 1 memory with cross-attention.

It asks four times, so its answer comes from what the sound says.

The red cross between options means blocked, both ways.

Each option reads the memory alone, so one option cannot pull another.

Question 2 cannot change question 1.

Each group has its own memory, with a red cross between them.

Tested in a small local check: 20 random shuffles of 8 options, at both head widths.

The scores moved with the options exactly, largest difference 0.0.

A published router, vLLM Decision-2.0, one audio model and the Liquid d1-600M trunk work this way.

The same sound appears in every row, so it is read once per question.

FRIDA and the audio part of Liquid d1-600M read the sound once.

Questions read the sound. The sound never reads them back.

Liquid d1-2B and d1-Omni-2B pack up to 16 questions in one row.

One published model packs every question of one case into one row.

All questions sit in one sequence, one after another.

A 109 M head lets the questions look at each other.

Whisper-AT: 7.2 M scored 32.8 mAP, 40 M scored 33.0.

mAP is audio tagging, not a decision score.

The writer reads the question, then the option's words; only the option's own pieces are averaged.

Five options give five vectors, one per emotion.

This is cross-attention: the option asks which sound tokens support it.

Four stacked blocks, so each option asks four times in turn.

The scorer has two layers: LayerNorm, 384 to 384, GELU, then 384 to 1.

Its 148,993 numbers turn each vector into one number; the results are still to measure.

Softmax makes the chances add to 1, so a bigger number takes a bigger share.

Green bars are the known answers: what training must reach.

The four questions have 2, 8, 5 and 3 options.

That is 18 numbers, and each question's chances add to 1 on their own.

Before training, every level gets the same chance.

The known answer is what the head must learn to give.

It adds up the target times the log of the chance, then flips the sign.

Lower is better, and the lowest it can reach is the target itself.

Brier takes the gap at every level and squares it, then adds them up.

It is weighted by 0.5 in the recipe.

For ordered levels, it compares the chance up to each level.

It is used only for the score questions, with weight 1.0.

Example only, with made-up numbers. The loss sees each score plus 0.5 times the log of the prior.

At answer time the raw scores are used, which then lean toward rare answers. It is for choice questions with skewed answers.

Soft cross-entropy 1.0, Brier 0.5, and RPS 1.0 for score questions.

Logit adjustment has strength tau 0.5, and it is not a weight.

Two ideas are left out: an order-twin term and reinforcement learning.

The head cannot see option order, so an order-twin term has nothing to fix.

In one public GitHub issue (#741), one seed, RL scored 79.1% against 80.5% for soft cross-entropy alone. A hint, not proof.

A published router got stuck answering the base rate, in 6 of 15 runs.

The spread inside a row fell from 0.62 to 0.01.

Hosted Liquid d1 picked the first of five identical options 97 to 98 percent of the time.

A multilingual published router picked the last option 285 times out of 290.

Ours cannot see position. Tested once in a small local check: scores moved with the options exactly.

sammyliu's speech head learned 1,330 decisions and scored Brier 1.01 on new clips.

Brier is the squared gap between our chances and the answer. Equal chance is 0.5 for two options.

Liquid d1-Omni-2B scored effective accuracy 85.2 on written and 63.4 on spoken Speech-MASSIVE intents, 0 is chance.

Only the input changed, so the gap is what speech costs.

Today's best scores 0.71 on emotion and 0.41 on age: accuracy against the answer key on our test.

The new head must lift both, and the first four alarms must stay quiet.