Where to split a sentence for streaming TTS is decided by one number

The split point for streaming text-to-speech follows from the real-time factor. I measured that factor on a machine with a full disk, got it 4x wrong, and shipped a constant that was never going to fail loudly enough to notice.

I build a desktop app where you talk to a VRM avatar and it answers out loud. Someone told me the gap between speaking and hearing a reply was too long.

I measured it and fixed it. Then I found out my benchmark had been running on a machine with a full disk and a load average in the low hundreds, which made the number 4x worse than reality, and the design decision I derived from it was wrong for weeks without ever failing.

The conclusion first: the split point for streaming TTS follows from the real-time factor, and you should store the formula rather than the number it produces.

Seconds are the wrong unit

You cannot compare TTS engines by how long they took. Longer sentences take longer. What matters is the ratio against the length of the audio produced.

import io, wave
with wave.open(io.BytesIO(audio)) as w:
    seconds = w.getnframes() / w.getframerate()

rtf = synth_time / seconds   # real-time factor

• r = 1.0 means 10 seconds of speech takes 10 seconds to synthesize
• r = 0.25 means 10 seconds of speech takes 2.5 seconds

Every downstream decision comes out of this one number.

Measurements

VOICEVOX 0.25.2, Apple M4, 10 cores, idle machine.

Note that longer sentences get a better ratio. Short ones carry fixed costs (model setup, leading and trailing silence), so 3 characters at 0.64 is the worst case. That is already telling you not to chop too finely.

Warm up once before measuring. The first call includes model loading and gives you a number you cannot use.

The split point falls out of the ratio

If you synthesize the whole reply before playing any of it, the app is silent the entire time. So you cut the first sentence, start playing it, and synthesize the rest while it plays.

The question is where to cut. Cut early and audio starts sooner. Cut too early and you finish playing the first part before the second part exists, so the sentence has a hole in the middle.

Let r be the real-time factor, D the total audio length, and p the fraction of the sentence in the first chunk.

• first chunk is synthesized at r·p·D
• first chunk finishes playing at r·p·D + p·D
• second chunk is synthesized at r·D (both chunks run back to back)

No gap requires that the second chunk is ready before the first finishes:

r·D <= r·p·D + p·D
   ->  p >= r / (1 + r)

At r = 0.25 anything past 20% works, so you can cut at an early comma and start talking almost immediately. At r = 1.0 you have to wait until halfway. The same code breaks or does not break depending on the machine.

// measured 0.23 to 0.64. short sentences carry fixed cost, so default high
export function splitAtComma(text: string, rtf = 0.5): [string, string] {
  const positions: number[] = [];
  for (let i = 0; i < text.length; i++) {
    if (text[i] === '、' || text[i] === ',') positions.push(i);
  }
  if (positions.length === 0) return [text, ''];

  const minRatio = rtf / (1 + rtf);   // the lower bound, from r

  // earliest comma that satisfies it = earliest possible start
  for (const pos of positions) {
    const ratio = (pos + 1) / text.length;
    if (ratio >= minRatio && ratio <= 0.7) {
      return [text.slice(0, pos + 1), text.slice(pos + 1)];
    }
  }
  return [text, ''];
}

Only the first sentence needs splitting. Everything after it gets synthesized while the previous chunk plays.

The part I got wrong

My first measurement gave r = 1.09. More than 4x worse than the table above. I believed it, concluded that local TTS runs at roughly real time, put r = 1 into p >= r/(1+r), and shipped "split in the middle".

I also tested thread counts and got 1.09 -> 1.47 going to 8 threads, so I concluded there was no headroom in parallelism.

Both conclusions were wrong. Here is what the machine looked like.

The disk was full, so swap could not do its job, and the load average was in three digits. The number was not a lie. That is genuinely how long it took at that moment. Which is exactly why I had no reason to doubt it.

The nasty part is that the wrong conclusion worked. The real lower bound is 20%, so cutting at 50% satisfies it. It was only ever too conservative. It never crashed, never produced a gap, never showed up in a bug report. There was no symptom to chase, so nothing pointed back at the measurement.

What I take from it:

• Check that the machine is idle before benchmarking. uptime and df. Thirty seconds.
• Look at ratios, not wall clock. "4.7 seconds" tells you nothing is wrong. "r = 1.09" is shaped like something you can recognize as absurd.
• Store the formula, not the constant. If I had written p >= r/(1+r) from the start instead of "split in the middle", re-measuring r would have fixed the behaviour everywhere, by itself, the moment I got a clean number.

The third one is the real lesson. A constant freezes an assumption you made on one particular day on one particular machine. A formula carries the assumption as an input, so correcting the input corrects the system.

Two smaller traps on the way

The playback side needs a queue. I sent chunks as they finished and the second one cut off the first, because synthesis is async and chunk two was ready while chunk one was still playing. When you add the queue, watch out that on an HTML audio element ended and error can both fire for the same clip. If your advance handler is on both without a guard, you silently skip a sentence. It reads like a synthesis bug, so I spent a while in the wrong layer.

I measured while the previous reply was still playing. I was watching a lipSyncAttached flag to decide when audio had started, and got 0.1 seconds. It was not fast. The previous answer had not finished. Wait for actual silence before starting the next measurement:

while state["lipSyncAttached"]:
    time.sleep(0.1)

Summary

• Confirm the machine is idle before you measure (uptime, df)
• Use the real-time factor, not seconds
• The split point is p >= r/(1+r). At r = 1.0 that is halfway, at 0.25 it is a fifth
• Keep it as a formula so it follows the machine instead of freezing one day's guess
• Queue the playback, and guard against ended and error both firing

These numbers come from building Wisp, a desktop AI agent with a VRM body. p >= r / (1 + r) is in the shipping code.