ITR Atlas
← All interfaces

MELDER Real-Time Lip-Reading (Pandey & Arif, 2024)

Measured by Pandey & Arif · ACM CHI 2024 (2024)

Video Silent speech 2024

Inputs

The measured or assumed values behind the calculations, each with its source.

rate = 5.59 word/min
Measured end-to-end entry speed, MELDER in the 20-participant stationary study (best of three real-time lip-reading models). The paper defines wpm = recognized words / (speaking time + computation time) × 60/5, so it already folds in the user's silent articulation and recognizer latency: a genuine measured rate, not an assumption. Mobile/walking: 5.31 wpm; a separate head-to-head put MELDER at 5.62 wpm vs Google Assistant voice ASR at 30.54 wpm.
P = 0.802
1 − 19.75% word error rate, MELDER in the stationary user study (≈19.86% in the Google Assistant comparison).
H = 5.0 bits/word
Shannon per-word entropy of English; the same per-word standard applied to the other silent-speech entries.

Strictest ITR

Each scoring method is an upper bound on the channel, so the headline is the strictest (smallest) one for this entry. Use the score selector on the home page to view any single method across entries.

Strictest Shannon (text) Recomputed
Word-entropy throughput (measured end-to-end rate)
Net of English redundancy, at the measured lip-reading rate
0.375 bits/s

  1. Error-corrected words per minute

    (1 − WER) × rate = 0.802 × 5.59 ≈ 4.5 net word/min

    5.59 wpm is a real measured entry speed (speaking + computation time); the 19.75% WER is from the same 20-participant stationary study.

  2. Shannon per-word entropy of English

    H ≈ 5.0 bits/word

  3. Information transfer rate

    4.5 word/min × 5.0 bits/word ÷ 60 s/min ≈ 0.375 bits/s

What counts as a bit depends on the action space. The number of distinguishable actions and how likely each one is are design choices of the task, not the sensing hardware. The same modality can present a fixed set of targets, a set pruned per step by a grammar or language model, or a continuous control space. Each of these changes how many actions are live and how the probability mass is spread, and therefore the information per selection. Read the action space below before comparing headline numbers across entries.

Action space

What the user can produce at each step, and how those options are distributed.

Structure
Context-dependent (the live set changes per step)
Size
Continuous
Prior
Context-conditioned: likelihoods depend on prior actions
Notes
A camera-based silent speech interface: a phone's front camera reads lip movements and a streamlined real-time lip-reading model transcribes silently mouthed phrases to text with continuous on-screen feedback. It is folded under the silent-speech color because it decodes articulation into language, but the sensing is optical (video), not muscle, hence the Video tag. The measured operating point is 5.59 wpm at ~20% WER, against ~30 wpm for voice ASR in the authors' own head-to-head. Because this is a measured end-to-end rate, it is not directly comparable to assumed-rate EMG silent-speech estimates.

Comparability The strictest bound here is the Shannon entropy of the output text, under one predictor held constant across the whole atlas (≈1 bit per character). That shared predictor makes it directly comparable to every other text entry (keyboards, spellers, silent speech and speech BCIs) regardless of prior or vocabulary size. For most text interfaces it comes out tighter than the raw-selection bounds, but not always. Where a small vocabulary makes Wolpaw tighter, that wins instead. Any Fitts, Wolpaw or log₂(N) figure shown below is another bound on the same channel. Switch the home-page score selector to compare one across entries.

Source

Authors
Pandey & Arif
Publication
ACM CHI 2024, 2024
Paper
10.1145/3613904.3642348