The oval finds your mouth by itself (red while it follows it); face the camera, keep still, then mouth a sentence. The words appear below as they are read. Without a server nothing leaves the phone. With one (below), each sentence's frames go there to be read; and if you agreed when it asked at the start, your sentences are kept for training (see below).
It knows 51 words, in one sentence shape:
command · colour · preposition · letter · digit · adverb bin lay place set · blue green red white · at by in with · a–z (no w) · zero–nine · again now please soon
Tap the sentence for another one. To place the oval by hand, drag it to your mouth (pinch to resize); it then stays where you put it until you turn this back on:
Any words — with a server
The page looks for thelip's own server when it opens; when it answers, every sentence's frames go there and come back read by an open-vocabulary model in a few seconds — English (LRS3, 19% word error rate in the lab), Spanish, French, Portuguese, Mandarin. Languages with no lip-reading model yet, Hebrew among them, read the phrases you taught it (below). Several people in the picture are read each on their own, even when they speak at once (their voices would mix, their lips do not): a frame around each face, red while it speaks, and what it said under it. You can also run example/thelip-server from the syrup repository on your own computer and put its address here.
Language (with a server; here, the 51 English words)
Your sentences, for training
When a server is connected, thelip asks once, at the start, whether it may keep your sentences for training. If you agree, for each sentence it keeps a small grey video of your mouth (96×96 pixels, 25 frames a second: the mouth alone, not your face, not the picture around it) and what you said, written down by speech recognition from the microphone or from an opened clip's sound; the sound itself is deleted once it is written down. A random id this page keeps for itself goes with them, so that only your own phrases are matched. Under the subtitle you see what was heard against what the lips gave; if speech recognition got it wrong, tap the subtitle and fix it. They train thelip's readers on real phones and real speakers; in a language with no reader yet, Hebrew among them, they are also how it learns your phrases: say one out loud once, and your lips alone read it after that. If you do not agree, nothing is kept and the microphone stays off.
The Lip is the lip reader of the syrup library. Its network is LipNet (Assael, Shillingford, Whiteson, de Freitas 2016), weights from rizkiarm/LipNet (MIT), trained on the GRID corpus (CC BY 4.0): 64 of 66 words on GRID's sample clips. A new face and a phone camera are harder than that; expect misses.