Language in motion
Studying the distance between words and signing.

A capture studio and the instruments that keep it honest.
An illustration built for this site in Blender. It explains the idea; it is not data from the project.
Making science education more accessible led me to a difficult research question: how can language connect to coordinated movement? Sign-Translator explores that question from studio video to a rendered avatar, while keeping generated motion distinct from linguistically validated American Sign Language. No Deaf signer has yet judged its output, and the project says so wherever a result appears.
More than a sequence of hand poses
Signing coordinates the hands, body, face, gaze, and timing. The project represents all of them on one clock: eleven channels at 24 frames per second, covering body and hand rotations, head motion, 52 face coefficients, gaze, blink and contact. Each value carries a mark for whether it was actually observed, so missing data stays visible instead of being silently filled in.
Two cameras, one skeleton
The motion comes from studio video filmed from the front and the side. A pose estimator finds keypoints in each view; the views are synchronised by the motion of the wrists and elbows, the cameras are calibrated from the footage itself, and matching points are triangulated into 3D. One finding from this stage: with both cameras fixed on the signer, focal length cannot be recovered from the geometry alone. Adding a rigid-bone-length term to the calibration brought the focal error from 40–47% to within 1%.
A vocabulary of movement
A residual vector quantiser turns each frame into 32 discrete codes: four body parts (body, left hand, right hand, face) refined through eight stages each. Splitting by part preserves distinctions that a single stream of coordinates would blur, and moving from one codebook to the residual design cut reconstruction error from about 19° and 11 cm to about 8° and 2 cm.
What did not work, and what replaced it
The first plan was to generate a whole sentence of motion from text in one pass. On the project’s instruments, no end-to-end generator separated from blank motion. The current path is composition: generate each word separately, sample it sixteen times, keep the sample a recogniser reads best, and join the words with predicted timing. The output follows English word order, which is not ASL grammar, and the project states this plainly.
Reading the gauge
Generated motion is scored by translating it back to English and comparing with the source sentence, always beside controls: real signing, blank motion, shuffled order and the wrong sentence. On a fresh set of sentences read exactly once, composition scored 2.40, real signing 4.03 and blank motion 2.05. The milestone was 2.55. It was missed, and the result was published without a retry.
Language has to survive the whole pipeline.
- 01
Put the motion on one clock
Front and side How2Sign video are aligned using landmark motion, then the cameras are calibrated, positions triangulated and joint rotations fitted. Reprojection and rigid-bone checks constrain the reconstruction. These are video-derived estimates, with provenance retained, rather than motion-capture ground truth.
- 02
Predict codes, not coordinates
An autoregressive transformer predicts the residual codes from text: a causal temporal model over frames, a byte-level text encoder, and a smaller depth model over the eight stages. Early free-running samples were recognised far less often than one-step predictions suggested; trimming, history dropout and masked losses closed most of that gap for isolated words.
- 03
Test what the score is measuring
An earlier retrieval evaluator scored motion with zeroed rotations surprisingly well: clip duration was leaking the answer. Fixed two-second windows, randomized offsets, and a duration-only control made that shortcut testable. The same suspicion is applied to every later instrument before it is allowed to grade the generator.
The release bar was fixed before any human rating: Deaf raters must understand generated sentences at least 70% of the time and within ten points of real signing. That study has been designed but not run. Until it is, nothing here supports a claim of understandable ASL.
Read the continuous-training experimentsThe honest edges.
- No Deaf signer has evaluated the generated motion. Automatic scores are a proxy, not comprehension.
- The output follows English word order and does not model ASL grammar.
- Model weights are not released, and the system is not a deployed translator.
From observations to motion.
English audio. Captions are available in all ten site languages; translations are machine-assisted.
Read the transcript
00:00 All that curiosity led me to Sign-Translator.
00:03 Lines from two cameras meet at each joint, which gives its position in 3D.
00:08 Because bones don't change length, they took the focal length from over 40% wrong to within 1%.
00:14 The motion is then compressed into codes, like a vocabulary, and a transformer writes them from English.
00:21 Then nearly everything failed.
00:22 The whole sentence generators read no better than standing still.
00:26 One of them passed 95 out of 95 motion checks and still couldn't carry a sentence.
00:32 So now the system generates each word separately and stitches the clips together.
00:37 Words it doesn't know simply vanish.
00:39 To score it, a model reads the motion back into English.
00:43 Blank motion scores 2.05, the generated motion 2.4, and real signing about 4.
00:49 It's a narrow gain, but it's a start.
00:51 Thanks for watching.
Playback could not start. Try the video controls or download the video.
An excerpt from my portfolio film: hand landmarks, reconstruction, generated motion, and evaluation limitations. The rendered avatar and automatic scores demonstrate research work in progress; they do not establish sign-language intelligibility.
From words
to movement.

A pose is only
the beginning.
A motion model eventually has to drive a body. This original hand-articulation render comes from the repository’s rig checks: a close look at the geometry and joint behavior behind an avatar.
Static rig test. It does not depict a translated sentence.
The system, and its boundaries.
The repository’s architecture map separates the runnable synthetic core from the larger translation design. The newer continuous-motion experiments described above follow a separate training path.
View full-size diagramCan language and movement meet?
A spatiotemporal graph network encodes joint motion. Language and motion vectors are normalized and aligned with a symmetric contrastive loss: matching pairs should be closer than mismatched ones. Retrieval tests probe that alignment.
View full-size diagramFurther explorationHow conditional diffusion was designed
A generative hypothesis.
This diagram documents the conditional-diffusion research design. The continuous experiments exposed a near-static failure mode; the selected continuous decoder instead uses discrete motion codes.
View full-size diagramFurther explorationWhat streaming motion requires
The timing is part of the interface.
The streaming contract distinguishes time to the first output from sustained throughput. It also specifies an append-only committed prefix so a later prediction cannot rewrite motion already shown. This is a design contract, not a measured production-latency claim.
View full-size diagramWhere this connects.
Finding a headingBoth drive a physical body within its limits: one rudder there, thirty finger joints here.
A better test for an agentThe same suspicion of graders: check the instrument before it scores anything.
CollegeAdvisorBoth would rather abstain than produce a confident answer the evidence does not support.Active research prototype. The diagrams below document earlier parts of the research design, and the hand image is a rig test. Neither generated movement nor a rendering check establishes linguistically valid ASL. Independent intelligibility evaluation remains necessary. Studio video: How2Sign (CC BY-NC 4.0).