← All work
Research / 2025–presentSTOP 05 OF 08 ON THE VOYAGE

Language in motion

Studying the distance between words and signing.

ROLE

Independent researcher & developer

CONTEXT

Sign-Translator

TOOLS & METHODS

PyTorch · 3D reconstruction · Residual vector quantisation · Autoregressive transformers · Blender

A hand-built island model for Language in motion
THE ISLAND, EXPLAINED

A capture studio and the instruments that keep it honest.

An illustration built for this site in Blender. It explains the idea; it is not data from the project.

THE QUESTION

Making science education more accessible led me to a difficult research question: how can language connect to coordinated movement? Sign-Translator explores that question from studio video to a rendered avatar, while keeping generated motion distinct from linguistically validated American Sign Language. No Deaf signer has yet judged its output, and the project says so wherever a result appears.

32motion codes per frame
24frames per second
2.40 / 4.03generated vs. real signing on the project’s back-translation score
0Deaf raters so far, by the project’s own count
01

More than a sequence of hand poses

Signing coordinates the hands, body, face, gaze, and timing. The project represents all of them on one clock: eleven channels at 24 frames per second, covering body and hand rotations, head motion, 52 face coefficients, gaze, blink and contact. Each value carries a mark for whether it was actually observed, so missing data stays visible instead of being silently filled in.

02

Two cameras, one skeleton

The motion comes from studio video filmed from the front and the side. A pose estimator finds keypoints in each view; the views are synchronised by the motion of the wrists and elbows, the cameras are calibrated from the footage itself, and matching points are triangulated into 3D. One finding from this stage: with both cameras fixed on the signer, focal length cannot be recovered from the geometry alone. Adding a rigid-bone-length term to the calibration brought the focal error from 40–47% to within 1%.

03

A vocabulary of movement

A residual vector quantiser turns each frame into 32 discrete codes: four body parts (body, left hand, right hand, face) refined through eight stages each. Splitting by part preserves distinctions that a single stream of coordinates would blur, and moving from one codebook to the residual design cut reconstruction error from about 19° and 11 cm to about 8° and 2 cm.

04

What did not work, and what replaced it

The first plan was to generate a whole sentence of motion from text in one pass. On the project’s instruments, no end-to-end generator separated from blank motion. The current path is composition: generate each word separately, sample it sixteen times, keep the sample a recogniser reads best, and join the words with predicted timing. The output follows English word order, which is not ASL grammar, and the project states this plainly.

05

Reading the gauge

Generated motion is scored by translating it back to English and comparing with the source sentence, always beside controls: real signing, blank motion, shuffled order and the wrong sentence. On a fresh set of sentences read exactly once, composition scored 2.40, real signing 4.03 and blank motion 2.05. The milestone was 2.55. It was missed, and the result was published without a retry.

INSIDE THE ENGINEERING

Language has to survive the whole pipeline.

  1. 01

    Put the motion on one clock

    Front and side How2Sign video are aligned using landmark motion, then the cameras are calibrated, positions triangulated and joint rotations fitted. Reprojection and rigid-bone checks constrain the reconstruction. These are video-derived estimates, with provenance retained, rather than motion-capture ground truth.

  2. 02

    Predict codes, not coordinates

    An autoregressive transformer predicts the residual codes from text: a causal temporal model over frames, a byte-level text encoder, and a smaller depth model over the eight stages. Early free-running samples were recognised far less often than one-step predictions suggested; trimming, history dropout and masked losses closed most of that gap for isolated words.

  3. 03

    Test what the score is measuring

    An earlier retrieval evaluator scored motion with zeroed rotations surprisingly well: clip duration was leaking the answer. Fixed two-second windows, randomized offsets, and a duration-only control made that shortcut testable. The same suspicion is applied to every later instrument before it is allowed to grade the generator.

The release bar was fixed before any human rating: Deaf raters must understand generated sentences at least 70% of the time and within ten points of real signing. That study has been designed but not run. Until it is, nothing here supports a claim of understandable ASL.

Read the continuous-training experiments
WHAT THIS DOES NOT SHOW

The honest edges.

  • No Deaf signer has evaluated the generated motion. Automatic scores are a proxy, not comprehension.
  • The output follows English word order and does not model ASL grammar.
  • Model weights are not released, and the system is not a deployed translator.
FROM MY PORTFOLIO FILM

From observations to motion.

Watch the full film
Download video

English audio. Captions are available in all ten site languages; translations are machine-assisted.

Read the transcript

00:00 All that curiosity led me to Sign-Translator.

00:03 Lines from two cameras meet at each joint, which gives its position in 3D.

00:08 Because bones don't change length, they took the focal length from over 40% wrong to within 1%.

00:14 The motion is then compressed into codes, like a vocabulary, and a transformer writes them from English.

00:21 Then nearly everything failed.

00:22 The whole sentence generators read no better than standing still.

00:26 One of them passed 95 out of 95 motion checks and still couldn't carry a sentence.

00:32 So now the system generates each word separately and stitches the clips together.

00:37 Words it doesn't know simply vanish.

00:39 To score it, a model reads the motion back into English.

00:43 Blank motion scores 2.05, the generated motion 2.4, and real signing about 4.

00:49 It's a narrow gain, but it's a start.

00:51 Thanks for watching.

An excerpt from my portfolio film: hand landmarks, reconstruction, generated motion, and evaluation limitations. The rendered avatar and automatic scores demonstrate research work in progress; they do not establish sign-language intelligibility.

VISUAL NOTES / FROM THE REPOSITORY

From words
to movement.

Original rendered hand-articulation test from the Sign-Translator avatar rig
THE RENDERING LAYER

A pose is only
the beginning.

A motion model eventually has to drive a body. This original hand-articulation render comes from the repository’s rig checks: a close look at the geometry and joint behavior behind an avatar.

Static rig test. It does not depict a translated sentence.

System diagram separating the runnable synthetic baseline from broader language, motion, and rendering research modules

The system, and its boundaries.

The repository’s architecture map separates the runnable synthetic core from the larger translation design. The newer continuous-motion experiments described above follow a separate training path.

View full-size diagram
Language and motion encoders project paired examples into a shared normalized embedding space, trained with a contrastive objective

Can language and movement meet?

A spatiotemporal graph network encodes joint motion. Language and motion vectors are normalized and aligned with a symmetric contrastive loss: matching pairs should be closer than mismatched ones. Retrieval tests probe that alignment.

View full-size diagram
Further explorationHow conditional diffusion was designed
Conditional diffusion design: noise is progressively denoised into motion under language conditioning

A generative hypothesis.

This diagram documents the conditional-diffusion research design. The continuous experiments exposed a near-static failure mode; the selected continuous decoder instead uses discrete motion codes.

View full-size diagram
Further explorationWhat streaming motion requires
Streaming design separating buffer, speech recognition, planning, motion generation, and rendering stages

The timing is part of the interface.

The streaming contract distinguishes time to the first output from sustained throughput. It also specifies an append-only committed prefix so a later prediction cannot rewrite motion already shown. This is a design contract, not a measured production-latency claim.

View full-size diagram
NEARBY ISLANDS

Where this connects.

CONTEXT & FURTHER READING

Active research prototype. The diagrams below document earlier parts of the research design, and the hand image is a rig test. Neither generated movement nor a rendering check establishes linguistically valid ASL. Independent intelligibility evaluation remains necessary. Studio video: How2Sign (CC BY-NC 4.0).