1. Home
  2. Knowledge graph
  3. avatar_lipsync
Project

avatar_lipsync

Open-source Flutter package that makes a face talk from one photo or a drawn face, lip-synced to streamed speech in real time.

  • #ai
  • #open-source
  • #flutter
  • #dart
  • #audio

avatar_lipsync is an open-source Flutter package I built for real-time lip sync: it makes a face talk from a single photo (or a drawn face), in sync with speech that arrives as a stream. I designed it for voice chatbots on realtime models, and the README has recipes for OpenAI Realtime and Google Gemini Live. It is a personal project I work on at night. This is an early version and it will keep improving.

Photo: @micaela_valente, used with permission. Right: the rig overlay, with the mesh and a live-motion heat map.

How it works

  • Ahead of the sound. Each audio chunk is analysed before it plays: an FFT every 10 ms measures loudness, formants and sibilance.
  • A JALI-style mouth model. The jaw follows loudness, the lips follow articulation, the lips close on m/b/p, and springs keep the motion natural.
  • One photo is enough. MediaPipe Face Landmarker finds 478 face points in the browser, and a Delaunay mesh animates the photo.
  • Alive. It blinks, nods and shifts its gaze.

Grid of the mouth shapes the model produces

Mouth shapes on "Portrait de Blanche Angers, actrice" (c. 1880), Paris Musées, CC0.

Status

Version 0.3.0 on pub.dev with 160/160 pub points, under the verified publisher azanello.com. A drawn face works on every platform and the photo mode on the web. Every tag push publishes to pub.dev from GitHub Actions via OIDC. MIT licensed. Links: live demo, where the photo lip-syncs to your microphone · pub.dev/packages/avatar_lipsync · GitHub.

Related