MediaPipe's open model quietly speaks ARKit's blendshape language

The setup

Google's MediaPipe is an open-source library several free webcam-based VTuber trackers build their face-tracking on rather than writing their own computer-vision model. Google's own documentation covers the specific task used for this purpose, Face Landmarker, and the project's GitHub repository hosts the code itself under an open licence. This is a living-document entry describing the current task and repository, not a dated release.

What the documents show

Google's documentation states Face Landmarker produces three outputs together: 'a complete face mesh for each detected face, with blendshape scores denoting facial expressions and coordinates for facial landmarks,' specifically '52 blendshape scores, which are coefficients representing different facial expressions,' plus a facial transformation matrix for positioning effects on the detected face. The documentation states guides exist for Android, Python, and web/JavaScript implementations. Separately, the GitHub repository states the project is licensed under 'Apache-2.0' and describes MediaPipe generally as offering 'cross-platform, customizable ML solutions for live and streaming media,' with processing that 'takes place on device' rather than a remote server.

The craft

The 52-count Google documents for its blendshape output matches the coefficient count in Apple's own ARKit documentation, covered separately on this site, which points to a deliberate interoperability choice: a tool built to read Apple's blend-shape naming can generally accept MediaPipe's output as a drop-in webcam-based substitute, without an iPhone. That is what makes MediaPipe attractive as a backend inside other free VTuber trackers - it does the face-detection and blendshape-scoring work so an application developer does not have to train a model from scratch, only build the interface around it.

Keeping the creator in control

Because MediaPipe's documentation states processing happens on-device and the code is openly licensed under Apache-2.0, a creator or small developer can, as a technical matter, verify or modify how their own webcam data is handled rather than trusting a closed pipeline - an editorial point that follows from the stated licence and processing model, not a guarantee about any specific application built on top of it. Any given free tracker using MediaPipe as a backend still makes its own separate choices about what it does with the resulting data once captured.

• Does the specific free tracker built on MediaPipe document its own data handling, separately from MediaPipe's own on-device processing claim?
• Is a 52-coefficient, ARKit-compatible output sufficient for the target rig, or does it need supplementing?
• Which of Android, Python, or web implementation does the creator's actual tool rely on, and does that match Google's current guide?

Google's own documentation and the project's open licence together explain why MediaPipe shows up inside so many unrelated free VTuber tools: it is a documented, reusable, on-device building block, not a single branded application.