How AI Face Shape Detectors Work: Landmarks, Ratios and Models

Landmark models take an image containing a face and output a fixed list of coordinates, each mapped to one anatomical position: the outer corner of the left eye, the tip of the nose, the deepest point of the chin. The list runs the same length and the same order for everyone. Point 33 means the identical thing on every face the model has ever seen.

That fixed ordering makes the rest possible. Because the index guarantees which anatomical position it refers to, downstream code can compute a distance between two named points without any understanding of faces at all. It is arithmetic on an array. Nothing more.

Training happens by supervised regression. Human annotators mark points across tens of thousands of face images, and the network learns to predict those coordinates directly. The objective is a distance error, measuring how far predicted points sit from human-marked ones, rather than a classification accuracy. So the model holds no concept of face shape whatsoever. It knows only where points go, which is exactly what MediaPipe Face Mesh was built to do at scale.

Face Mesh outputs 468 landmark points from a single RGB image. Each carries three coordinates: x and y in image space, plus a z value giving approximate depth against a reference plane near the middle of the head.

Two stages run in sequence, which is a deliberate efficiency decision rather than an architectural accident.

Stage one: face detection

Detection sweeps the full frame and returns a bounding box plus a handful of rough keypoints: eye centers, nose, ear tragions. This network stays small and fast because the job is easy. Find roughly where a face sits, do not describe it. Those keypoints then rotate and crop the image so the face arrives upright and centered before anything else looks at it.

Alignment strips most in-plane rotation variance out of the problem, so the second network never has to learn tilted heads. On video, detection can be skipped on most frames by reusing the previous frame's positions to predict the next crop, which hands the workload to the regression stage.

Stage two: landmark regression

Regression takes the cropped, aligned face, resizes it to a small fixed resolution, and passes it to the mesh network. The architecture is a convolutional encoder: stacked layers applying learned filters across the image, building from edges and gradients early to composite features such as eye corners and lip boundaries deeper in. Newer variants add attention blocks over the eyes and lips, spending extra resolution where small errors show most.

The output head regresses rather than classifies. It emits that flat vector of 1,404 values, 468 points times three coordinates, reshaped straight into an array. No search, no fitting loop, no iterative refinement at inference time. One forward pass produces the whole mesh.

The z coordinate needs its caveat. Depth here is learned and relative, inferred from monocular cues the network absorbed during training on images paired with synthetic 3D fits. It estimates head pose usefully and flags when a face has turned away, but it is not a metric measurement and should never be treated as one. Raw coordinates, depth included, still cannot classify anything on their own.

Normalization solves a problem raw coordinates cannot. Stand closer to the camera and every coordinate grows. Step left and every x value shifts. The geometry that distinguishes one outline from another has to survive both changes.

Ratios do that, because dividing one distance by another cancels scale entirely. A few representative examples:

• Face length over cheekbone width, computed from the chin point, the mid-hairline point and the two outermost cheek points.
• Jaw width against cheekbone width, which separates tapering outlines from parallel-sided ones.
• Forehead width against the same cheekbone figure, the comparison that distinguishes heart from diamond.
• Jaw angle, taken between the vector running from the mandible corner to the ear point and the vector running from that corner to the chin.

Since a ratio divides a number by another number in the same units, doubling your distance from the lens leaves every value untouched. Translation cancels too, because each distance runs between a pair of points rather than against a fixed origin.

Rotation gets handled separately. Before any division happens, the mesh is aligned to a canonical orientation using the depth coordinate and a rigid transform, so a head tilted ten degrees to the left produces roughly what a level one would. Good, but not unlimited.

Those timings are typical for a mid-range laptop with WebGL available. Regression dominates the budget, which is why detector reuse across video frames matters so much in live camera modes. The dozen ratios it feeds forward are all the classifier ever sees.

Classification takes the ratio vector and scores each candidate shape. The model stays small, since its input is a dozen or so numbers rather than an image, and nothing larger would earn its keep.

The design choice that matters is the output being a distribution rather than a label. Scores across all seven categories sum to one, and the reported result is simply the largest. A face returning oval at 0.78 and oblong at 0.16 is being described accurately: clearly oval, carrying some of the elongation that characterizes oblong.

That spread tells you more than the winning name does. Real faces occupy a continuous range of proportions, and seven categories are bins imposed on that range for convenience. A result of oval 0.41 against heart 0.38 is not a failure. It reports correctly that your outline sits near a boundary, and that styling advice from both categories will partly apply. Treating a 41 percent result as equivalent to a 90 percent one is the commonest misreading of any classifier output.

Scope deserves saying plainly too. This pipeline measures the geometry of a face outline. It infers nothing about age, health, ancestry, mood, or the person behind the photograph. The face shape overview treats the categories as descriptive geometry precisely because descriptive geometry is all the measurement supports. Where that measurement happens matters just as much.

Browser execution covers the entire pipeline described above. Model weights download once as a static file, the same way an image or a stylesheet does, and inference then runs locally against pixels already sitting in the page.

Two technologies make that practical. WebAssembly supplies a compiled, near-native target for the model runtime, so the numerical work runs at speeds a JavaScript interpreter could not reach. WebGL exposes the GPU through the graphics pipeline, letting convolutional layers execute as shader operations across thousands of pixels at once. Together they turn a multi-second inference into a sub-30-millisecond one.

Privacy here follows from architecture, not from a policy promise. No upload step exists in the pipeline because there is no server to upload to. The model sits on your machine, the pixels sit in your machine's memory, and the only thing crossing the network is that one-time weight download. Close the tab and the image data goes with the page. That holds equally for the face shape detector, the symmetry test, and the skin tone analyzer. What the architecture cannot fix is a poor photograph.

Assumptions sit under every stage, and all of them concern your photograph. These five break most often.

Head yaw and pitch

Yaw turns the head left or right; pitch tips it up or down. Both compress the projected outline. A face turned 20 degrees away puts the far cheekbone closer to the nose than it sits in life, which narrows the measured width across the zygomatic arches. Pose correction using depth handles small angles well, then degrades progressively past roughly 15 degrees, because the far side genuinely stops being visible. Pitch does the greater damage, since it distorts the length figure driving the oval-against-oblong and round-against-square decisions.

Wide-angle selfie distortion

Front-facing phone cameras use short focal lengths. At arm's length that produces visible perspective distortion, magnifying whatever sits nearest the lens, typically the nose and forehead, relative to the jaw and ears. The effect runs strong enough to shift a genuine oval reading toward heart. Backing the camera off and switching to the rear lens reduces it sharply, because subject distance drives the effect more than the lens does.

Hair covering the hairline

Mid-hairline anchors the length measurement. A fringe, a low hat, or hair swept forward forces the model to estimate that point from surrounding context instead of observing it. Errors there propagate straight into the vertical ratio, the single most influential number in the whole classification.

Uneven lighting

Strong side lighting draws a hard shadow edge that convolutional layers can read as a contour. Backlighting does the reverse, flattening a face into a silhouette with no internal gradients to work from. Diffuse frontal light, from a window, an overcast sky, or a room with several lamps, yields the steadiest point placement.

Extreme expression

A wide smile lifts and broadens the soft tissue over the cheekbones and changes the visible mandible line. The model tracks those changes correctly, because it tracks the face as it currently is, but the ratios then describe a smiling face rather than a resting one. Neutral, lips together, is the reference state the categories were defined against. Knowing all five tells you how to read what comes back.

Output from this pipeline is a measuring instrument with known precision, not an oracle. Given a well-lit, front-facing, neutral photograph with the hairline visible, it reproduces the geometry a careful hand measurement would find, and it reproduces it identically on every run against the same image. No tape can claim that.

Feed it a backlit shot taken at arm's length with a fringe down and it still returns a name, because classifiers always return something. Confidence is the honest signal there. A tight result on a good photograph is worth acting on. A flat spread on a poor one is worth retaking the photograph for. That distinction matters more than any detail of the network architecture. Further reading on how these measurements feed practical decisions is collected across the blog and on the main site.

An automated facial morphology classifier, then, is a short chain of checkable jobs rather than magic. A landmark model puts ordered points on anatomy without knowing what a face shape is; Face Mesh reaches its 468 across two stages, detection then regression; division turns those coordinates into dimensionless ratios that survive distance and framing; a small classifier spreads confidence across seven categories instead of committing to one name; WebAssembly and WebGL run the lot inside a browser tab, so the pixels stay on your machine; and five photographic conditions, pose, lens, hair, light and expression, set how much the result is worth. Weigh the confidence number, not just the label. Then take a better photograph and run the detector again.