Skip to content
sp.
All projects

Case study · 2025–present

Tinted

Computer vision that gets skin tone right across the full range, then recommends makeup that actually matches.

MediaPipeOpenCVCLIPClaude HaikuFastAPINext.js
01

The problem

Most beauty tech gets skin tone wrong for anyone who isn't light-skinned, because camera white balance and lighting swamp the signal. Tinted corrects for that and classifies tone across the full Monk scale.

02

Decisions, and what they cost

Every architecture is a set of trade-offs. These are the ones I made, the alternatives I rejected, and why.

Classical CV preprocessing in LAB color space

vs. raw RGB into a model

Lighting is the dominant error source, and correcting it deterministically beats hoping a model learns invariance. LAB separates lightness from color so classification works on the right axes.

Monk Skin Tone scale

vs. the older Fitzpatrick scale

Monk was built for inclusive tech, with real coverage of deeper skin tones. That's the exact failure mode this project exists to avoid.

CLIP shade matching with embeddings precomputed at import

vs. a hand-built rules engine, or embedding per request

CLIP matches visual similarity without labelling thousands of products, and precomputing the whole catalogue reduces per-request work to one matrix multiply. There's a Lab-distance fallback if the model fails to load, so the feature degrades instead of dying.

A four-check quality gate that refuses the image

vs. always returning an answer

Blur, over/under-exposure, yaw and pitch are each checked with a specific, actionable error message. For a product whose whole purpose is getting tone right, a clear refusal beats a confidently wrong reading. The pitch estimator is still marked provisional in the code pending calibration across the eval set.

Per-patch outlier rejection with a hard floor on survivors

vs. averaging every sampled patch

17 landmark patches are sampled on flat malar and forehead skin, then discarded if they read as an edge, a specular highlight, or deep shadow, with a 10 percent trimmed mean over what remains. If fewer than five patches survive, the request fails rather than guessing.

Sliding-window rate limits plus a fail-closed daily spend cap

vs. trusting upstream quotas

Per-minute and per-day limits with correct Retry-After headers and bounded memory, on top of a cap that meters real token usage against model pricing and stops at a dollar ceiling. A public demo calling a paid API needs a hard spending floor, not good intentions.

03

Evidence it works

  • 77-test pytest suite across the preprocessing and classification pipeline.

  • Deployed end to end: FastAPI on Render, Next.js 16 and React 19 on Vercel, behind a typed API client.

  • The seven-stage pipeline streams over SSE from a worker thread, so the UI narrates each stage instead of showing a spinner.

  • Classification returns the distance to all ten Monk reference tones, not just the winner, so the UI can show a range with match scores and let the user override.

  • Interface colours are held to measured WCAG AA contrast, and the palette is deliberately near-neutral because a tinted background shifts perceived skin tone and would corrupt the exact judgement the product exists to make.

04

What I'd do differently

I'd add a labeled eval set of diverse faces with per-tone accuracy reporting, the same eval discipline I applied to the research agent.