The Defect Classifier With No Feature Vector to Check

I built a manufacturing visual defect-detection model once, classifying part images on an assembly line, with an injected gradual lighting-condition shift starting on day thirty-five, standing in for a camera housing that had started letting in more ambient light over time. The model consumed raw image arrays directly. There was no fixed tabular feature vector anywhere in the pipeline.

The problem

That last detail turned out to matter more than I expected going in. Standard tabular drift monitoring, the kind that runs a per-feature statistical check like Population Stability Index, simply doesn't apply here, because there's no named feature to run it against. So the team had nothing set up to watch for drift on this model at all. The lighting shift gradually pushed real images outside the distribution the model had been trained on, and the defect-escape rate, real defects the classifier failed to flag, climbed slowly and invisibly. Nothing alerted, until a downstream quality audit around day fifty-five finally surfaced an unusual cluster of customer-reported defects that should have been caught on the line. That's a twenty-day blind spot with no earlier signal available anywhere in the system. Nobody was looking away from the problem; nobody had a vector to look at in the first place.

The uncomfortable part of this failure sits in the tooling assumption underneath the effort, not the effort itself. Every drift-monitoring approach I'd used up to that point assumed a named, fixed-length feature vector existed to compare day over day. Unstructured inputs, images especially, don't offer that assumption for free, and pretending they do just means building a check that silently monitors nothing.

The pattern

For unstructured inputs with no fixed feature vector, I measure drift as a shift in the model's own embeddings instead of a shift in named features. Specifically, centroid and covariance movement in the model's penultimate-layer representation, the layer right before the final classification decision, plus a perceptual-hash distribution shift across the daily image batch, both compared against a training-time baseline. Because the embedding space directly reflects what the model actually sees, this shift is detectable almost as soon as the underlying condition starts changing, independent of any labels.

Running that against the same defect-detection scenario, the embedding and perceptual-hash monitor caught an alert within about five days of the true day-thirty-five shift, cutting days-to-detection under ten against the twenty-day blind spot the unmonitored version left open. The defect-escape rate stayed measurably lower, because the earlier alert triggered a recalibration response before escapes had a chance to accumulate. This isn't an idea I invented from scratch, either. Arize AI's writeup on measuring embedding drift confirms centroid-based vector drift as the standard, vendor-documented approach for unstructured-input drift in computer vision and natural language processing, and Fiddler AI ships a dedicated vector-monitoring surface built for exactly this purpose. Two independent confirmations of the same underlying shape, on top of the reasoning that got me there in the first place.

Design considerations

What the embeddings actually represent matters here. A real vision model's embedding space is considerably messier than anything a controlled test can fully capture, full of legitimate variation that has nothing to do with drift: different part orientations, different lighting that's expected rather than anomalous, ordinary manufacturing variance that the model already handles correctly. Tuning the centroid-shift threshold against that kind of legitimate noise is a real calibration problem, not a one-time setup step, and I'd expect to revisit that threshold after watching it run against real production variance for a while, not just at launch.

There's also a real limit to what this catches: it tells you the model's internal representation of the input has shifted. It has nothing to say about why. A lighting change and a genuine change in the part's physical appearance can both move the same centroid in similar ways, and the alert alone can't distinguish an environmental artifact from an actual manufacturing change worth investigating on its own. Someone still has to look at the flagged images and figure out which one it is. The monitor buys you the early warning; the diagnosis is still a separate, human step.

I'd also push back gently on reaching for this pattern by default for every unstructured input. Building and maintaining an embedding-drift monitor is real ongoing work, computing centroids, tracking covariance, maintaining a perceptual-hash baseline, and it only earns its keep when the input space is genuinely unstructured and the cost of a slow defect-escape climb is genuinely expensive. If your unstructured model's failure mode is cheap to catch some other way, a downstream audit that already runs frequently enough, say, this is more infrastructure than the problem needs. I reach for it specifically when a twenty-day blind spot is the actual risk sitting on the table for that particular model.