The Wrong Metric Cost Us 6 Months

When I joined Signzy as Product Owner in January 2024, the brief was clear: the KYC onboarding process was too slow. Banks were losing customers at the identity verification step. The fix, presumably, was to speed up processing.

So that's what I started trying to do. I looked at our backend infrastructure. I looked at how long document processing took. I talked to the engineering team about queue times. I was solving the wrong problem — and I didn't know it yet.

The thing that changed everything was a metric. Specifically, switching from one metric to a different metric. That's it. The same team, the same product, the same infrastructure — but a different number to optimize. And the roadmap looked completely different as a result.

What We Were Measuring

Our north star metric was average KYC processing time: how long, on average, from when a customer submitted documents to when we returned an approval or rejection decision.

This seemed reasonable. The complaint was that KYC was slow. Measuring "time to process" felt like measuring the complaint directly.

But average processing time has a hidden problem: it's a mean. And when you have a bimodal distribution — which we did — a mean lies to you.

The "3 days average" was technically accurate. But it was averaging three very different populations. Users who passed on the first try were through in under 2 hours. Users who failed and tried again were stuck for 36+ hours waiting for a second review. And users who failed and abandoned — who never showed up in our "average processing time" at all, because they never got a result — dragged down our conversion without appearing in our primary metric.

What the Data Actually Said

Three months into the role, I pulled the full transaction logs and mapped the funnel stage by stage. Not "where does the average user end up" — but "where specifically are users dropping off, and at what rate, and at what time in the session?"

The finding stopped me cold.

60%+ of KYC failures happened in the first 10 minutes of the session. Not after backend processing. Not during the review queue. In the first ten minutes — during the document upload phase, before anything reached our backend at all.

These users were uploading low-quality images — wrong lighting, cropped edges, blur — and getting a generic "document submission rejected" email hours later. By the time that email arrived, they were gone. And they never counted in our processing time metric because they never received a decision.

The backend processing was actually fine. The "3 days" average was masking two hours of actual processing time, blended with days of wait time from users trying to figure out why they'd been rejected and what to fix.

The Metric Switch

I changed our north star metric from "average KYC processing time" to "first-attempt success rate": the percentage of users who pass KYC verification on their first submission, without needing to resubmit.

This one change — one sentence in a Confluence doc — changed the entire roadmap. When your north star is "average processing time," you build faster backend infrastructure. When your north star is "first-attempt success rate," you build guided capture UI, real-time quality feedback, and specific rejection error messages.

Those are completely different products. We had been building toward the wrong one for months.

What Happened Next

With the new metric, the prioritization matrix looked completely different. The data science team wanted to retrain the liveness detection model — it was technically compelling and would improve accuracy. But our analysis showed 60% of failures were at document upload (a UX problem), not liveness (a model problem). We would spend 8 weeks fixing 40% of the issue.

Instead, we shipped guided capture UI in 3 weeks: a mobile-first experience with a frame overlay, real-time lighting indicator, and auto-capture when quality threshold was met. Users no longer guessed whether their photo was good — the UI told them before they submitted.

First-attempt success rate improved 28% in the first month. Without touching the model.

The model retrain followed in Phase 2 and delivered the remaining accuracy improvements. But the sequencing was right — we fixed the bigger problem first, with less risk and faster delivery.

Why This Happens

Bad north star metrics are not usually careless choices. They're often the most intuitive choice — they measure something real, they're easy to explain, and they feel connected to the complaint. "KYC is slow" → "measure how long KYC takes." The logic is airtight.

The problem is that the most intuitive metric often measures a symptom rather than the mechanism. "Time to approve" measures the downstream consequence of the failure. "First-attempt success rate" measures the failure itself.

There's also a structural bias: metrics tend to reflect what's easy to instrument. Backend processing time is logged automatically. Document upload failure rates require building specific analytics. The metric you have tends to become the metric you optimize — even when it's not the right one.

A Heuristic I've Kept Since

When a client or stakeholder gives me a metric to optimize, I now ask one question before I start: "What would need to be true for this metric to look good while the underlying product is still broken?"

For "average processing time," the answer was: the metric looks good if you exclude users who abandon. The metric was optimizable without fixing the actual problem. That's a bad metric.

For "first-attempt success rate," the answer is: this metric only looks good if users succeed on their first try. That's the actual goal. That's a good metric.

You can apply this test to almost any metric. Revenue looks good while customer satisfaction falls. DAU looks good while engagement per session collapses. Time-in-app looks good while users are confused and can't find what they need. The test forces you to ask what the metric is hiding.