Working with ML: Trust, Questions & Project Value

In plain language: when to lean on a model versus your own judgment, the questions that separate a robust analysis from a fragile one, and how to tell whether an ML project is even worth starting. Written for leaders, not mathematicians. No maths required.

1 — When to trust a model vs human judgment (b25)

Trust the model more when decisions are high-volume, repetitive, and stable, and when errors are cheap and reversible — a mislabelled email or a slightly-off recommendation costs little and is easily undone. Rely on human judgment when decisions are high-stakes, novel, or rare — the situations the model saw little of, where a single mistake is expensive or hard to reverse. In those cases keep a person in charge and use human-in-the-loop review, where the model proposes and a person approves before anything happens. The rule of thumb: match the level of automation to the stakes.

• Automate freely: high volume, repetitive, stable, errors cheap and reversible.
• Human-in-the-loop: meaningful cost of a mistake, but the model still adds speed.
• Human judgment leads: high-stakes, novel, or rare — the model only advises.

2 — Questions a leader should ask a data scientist (b26)

You don't need the maths to pressure-test an analysis — you need the right questions. Ask any data scientist to walk you through these, and listen for confident, specific answers:

• What question are we actually answering? If it can't be stated in one sentence, stop.
• What data did this use — and who's missing from it? Gaps in the data become gaps in the decision.
• What assumptions does it rest on? Every model carries assumptions; name them out loud so you can judge them.
• How was it validated? Look for results on held-out data the model never trained on, not on the data it already saw.
• What's the uncertainty? A number without a range is only half an answer.
• What could make this wrong? A good analyst can already tell you where it breaks.
• How will we monitor it once it's live? The world drifts; yesterday's model quietly goes stale.

Asking these separates a robust result from a fragile one — and does it without a single equation.

3 — Is an ML project worth doing? (b32)

Before anyone builds anything, triage the idea on three axes. Value: does a better prediction change a real decision and move a metric you care about? If a sharper forecast wouldn't change what anyone does, the project has no value. Feasibility: is there enough good data and a clear target to predict? Without the raw material and a well-defined goal, even a valuable idea stalls. Risk: what are the regulatory, ethical, and failure costs if it goes wrong? High risk raises the bar for how good, explainable, and monitored the model must be. The honest truth: many problems don't need ML at all — a simple rule, a dashboard, or a process fix often wins. Start with the decision, not the algorithm.

Interview one-liners

• Automate the cheap, repetitive, reversible decisions; keep human judgment on the high-stakes, rare ones — match automation to the stakes.
• To pressure-test any analysis, ask what question, what data, what assumptions, how it was validated on held-out data, and how it'll be monitored.
• Judge an ML project on value, feasibility, and risk — start with the decision, not the algorithm; many problems don't need ML.

Plain-language explainers on working with ML teams and scoping projects.