How to Screen Data Scientists: 9 Questions and What to Listen For Screening Interview Template

Pick the model you built that had the most impact. What decision or process changed because of it, and is it still running today?

What this assesses: The last clause does most of the work. Strong answers name the decision that changed, name who owns it, and describe what happened after launch: a retraining cadence, a threshold that got tuned twice, an owner who inherited it, or an honest account of why it was eventually turned off. Weak answers stop at model performance, quoting an AUC or an F1 with no downstream consequence attached. Treat clean unattributed impact numbers with suspicion, especially a round percentage lift with no baseline and nobody named as the person who measured it. A candidate who does not know whether their model is still in production usually handed it off at the notebook and moved on.

Tell me about something you built that nobody ended up using. Why did that happen?

What this assesses: Everyone with real tenure has one of these, so an answer of none is either inexperience or editing. Strong answers diagnose the cause specifically: the request came from someone who could not act on it, the output did not fit an existing workflow, the latency made it useless at the moment of decision, or the team it was built for got reorganized mid-project. The best answers name the habit they changed afterward, usually confirming the decision and the consumer before writing code. Weak answers blame stakeholders for not understanding the work, which tells you how the next dead project will get explained.

Describe a model that looked strong in offline evaluation and worse in production. How did you find out, and what was actually wrong?

What this assesses: The detection half matters as much as the diagnosis. Strong answers say monitoring caught it, or a shadow deployment disagreed with the live system, or a business metric moved the wrong way and they traced it back. Then the cause is specific: a feature computed differently in serving than in training, a label that leaked future information, a random split where the data was temporal, drift after a product change, or a population the training set never contained. Weak answers say the model overfit and stop. Someone who has only ever evaluated offline will describe a cross-validation score rather than an incident.

Where did the training data for your last project come from, and how was the target label actually defined? What was wrong with it?

What this assesses: Ask this and a lot of resumes get shorter. Strong answers trace the data back to the systems that produced it and are precise about the label: churn as no login in 30 days, fraud as a chargeback filed within 90, defect as whatever the inspector on shift marked. They name the flaws they had to live with, including class imbalance, backfilled history, event logging that changed midway, or a definition that shifted inside the training window. Weak answers describe a clean table someone handed them. That candidate has worked on datasets, not on data, and your data is not a dataset.

Tell me about a time you shipped the simpler approach when you could have built something more sophisticated. What made that the right call?

What this assesses: This is a discipline question. Strong answers tie the choice to real constraints: how many labeled examples actually existed, how often it had to run, who was going to maintain it, and how much of the gain the complex version really bought. A logistic regression that ships and stays maintained beats an ensemble nobody can debug at eleven at night. Weak answers give the principle without an example, or quietly answer a different question about a complex model they were proud of. A candidate who has never chosen the simple option will build you something you cannot staff.

Walk me through an experiment you designed. How did you size it, and what would you have done if the result came back flat?

What this assesses: The second half is the real question, because flat is the most common outcome. Strong answers size the test from a minimum effect worth acting on rather than working backward from a p-value, name the unit of randomization, and describe guardrails: a fixed horizon, no peeking, correction when several variants ran at once. On a flat result they ship the simpler option or stop, and they can point to a time they did exactly that. Weak answers describe slicing the data until some segment turned significant. Candidates say this in writing more often than you would expect, and it is useful when they do.

Describe a time your analysis pointed the opposite way from what a stakeholder wanted. What did you do?

What this assesses: You are testing spine and translation at once. Strong answers describe the conversation concretely: what they showed, how they framed uncertainty in terms of the decision rather than in confidence intervals, what they conceded, and what happened next. Sometimes the honest ending is that the stakeholder shipped it anyway and the data turned out right. Sometimes the stakeholder had context the data did not capture. Both are good answers. Weak answers either fold immediately to keep the peace, or describe winning an argument with no account of what the other side was actually worried about.

How do you use AI assistants in your day to day work, and tell me about a time you had to override what one gave you.

What this assesses: A judgment question, not a loyalty test. Strong answers are specific about where the tools carry weight, usually boilerplate, unfamiliar APIs, first-pass exploratory code, and refactoring, then name a concrete override: a test suggested for the wrong data shape, a dataframe operation that silently changed the row count, a plausible method that assumes independent observations when theirs were correlated, an evaluation loop that leaked the test set. The signal is that they read the output closely enough to catch it. Weak answers sit at either extreme, refusing the tools on principle without a reason, or describing a workflow where generated analysis reaches stakeholders unread.

What in your stack are you genuinely strong in, where would you need to ramp up, and what would make you turn this role down?

What this assesses: Calibration beats coverage. Strong answers claim two or three things confidently, mark the edge of that knowledge, and are direct about the gaps: fluent in Python and SQL including query plans, has fine-tuned models but always on someone else's infrastructure, has never owned a pipeline that had to run unattended. The second half surfaces deal breakers while they are still cheap, whether that is a role with no engineering support, work that turns out to be mostly dashboard maintenance, or an onsite requirement they cannot meet. Weak answers list twelve tools at one level and claim nothing would put them off. You will find the real answer at the offer stage instead.