Interviewing Data Scientists Without Getting Lost in the Maths
Data scientists speak a language that many auditors do not. They use terms like cross-validation, regularisation, hyperparameter tuning, and precision-recall trade-offs. An auditor does not need to understand the mathematics to audit effectively. What you need is the ability to ask intelligent questions about governance and trace responsibility. Auditing is about process and control, not technical depth.
Start every data scientist interview with context. Explain that you are auditing the AI management system, not evaluating their technical competence. You are not there to judge their model’s mathematics; you are there to understand how the organisation governs the systems they build. This framing makes the interview collaborative, not adversarial. Data scientists are more candid when they know they are not being judged.
Ask about approval processes first. Questions like these work: “Walk me through how you get approval to train a model. Who decides whether your data is suitable? Who approves the final model before it goes to production? What steps do you take between training and deployment?” These questions are procedural, not technical. The data scientist explains governance, not mathematics. Understanding the approval chain reveals whether oversight exists.
Ask about data provenance. Provenance means origin and history. Questions like these work: “Where does your training data come from? Who prepared it? How was it validated before you used it? Are there known gaps, biases or limitations?” The data scientist does not need to teach you statistics. They need to explain whether they understand their data, where it came from and whether it is fit for purpose. If they cannot answer, that is a finding. Good data scientists can trace data to its source.
Ask about monitoring and feedback loops. Questions like these work: “Once your model is in production, how do you know if it is still performing well? What metrics do you track? Who receives alerts if performance drops? What happens then?” These questions reveal whether the organisation monitors systems after deployment or treats launch as the end of responsibility. Many organisations forget about models once they are live.
Ask about version control and change history. Questions like these work: “If I asked you to explain why your model changed between last month and this month, what evidence would you show me? Do you have a log of what changed? Who approved the change?” These questions reveal whether the organisation can trace responsibility for changes or whether systems evolve without oversight. Version control is essential for audit trails.
Ask about edge cases and failure modes. Questions like these work: “What situations cause your model to perform poorly? What inputs make it fail? How do you handle that in production?” Data scientists usually know these answers. If they have thought carefully about failure, that is good governance. If they have not, that is a risk. Understanding failure modes prevents being blindsided.
If a data scientist uses mathematics you do not understand, ask them to translate. Phrases like “we used L2 regularisation to avoid overfitting” can be rephrased as “we used a technique to prevent the model from memorising the training data.” If they cannot explain it simply, they may not understand it themselves, which is a competence concern. Clear communication is a sign of understanding.
Listen for hedging language. If a data scientist frequently says “I think,” “probably,” “I am not sure,” or “no one has asked me that before,” those are signals of uncertainty. Uncertainty is not automatically a problem, but it suggests gaps in governance or monitoring. Repeated uncertainty across many questions indicates a control gap.
End with a summary. “Based on your answers, here is what I understand about your model’s approval process, data provenance, and monitoring. Have I captured it correctly?” This gives the data scientist a chance to clarify and confirms your understanding. Summary statements prevent misinterpretation.
Take written notes during the interview. Notes allow you to capture details accurately and allow the data scientist to see that you are taking their answers seriously. After the interview, circulate your notes to the interviewee for confirmation. This practice, called “closing the loop,” ensures you have not misunderstood. It also gives data scientists a chance to correct errors or add context you missed.
A final tip: appreciate the data scientist’s time. They are not required to support your audit; they do so as part of their job. Thank them for their answers, acknowledge that they have explained complex concepts clearly, and explain how their answers will help the organisation improve governance. An appreciative auditor is more likely to get honest answers than a critical or suspicious auditor.