Stewart Kettle | 12 Apr 2024

RCTs (randomised controlled trials) are the gold standard for hypothesis testing: is a new way of running a service or programme more impactful than the existing approach? The dramatic increase in the use of RCTs by policymakers and practitioners over the past decade has been hugely positive.

But what about hypothesis building? How do we identify problems with programmes, and which changes to test? For this we often turn to qualitative research, but while it produces valuable insights, it can be slow to undertake and challenging to conduct at scale.

At Colectiv, we've developed an AI interviewer to help fill this gap. We can use it to conduct semi-structured, qualitative interviews. Each interview is informed by research questions and research topics that we design and refine in advance, just as they would be in a human-conducted interview.

Then the AI magic happens. Because we are using generative AI, our Colectiv interviewer generates a series of new questions in natural conversations. The AI interviewer responds to what each respondent has already said, following up and exploring answers in more depth. Each interview is a unique, dynamic exchange between the interviewer and the respondent, just as it would be in a human-conducted interview.

This is important because the AI interviewer can drill down to much more nuanced insights than a survey. The results of our recent project with UNICEF on the distribution of hygiene kits show the kind of information we can gather from hundreds of people in two days.

"I observed that men thought they had the right to cut in front of women, trying to disrupt the line (Earthquake responder distributing hygiene kits in Turkiye)"

We don't think AI interviews are a silver bullet: they'll have their place amongst other methods to be used when appropriate. But because they can operate at scale, they can be integrated into programme management for system level change. Imagine a world where managers can access the combined wisdom of people using and delivering services almost instantly, enabling them to generate new hypotheses to test with RCTs.