The ABCs of SBC

Data Science and SBC

• UNICEF SBC • Season 2 • Episode 6

Share episode

How can behavioural data science improve social and behaviour change (SBC) programs? Where do machine learning and AI genuinely add value? And what can goat markets teach us about human behaviour?


In this episode, we explore the emerging field of behavioural data science: how it reveals patterns, frictions, and hidden levers in datasets that weren’t necessarily collected with human behaviour in mind.


Recorded at the Behavioural Horizons Workshop at UNICEF’s Office of Strategy and Evidence (Innocenti) in Florence, we hear from behavioural scientists, data scientists, and humanitarian practitioners working at the frontier. They share why better data starts with better questions, why context and culture still matter in a world of algorithms, and how behavioural data science can narrow the gap between evidence, decisions, and people’s lived realities. It’s a candid look at the opportunities, risks, and human choices that determine whether behavioural data science can help drive meaningful behaviour change.

You’ll hear from:

  • Benjamin Hickler, UNICEF Office of Strategy and Evidence, Innocenti
  • Luke Montuori, Senior Psychometrician
  •  Rebecca Moreno Jimenez, Innovation Team, UNHCR
  •  Rafael Batista, Princeton University
  •  Chiara Cappellini, Behavioural Science Group, UAE
  •  Alexandra DeFilippo, Sistema Futura
  •  Patrick Forscher, Busara Center for Behavioural Science

Resources:

The views and opinions expressed by the contributors are their own and do not necessarily reflect the views or positions of UNICEF or any entities they represent. The content here is for information purposes only.

The ABCs of SBC is hosted by Qali Id and produced and developed by Helena Ballester Bon in partnership with Common Thread.

Check out UNICEF’s latest publication on Social and Behaviour Change, Hidden in Plain Sight, a celebration of the everyday heroes on the frontlines of public health outbreaks, or the first publication, Why don’t you just behave! For more information about UNICEF SBC, check out the programme guidance.

We care about what you think — you can share your thoughts on the podcast using this feedback form. For all other inquiries, please contact sbc@unicef.org.

Qali:

You are listening to the ABCs of SBC and today things sound a little different. Not a studio, not a Zoom call. This time we're at Behavioral Horizons, a workshop of behavioral scientists, data scientists, humanitarians technologists, and other big thinkers. They met at the UNICEF Office of Strategy and Evidence in Innocenti, in Florence, Italy to ask: what problems can behavioral science and data science help solve? And what solutions from these fields show the most promise for making a real difference to the people and communities UNICEF serves?

Ben:

Ben H: We recognized that a happening in the world of applied behavioral science for international development and in humanitarian contexts. And on the other hand, we had, colleagues and academics doing really interesting stuff in, , data science, in associated fields like machine learning and artificial intelligence. The world is changing rapidly and there are a number of pressing challenges that we need to address, and our tools and our approaches to international development, , need , to catch up

Ben:

And so that was where this focus on behavioral data science was born. To build bridges between the worlds of applied behavioral science and data science in international development.

Qali:

That was Ben Hickler from UNICEF's Behavioral Insights Research and Design Lab, or Bird Lab host and co-organizer of the event. UNICEF is no stranger to behavioral data science, having used the power of algorithms to rapidly understand community needs and perceptions and using machine learning to not just spot patterns and data, but to work out causal inference and what's really driving change. That's the promise of behavioral data science. Combining behavioral science theory with data science methods to understand, predict and influence human behavior in ways that are more reliable, more contextual, and more scalable. And here in Florence we heard how quickly this field is changing and the opportunities and risks that lie ahead.

Luke:

Cultures change, the language we use changes, the technological assumptions underlying how that data's collected change. And all of these things, if not monitored and prepared for and addressed change our ability to see the world.

Rafael:

I, I focus on the data. I think that so much of this is data driven. And if you don't have the data or the data is poor quality, then there's gonna be serious limitations.

Alexandra:

If we focus too much in what we see and what we know, what are we not doing? What are we not taking into account?

Qali:

 That was Psychometrician Luke Montuori, Princeton University's Rafael Batista, and  Sistema Futura's Alexandra De Filippo. These experts and more will explain how we can harness behavioral data science for more accurate measurement, prediction, optimization, and synthesis of evidence. This comes at a critical moment for the field. As overseas development assistant faces unprecedented cuts. Organizations like UNICEF are being asked to do more with less. At the same time, access to digital data is expanding rapidly. More people have phones, more data is available, and governments are digitizing historic data sets. Together, this creates an opportunity to use data more intelligently to maximize impact. But making better use of data isn't just a technical challenge. It starts with a more basic question. What are we actually measuring and why?.

Luke:

L uke: I view measurement structured perception. And psychometrics is concerned with how we can best measure those things that we can't really, , we can't see. Uh, traditional psychometrics is, , concerned with defining constructs, psychological constructs, and then just figuring out how, okay, what, what data can we use to, to give us insights about that? But without understanding the structures of the data, the assumptions that underlie the collection of the data and the reason you're collecting that particular data to begin with, you can't really, , stress test your assumptions, , and then the the inferences that you make on the basis of that data going forward.

Qali:

So measurement for researchers like Luke is an exercise in constantly questioning each step in the data collection process. And not only your own assumptions, but also the tools and instruments you're using. Instruments that are often thought to be a fixed standard. Instead, he argues we need to be constantly validating tools and to be sensitive to shifts as culture, technology, and language evolve..

Luke:

When I started in this , I would see that maybe an instrument had been validated. Right? Okay. So this is now a tool that we use to define this particular psychological construct. And in some ways that's seen as a, a gold stamp, and that means forever that that thing is the way we define that thing. Validity is something that you constantly have to be establishing and reaffirming in every context that you're in, so you constantly need to be revisiting your senses. Am I right about this? Are our assumptions correct? And if you start there, then you can have more faith in what's happening at the end and result.

Qali:

It's a valuable reminder that good results don't start at the end of a project before you trust the outcomes of any analysis. You have to be confident in the tools and assumptions you started with or otherwise risk misrepresenting the very reality we're trying to measure

Luke:

Quality issues are at the heart of validity and reliability. And when those things aren't done correctly, any evaluation that you do is gonna be off the mark. Right? It's essentially that if you are building on sand, then it doesn't matter , how high the building that you've built is. Like it may fall apart at some point because your assumptions were wrong from the beginning.

Qali:

It sounds simple enough. Are we checking our assumptions as much as we should to explore what happens when predictions are tested in the real world? We turn to a crisis and an unexpected behavioral signal.  Rebecca Moreno Jimenez, from UNHCR's innovation team has spent a decade working on prediction AI and computational social science for humanitarian operations. Her models help UNHCR anticipate displacement, who might move when and what they'll need in terms of food, shelter, and infrastructure. But early in her work in Somalia, something wasn't adding up.

Rebecca:

Population movement that it's a behavioral choice. Right. You could be in a country that is war torn and choose not to move because there's elements on a human level, uh, behavioral level that maybe you're a caregiver or maybe you have elders or children that cannot move easily and or because of that same reason you're moving first. Right? So it's a, it's such a nuance it's like you cannot do. Pure predictions without the behavioral element. And in this case, the behavioral element ended up being, uh, what I call the goat story.

Qali:

Yep. You heard that right? The goat story. Early on in her work, the models she'd designed to understand people's movements in Somalia weren't matching reality. Why people were moving seemed obvious at first from the conflict data. The reasons were predictable, but the patterns of movement were not..

Rebecca:

They move because country, a violent conflict of fatalities, like people that are dying either on the streets or other rural areas. And, and there's also conflict that is ethnic and conflict among, uh, I would say religion and groups and extremist groups, right? So in this conflict altogether merged, uh, some people choose to go to certain areas and some people choose to certain others. And at some point some of the models for me were not fitting. And then I talked to one of our, anthropologists that was an expert on Somalians, said, because you have the wrong map. I remember it was an interview. so I was in the border with Somalia in Ethiopia. And so I was like, um, can you please describe what happened, and describe exactly the route. That you took. The woman told a harrowing story of how a local militant group insisted on a bribe that she could not pay. She feared for her life and the safety of her husband and kids And they said, we're gonna return and say no, the next time they're gonna kill my husband. So I was so afraid. Then I took my kids, I took my, husband and I, I sold my goats and I flee to the nearest border here. And I was like, okay, husband, kids, logical. Goats, like, did I hear correctly?

Qali:

Her translator explained that the goats were the family's bank account. Rebecca saw another level of data for her analysis. Where were people selling their livestock?

Rebecca:

In economics, the pricing of a commodity, like a goat drops whenever you're selling, right? Because there's an overflow of the price. That means that I would see a drop in the market price. And it's like, okay, if the market price is dropping, then, it's an early warning sign. It's not the factor like climate or violent conflict that is making them drive displacement, but it's. It's a behavioral pattern that I'm seeing observing and it's actually quantifiable.

Qali:

From there, the food and Agricultural Organization's data on livestock prices was brought into her dataset to add yet another variable to increase the sensitivity of her predictive modeling.

Rebecca:

This is when it's like you cannot do. Pure predictions without the behavioral element. I always tell the teams that are doing these kind of simulation prediction and behavioral data science, ....... find your goats. This is behavioral data science in action. At its best when data meets lived experience and models adapt to reflect the way people actually make decisions. This is something Alexandra DeFillipo of behavioral systems firm systemic futura, has been thinking about how behavioral models capture context.

Alexandra:

This event has made nuances in context in culture. Demand that we think a lot more carefully about how we calibrate models, how we contextualize them and how people behave in response to the same event. So unless we're able to represent that well, in our models, we probably are not, are not getting the right solution. So how do we gather data or an understanding of how people think and behave and calibrate models to that? That's what I've been thinking about.

Qali:

Next, I wanted to understand how behavioral data science could help connect the dots between all the evidence we already have. That question leads us to Chiara Cappellini, Principle Behavioral Scientists at the Behavioral Science Group in the UAE, who has been exploring how new technology can bring together scattered insights on what works in international development..

Chiara:

Chiara: The one thing that how we synthesize evidence. , To inform decision making using new technology. So we have so much, so many different pieces of puzzles on what works. , And these are distributed in academic literature, in great literature and data sets that are behind, closed doors of, you know, these different mult multilaterals or different nudge unit. And can we kind of act as a coordination mechanism? 'cause we need to know what works to be effective in how we do international development. The vision is. Almost, we have this dashboard where we have key research questions like, how do you boost immunization or how do you, , improve child nutrition? And then there's almost like a self updating, answer on some of the most effective behavioral interventions in low and middle income countries that are being fed from gray literature, data sources, and so on.

Qali:

This vision of a constantly updated knowledge base of what's worked driven by computational power in AI would allow policymakers, researchers, and practitioners to see what's working where and for whom. This is a way to bring decision makers closer to the voices of communities and to create accountability by making evidence visible to everyone who needs it. And for Chiara, this isn't an impersonal evidence machine. It's a way to close the gap between what people actually experience and how decisions are made on their behalf..

Chiara:

Chiara: When we come to a amp up when we talk about participatory solution design. These tools also allow us to analyze and synthesize this data very quickly, which also creates a direct link between the people that are being consulted and the decision makers. And I think also we were talking about some really interesting stuff around if these dashboards that are synthesizing this data are more widely available towards all the decision makers, they can also increase accountability because then there's this, this transparent,. Overview of what people are that can be shared across these circles and therefore there's also increased accountability to respond.

Qali:

If Kiara's work is about bringing evidence together, the next question is how we make the most of it. We spoke to Princeton researcher Raphael Batista, who's been exploring how new insights can be drawn from existing data?

Rafael:

Rafael: My background is I'm a social psychologist, uh, by training. And, and late, like the thing that I've been curious about is how can we. Discover new insights at the intersection of behavioral science and large language models specifically. And so thinking about taking data sets that exist in the world and trying to uncover new behavioral insights, , with these data sets. Often the conversations around optimization is, you might have a set of. A dozen messages that could go out. And then you try to find among those 12 messages, well, for whom does it work best? And then you try to optimize, What I'm suggesting is that we can also optimize the content of those messages to individuals, maybe even at a hyper localized. Level by kind of households even.

Qali:

The ability to customize messages to individual and household needs and motivations is an exciting space to explore. One way he's doing so is by studying text headlines to be more specific..

Rafael:

Rafael: I have one project newspaper headline data, and the where the outcome variable is simply engagement. Do people engage with a certain message, a certain headline that they see online? , And from there we can learn, well, what is it about a headline? , That leads people to engage with this message. Now, this is one example, but the framework that we proposed can be thought of in many other places. , We can think of other cases such as with unicef, we've started to think about, an instance where you want to send out messages, to a population to get them, , to either wash their hands, , or avoid social contact. And what is the underlying patterns that might drive that behavior. And so the thing, the challenge that we have in, in that project is how do we measure that in a way that we can reliably use to then do all of the machine learning stuff that we've been thinking about. The data is often the challenge that we have. But, uh, at a high level, the thing , we're doing is if you have some texts and you have some outcome variable or something that you measure. Then we should be able to extract these patents, these insights.

Qali:

Raphael, like many in the room, is a cautious optimist when it comes to the potential of ai, but there is more. He argues that organizations like UNICEF can do..

Rafael:

Rafael: One thing that that just a behavioral feature of people working on artificial intelligence and machine learning algorithms... we like it when things are easy, and so when we go to train a new model or sort of run some experiments, we look for data that's easily accessible. , UNICEF has access through its partners and also through its own data warehouses of a lot of data. So making some of that easy can also shape the conversation, right? And so if we can surface and make easy to access some of these data sets, I think you can really shape the conversation and the technology that's built in the future. This space, data availability, data access, and data relevance was a through line throughout the meeting, and it's not easy. Patrick Forche, a director at the Bosara Center for Behavioral Science, summed up the challenges this way.

Patrick:

Patrick: The people in things like, , data scarcity or . How to build up an ecosystem that, , is supportive of generating ev relevant evidence. , How to ensure, , proper community oversight over solutions. We also have people. Behavioral science. That's my main background actually. Who might be thinking about, , interventions and how to untangle what process is happening to change a behavior, how to optimize for that, and how to use behavioral science. . In a way that solves, , the problems of the people who want the solutions. And then we have people in more of the data science world who are thinking about algorithms and, , sophisticated tech. Um, especially gener generative ai, AI right now. And , there could be a space for all of those things come together. But in, in my experience so far, it's been,. A challenge to find the right everyone's talking about the same thing and, , headed towards a common set of solutions.

Qali:

Obviously one workshop isn't going to solve everything, but judging by the energy in the room, a conversation started that people actually wanted to keep going. I next turned back to Raphael. Given his extensive work in the field of behavioral science and ai, I wanted to know where he felt the potential of AI was going.

Rafael:

Where I sit is that, this can fundamentally change the world. I think this technology is one that, that is gonna open up a lot of doors, , or it can open up a lot of doors. , But I don't believe that it just will simply because, there is still a lot of human decisions going into how these algorithms are being built, how they're trained, how they're deployed, all of these are human decisions. Really, whether or not we achieve that potential is gonna still be a very human, , function.

Qali:

And that raises a very practical question, where do we start? For many in the room, the answer was clear.

Patrick:

I think the place that makes the most sense for people to come together is, around like generating the right data sets or, thinking about ways to validate data sets that can then be used for other purposes. It's not about necessarily generating sophisticated tech or, , developing a new algorithm, but instead, , figuring out the problems that matter to the people, , who should be benefiting from development efforts. And then maybe just deploying a little bit of resources, to try to solve that problem, , and ensure that is relevant and useful to those decision makers.

Chiara:

The opportunity we're not talking, , enough about is now that we have more and more accessible computational power, , we could almost as behavioral scientists go beyond the framework or the paradigm of nudging to towards more boosting. and coaching people to, to resolve their own biases that are hyper tailored to them and creating sustainable strategies. And everyone could kind of be a bit more of a behavioral scientist themselves.

Luke:

You have people starting to use AI in the development of tools, , and saying, this is, we can do this, it's possible. And others saying, no, you are being irresponsible with data. And then you have other people saying, oh, there are new types of tools actually that we can build, or new ways that we can measure using these tools. I always find myself in that group. I think that's the more interesting place to be.

Qali:

AI clearly brings with it both optimism and caution, potential and power. And this event comes at a moment of experimentation and discovery, A moment to try things, test things, combine disciplines that weren't speaking to each other before. A moment where UNICEF and so many partners in this room are exploring what's possible without losing sight of the people these tools are meant to serve. So the conversation doesn't end here. If anything, it's just beginning.  Thank you to everyone who spoke to us at the Behavioural Horizons event including Luke Montuori, Rafael Batista, Tanya Accone, Patrick Forscher, Rouella Mendonca, Joshua Pate, Chiara Cappellini, Alexandra De Filippo and Rebeca Moreno Jimenez, And of course we are grateful to our hosts Benjamin Hickler and Ukasha Ramli, and the incredible team at  UNICEF's Office of Strategy and Evidence, Innocenti.

Qali:

If you'd like to dive deeper into behavioral data science, have a look at the show notes.

Qali:

Please do share the podcast with someone you know would enjoy your benefit from these conversation. It really helps more people find it.

Qali:

I'm Qali Id,. See you next time.