What can we learn from a ‘null’ study result?

{***Pause/Music***}
{***Noah***}

Coming up on Harvard Chan: This Week in Health…

What can we learn from a negative result?

{***Katherine Semrau Soundbite***}
(There is disappointment that we didn’t have an impact on mortality, but everyone has felt we were successful in that we, one, put the null result out there, and two, that we are continuing to dig into the data to really understand what we can learn. I think the mistake or shame would be if we said, ‘Oh, it’s a null result’ and we didn’t really pay attention anymore.)

Researchers from Ariadne Labs recently ran an ambitious trial to see if a simple checklist could improve childbirth care and prevent deaths in one of India’s poorest states.

The results were mixed: The checklist boosted quality of care, but didn’t move the needle on mortality.

In this week’s episode: An in-depth conversation about why that “null” result can actually teach us a great deal about strategies to improve maternal health—and the way public health research is conducted.

{***Pause/Music***}

{***Noah***}

Hello and welcome to Harvard Chan: This Week in Health. It’s Thursday, May 2018.

I’m Noah Leavitt…Amie Montemurro is off this week.

We begin this week by taking you back to late January, 2017.

A group of researchers from Ariadne Labs are sitting together around a screen waiting to see the results of their BetterBirth study.

The moment represented the culmination of years of work.

The randomized study of 300,000 women in Uttar Pradesh, India was one of the largest maternal health trials ever.

Researchers from Ariadne and colleagues in India wanted to see if following the World Health Organization’s Safe Childbirth Checklist could improve quality of care and improve health outcomes for women and children, particularly focused around the time of labor and delivery.
Both the stakes and hopes were high.

Globally, 300,000 women a year die around the time of childbirth and five million newborns die as stillborn or in their first month of life.

But the results of the study were mixed.

{***Katherine Semrau Soundbite***}
(And of course, we had that unblinding moment where we saw the results for the first time and saw that it was a null result. I will admit that there were tears.)

That’s Katherine Semrau, director of the BetterBirth program at Ariadne Labs and the lead author of the study.

And what those results showed is that while the checklist improved the quality of care during labor and delivery it did not reduce death rates.

It was not what Semrau and other researchers were hoping to see.

But, there was little time to sulk.

A month after that “unblinding” moment they needed to present the results to the Royal College of Obstetricians and Gynecologists congress in South Africa.

{***Katherine Semrau Soundbite***}
(And some colleagues here that work in serious illness reminded me that this is what we call SARAH. So, the process is, pretty much like the process of grief, but reflects this acronym SARAH, which is surprise, anger, rejection, acceptance, and hope. And that is a process we had to go through as part of the team here at Ariadne, as well as with our colleagues, about what is the data telling us.)

And that’s the focus of today’s episode—an in-depth conversation with Katherine Semrau: What can we learn from a study with a null result? What are those next steps after that initial disappointment?

It’s an important question, and one that’s not discussed enough in science and public health.

Just last week, Margaret Hamburg, president of the American Association for the Advancement of Science, spoke about this at the Harvard Chan School while delivering the 167th Cutter Lecture on Preventive Medicine.

She discussed the important role of failure in advancing scientific progress.

{***Margaret Hamburg Soundbite***}
(Failure comes with the scientific process. It’s hard to imagine a world in which we could have great breakthroughs and discoveries without some failure along the way. But our systems are not designed to accept this. Success, not failure is rewarded. And so much so that negative studies are generally not published. Information with great value for future research is not shared.)

And in the case of the BetterBirth study, there’s a lot to learn about strategies to improve maternal health—as well as better ways to conduct public health research.

In many ways the BetterBirth trial is remarkable not just because of its size, but its rigor—and we’ll talk to Semrau about how she and her team were able to organize a study with so many moving parts.

But I began our conversation by asking her to explain what the Safe Childbirth Checklist is—and why researchers believe it is key to improving maternal health.

{***Katherine Semrau Interview***}

KATHERINE SEMRAU: So the WHO’s Safe Childbirth Checklist is focused on, what are the 28 key practices, or behaviors, or available supplies that should be at the ready for all women and newborns, from the time of admission, until the time of discharge at the facility? So it’s focused on four pause points, so at admission, just before the woman pushes to deliver the baby or a cesarian section is completed, within the first hour after delivery, and then at the time of discharge.

Really, the checklist was developed and designed based on the experience of the safe surgical checklist, which we’ve been using here at Ariadne and developed in conjunction with the World Health Organization. And the Safe Childbirth Checklist takes 50 different guidelines around maternal and newborn health and condenses it down to those most critical aspects. So using the methodology from the safe surgery checklist lessons from aviation, we focused on the Safe Childbirth Checklist to be implemented, again, around the time of labor and delivery.

NOAH LEAVITT: And so with this study, what were some of the things that you were looking at specifically? I guess, how are you measuring success with this?

KATHERINE SEMRAU: The BetterBirth trial was a matched pair cluster randomized control trial that was designed to answer the scientific question, or address the hypothesis of, does the Safe Childbirth Checklist, along with peer coaching, improve adherence to practices by birth attendants? And then does that, in turn, result in lower rates of mortality for moms and babies? So we used this implementation across 120 sites.

So we had 60 sites that received the intervention, 60 sites that were control sites. And we measured using direct observation. So an independent nurse was in the health facility watching the care that was being delivered in a subset of the facilities. So we observed just over 5,000 deliveries to see what kind of practices were adhered to.

So were nurses washing their hands before they were conducting a vaginal exam? Were they using oxytocin, which is a drug to prevent hemorrhage, within the first minute postpartum? Were they practicing skin-to-skin to make sure that the baby was warm and had prevented hypothermia, basically?

So looking at these basic practices, we observed the adherence to those measures of care. And then we also followed up all the women and children who were delivered in the 120 facilities to find out what were the health outcomes, first of all, at the facility. And then we followed the families up at their homes to understand how they were doing at seven days after delivery.

NOAH LEAVITT: So what did you find in terms of both adherence and then down the line when you did the followup to look at the health outcomes?

KATHERINE SEMRAU: So we had really interesting results. So first of all, we saw a dramatic improvement in adherence to the basic practices between the intervention sites and the control sites. In intervention sites, about 70% of the practices were adhered to compared to 40% in the control sites. So that seemed like great success. And we were really excited to see that this quality of care could improve.

But we did not find an impact on maternal or perinatal mortality or morbidity. So we really are now unpacking those results to try and understand, where was the connection between adherence to these practices? But yet, we didn’t see an impact on mortality itself.

NOAH LEAVITT: And I guess, given the large size of this trial, I mean, seeing that level of adherence must be– I mean, that alone must be a success, right?

KATHERINE SEMRAU: I think it is a success. I think we can learn a lot from the trial about how to help birth attendants adhere to the highest quality standards. But we also are learning the lessons about, what are the other components of the health system that have to be functional, for both providers and their families, to make sure that we actually have used the impact on the mortality results that we’re looking for?

So using a randomized controlled trial design gives us the ability to look very rigorously through standard measures that was done both in the intervention and the control site and being able to have a large enough sample size that you alluded to. We followed nearly 160,000 mother-infant pairs. So at this kind of scale, we’re able to look more deeply at the data now and really investigate where are the other changes we need to make going forward.

NOAH LEAVITT: I do want to dive into that maybe a little bit later on. But you talk about the rigor of the study. And I guess, for people who maybe aren’t familiar with designing a study with all that goes into it, I mean, can you put that in perspective, the challenge of a study with, you mentioned, 160,000 plus mother-infant pairs, all the followup? And what were some of the challenges you faced? And how did you actually put that into practice?

KATHERINE SEMRAU: So the first part of designing a study in using a randomized controlled trial design is to, first of all, you calculate your sample size. You figure out how many births you need to follow. And in our study, we were testing, can there be a 15% decrease in mortality? So that meant that the sample size had to be quite large.

The challenges of following up 160,000 families in the field come down to logistics. They come down to coordination and cooperation into ensuring that you’re offering families an easy, successful way to be followed. So for example, in some cultures– in my previous work in Zambia, coming into a family’s home within the first month after the birth of a child is not necessarily culturally appropriate, especially if it’s a stranger coming into the home.

My experience in India at this time with the BetterBirth Trial, we were able to use a call center to be able to trace families. And so the growth of technology, the accessibility to phones and smartphones, dumb phones, either way– But really, the accessibility to technology, so that you’re able to trace families, was very, very powerful.

So there’s the tracing of the family. But there’s also the rollout of the intervention itself. When you’re doing it across 60 intervention sites, how do you get fidelity to the protocol? How do you ensure that the coaches, the peer coaches, that are working on promoting the checklist and working with frontline providers– are they all doing the same thing? How do we know?

So really it was about coordination, partnership, ensuring that there were protocols that people were able to rely on, but also having systems in place through the trial that enabled high-quality and high-fidelity to the protocol that we had.

NOAH LEAVITT: So I know some of the interesting figures we didn’t like looking back on the rigor was 99.7% fall-out rate with women and then 98% dead accuracy. So could you explain a little bit in what those numbers mean? And then how does that compare to other studies being done around the world in maternal health, or just maybe public health in general?

KATHERINE SEMRAU: Sure. So the followup rate we are very proud of, of 99% of the women where mother-infant pairs were followed. And honestly, I attribute that success to a couple of things. First of all, the technology and the ability to use phones to follow up families have high rates, high accessibility to cell phones within families or within the communities. And if the family themselves didn’t have a cell phone, the ASHA, who is a community health worker of sorts, would connect us with the family themselves.

So it really was about using the community health systems that were already in place and tapping into those. In my epi training, when I was doing my MPH and doctoral work, many studies, often, you try to get at least 80% followup to feel like you have good representation of the population that you enrolled in those kinds of things. So if you hit that 80% mark, it feels great. And of course, above that is even stronger.

But here, we were able to have such high rates of followup. Maternal and newborn health studies have a real mixed success rate of being able to do a followup. Some of it depends on the kinds of intervention that are being done. Some of it depends on communities, or context in which you’re working, and accessibility. But traditionally, we’ve used home visits in maternal and newborn health to be able to follow up moms and babies, be able to see how they’re doing, capture information about them.

But home visits can be incredibly expensive. And when we were working across this vast expanse of Uttar Pradesh, which has a population of 5 million– excuse me, 5 million births a year, 203 million population. To be able to follow up across the entire state of Uttar Pradesh, we knew that home visits alone were not going to get us to be successful.

We also had the cultural context about who was coming to the home and visiting a family. And safety and security wise, we needed, typically, men were the ones who were doing the home visits. But the questions that we were asking about maternal health and newborn health were more appropriate to be asked by women. And so we ended up having the call center, where we were able to follow these families up. And it was all staffed by women, 38 women who worked in Lucknow and were able to call across the state and do the followup.

So that 99% followup we feel really great about with respect to the accuracy that you were asking about. So earlier, I mentioned you want to make sure that you have high quality, rigorous data. And we had a data quality assurance protocol that was part and parcel of the main trial protocol. And it really focused on audits, feedback to data collectors themselves, and also supervision.

And so we actually had– if one nurse was observing a labor and delivery that was happening in a facility on a routine basis, there was a supervisor who came and watched the exact same delivery, captured the data separately, and then they compared their notes. So we were able to assess how accurate were the assessments that the nurses were doing and the data collectors were doing. So there was that set of accuracy.

We also did validation studies of the call center to make sure that the people that we were calling, we thought we were calling, are the same families that we actually spoke to, and that the outcomes were similar, or the same and consistent. And so that’s where we get that accuracy number is through this data quality assurance protocol and also being able to follow up families at such a high level.

NOAH LEAVITT: And so because you touched on this earlier, because you have such robust data, you can now dig into it and maybe see some of the factors that might be driving the health outcome. So are there any– and you touched on, for example, the health system, that the health outcomes might be in part driven by weaknesses in the health system, for example. So now going forward, what are some of the areas that you’re digging deeper into in terms of maybe things you can tweak, increase the effectiveness of the Safe Childbirth Checklist?

KATHERINE SEMRAU: Great question. So a couple of areas that we’ve really been focused on are looking at the interconnectedness within the health system itself. So the trial was conducted at what’s called a primary health center, a community health center, or a first referral unit. These are really like a frontline health facility in Uttar Pradesh. They have labor and delivery services. These facilities that were enrolled in the trial had at least 1,000 deliveries a year.

So they can’t necessarily handle very, very complicated cases. And most of these places cannot do c-sections themselves. So women need to be referred for a cesarean section. Women need to be referred for more complicated cases. And so one of the areas we’re investigating now and trying to understand is what is the referral pathway?

What were the different modes of transportation that women took to get to the higher level facility? What were the concerns with referral? So that’s one area.

Another area is looking at provider skills, and motivation, and understanding more about the birth attendants themselves. Have they received sufficient training in their schooling? Have they enough continuous medical education that’s underway so that they’re continuing to get supervision and support? Do they have the supplies and the medical equipment necessary at the facility all the time?

So really trying to understand where the points of intervention going forward that we can actually make a dent in maternal and neonatal mortality.

NOAH LEAVITT: Is what you’re finding that the checklist model works best in a health system that’s already strong? Do you think that’s the case that adherence is great, but there needs to be a stronger support system around the professionals, make sure they have the training, et cetera?

KATHERINE SEMRAU: I think that’s part of the answer. Where I think we’re seeing great success with the checklist, in Namibia, they have recently published their work from a district hospital in Gobabis, where they’ve had impact on stillbirth rate with using the Safe Childbirth Checklist. They have a different level of birth attendants. They’re using midwives. It’s a hospital facility. It has c-section capability.

There are other functionalities at that particular site that are different. In August of 2017, we coordinated with the World Health Organization to bring together countries that are implementing the Safe Childbirth Checklist and learned a lot about these different strategies, different locations, different places that are implementing the checklists within a hospital, within a hospital, some within a frontline facility, and still having great success.

So I think it’s about finding the right combination of interventions that can be effective in the local context, upgrading facilities to ensure that they are actually able to provide the quality of care that we are demanding on the system. Because once we promise women, we encourage women, to come to a health facility to deliver, we need to be able to deliver back for them the care that was promised to them. And so that’s where I really see our research going now and our work going forward is, how do we strengthen the whole system? And how do we identify where the gaps are and then how they’re going to be addressed?

NOAH LEAVITT: And so, I know, Uttar Pradesh, I believe, is one of the poorer states in India. So how does that play into the findings in terms of potentially translating, even to other states in India. So is that something you look at going forward, this was a really low resource setting? How might this translate into maybe a state in India that maybe has a better health system?

KATHERINE SEMRAU: So interestingly, in Karnataka state as well as in Rajasthan, which are settings that Jhpiego and other partners have been working, have been implementing the Safe Childbirth Checklist, or similar tools, and have found success was being able to improve the quality of care there. What we’re seeing in Uttar Pradesh is, I think, our data gives us insight into what can help, what can work partially, and then where the areas that need to be strengthened going forward.

And there are lots of groups that are really trying to tackle this. There are partners with the technical support unit with Jhpiego with Population Services International that are really trying to improve the quality of care that’s happening at facilities.

NOAH LEAVITT: I’m guessing a critical part of this is partnering with the health system. So what is their reaction? I mean, is that challenging at all, to come in and say, well, here are these recommendations that we have? How do you go about working with the health systems?

KATHERINE SEMRAU: So I think that’s one of the most important parts of the trial itself was actually, Ariadne Labs, led– it really was the partners on the ground. So co-principal investigator from Community Empowerment Lab in Lucknow, co-principal investigator from the Nehru Medical School in Karnataka, along with Atul and myself helped lead the study.

But we had partnership with the government of Uttar Pradesh, the government of India, the health system there. Population Services International was the implementing partner on the ground that has extensive government relationships. So in order to make a trial like this at this size and scale work, but I think it’s true of any kind of public health intervention, is these relationships with the government. Ensuring that you’re answering a question that is important to the government itself and to the stakeholders on the ground is really critical.

Coming to them with these results, we’ve had mixed reactions across the globe about how people have taken in the results. Some people are very excited to see that we were able to get adherence to practices improve. We were able to see quality of care change. There is disappointment that we didn’t have an impact on mortality. But I think across the board, everybody has felt that we were successful in that, one, we put the null result out there, two, that we are continuing to dig into the data with our partners, with stakeholders on the ground, to really understand what we can learn.

I think the mistake, or the shame, would be if we just said, oh, it was a null result. And we didn’t pay attention anymore. The opportunity is to really learn in advance what we’re doing around maternal and newborn health, particularly in Uttar Pradesh, because this is specific to this particular site.

I think the other thing that has been really exciting for me to see is what’s changed in India over the very brief time that I’ve been working there in that there has been a program that encourages women to come to the facility through payment, like a cash transfer program, conditional cash transfer program. But now that we’re seeing more programs like Lakshya, which has just been initiated by the government of India, that’s really focused of quality of care in the labor and delivery room.

And they’re not looking just to the public sector. They’re looking at the private sector too, because women in India are delivering across a variety of sites and settings. And we need to make sure that all of them have that high-quality care. So I see it as a real focus of the government going forward, which I think is great.

NOAH LEAVITT: And when you talk about quality of care, I guess, is that in itself a change that maybe step one is increasing access to care? And then from there, you can actually focus on improving the quality? So you could talk a little bit about that aspect of maternal health.

KATHERINE SEMRAU: Sure. I think over the years there has been a huge focus on getting women to access facility-based delivery. We’ve done that. We got women into the facility, about 70%. Just over 70% of women worldwide deliver in some kind of health facility. And there was this great expectation if we got access, that we were going to knock mortality down immediately.

And we didn’t see that in some settings. And there was great disappointment about that and concern, because now we have all these women coming into the health system, into a system that, in some places, wasn’t ready for it, wasn’t necessarily prepared for the number of women coming in. But now, it really is the focus of what’s the quality that they’re getting once they’re there.

And making sure that we’re, again, delivering on that promise of, we encourage you as a pregnant woman, and your family, to come and deliver at this health facility. Now we’ve got to make sure that the care that you get is accurate, and right, and fully complements with the care that is required.

NOAH LEAVITT: So given the rigor of the size of this study, I guess what is the potential impact, I think, around public health of doing a study this large? Are there lessons that people, not just working in maternal health, but working elsewhere in public health, can maybe take from what you were able to accomplish with this study?

KATHERINE SEMRAU: First of all, I think the message is that it’s possible. It takes a ton of coordination, and partnership, and you have to have great clarity about the goals and the objectives of what you’re trying to meet. So that’s first of all. But I think that’s true with most projects.

I think the second piece that we found very effective and helpful with running a trial like this was having the standardized protocols out there. But ensuring that there were protocols of the protocols, so having the data quality assurance protocol, making sure that there were tool guides, questionnaire guides. So every questionnaire that was filled out, there was some kind of explanatory document that went along with it that there was support and supervision for all the members of the staff.

I mean, we had a field team of just over 300 people to run this trial. We had a team back here in Boston and partners in Washington DC. They were all part of making this happen. I think some of the other lessons learned around implementing a trial at this scale and size is that there are lots of questions about, is this the right study design? So is the randomized controlled trial the right study design?

I think in epidemiology, there’s some great methods that are coming out with using adaptive design, using stepwise design, these other kinds of methods that I think will be really helpful in the public health space going forward. But we were able to answer a really knotty question, tough question, with this trial design in itself. And it was able to work.

I think the other thing is that getting a lot of feedback from your stakeholders and your partners on the ground about the appropriate methods for followup, how the questions are being asked, the order of the questions, how they are interpreted when they’re translated from English into Hindi and back translated, or into other languages. Being very, very thoughtful about piloting your study and project I think is really important.

We learned a lot about, in the pilot phase of the trial, we initially had doctors coaching nurses. And that failed miserably. So then we changed the model to be about peer coaching that was about nurse-to-nurse coaching. Another example is when we started the trial initially. We had men asking– when the home visits were conducted, men were asking women the questions directly and to their families.

That failed miserably. We only caught that because of the data quality assurance protocol that we had that told us that the answers we were receiving from the women when a man was asking the questions in the home versus what the answers were when the woman was answering the phone by herself and being asked by a female interviewer, two different sets of answers.

So we recognize that really those cultural pieces about how you’re asking the questions, who’s asking the questions, and what the kind of information is that you’re gathering is really important. And that takes time and, again, back to partnership. It’s just so important.

NOAH LEAVITT: So going to that cultural sensitivity about really understanding the people that are involved in the study.

KATHERINE SEMRAU: Absolutely.

NOAH LEAVITT: Do you think that doesn’t happen enough in public health?

KATHERINE SEMRAU: I do. I think sometimes in public health, we go in with a solution that hasn’t been necessarily adapted in an appropriate enough way. Or it’s a solution that’s trying to find the problem. So one of things I think about with a vaccine campaign, for example, vaccines can be amazing drugs and tools. But being clear about the communication pathway, the education pathway for the community, is an appropriate messaging. And method is really important.

Or I think with something like this in maternal and newborn health, so thinking about the connections with health care facilities and postnatal care. In India, they have an extensive ASHA network and Angawadi worker that is part of the public health system. So tapping into all of their lessons learned, I think, is really important so that it’s part and parcel of the system and not built parallel to it. And I think in public health, sometimes, we end up building parallel systems, not intentionally.

NOAH LEAVITT: So just a last question, you talked about, a few minutes ago, how the fact that even sharing this null result, people were kind of surprised in a good way. I’d be interested to know, take me inside your head when you realize that maybe the result wasn’t what you expected. But so how do you shift from, OK, we didn’t get what we expected, but here’s what we can do moving forward? How do you put a positive spin on it, I guess?

KATHERINE SEMRAU: Absolutely. Great question. So I have run three large clinical trials in my career. They’ve all been null results. So I’ve had a little bit of experience with this. But I will tell you as part of the BetterBirth Trial process, we had a very clear pathway about when we were going to unblind the data and when we were going to look at the data. And we had an amazing team here at Ariadne, and with our partners, and co-principal investigators, where we started to investigate the data.

And of course, we had that unblinding moment when we saw the results for the first time and saw that it was a null result. I will admit that there were tears. And part of that is just because we hoped that this was going to have a huge impact. And some colleagues here that work in serious illness reminded me that this is what we call SARAH. So the process is pretty much like the process of grief, but reflects this acronym, SARAH, which is surprise, anger, rejection, acceptance, and hope.

And that is a process we had to go through as part of the team here at Ariadne, as well as with our colleagues, about what is the data telling us? So at first, yes, there’s the anger and the rejection. And we don’t want to think that this is really what the result is. But as we came around the acceptance phase, we were really able to start unpacking and looking more in-depth at the data.

And I think now we’re at the phase where we’re really hopeful and helpful. We think that the data is powerful. We think it tells an important story. It gives us insight into the health system, overall.

But more importantly, what do we need to do for mothers and newborns to ensure that their care is of the highest quality? And so going through that process allowed us the time and space to get to the point where we were able to write the paper, submit it for publication, go through the process of putting a big trial null result out there. And then also work on the messaging for the community and stakeholders so that we were able to be very clear about what we have found, and that the messages weren’t muddled, and that we’re very honest about where we’re seeing improvement, and then where we see there was an impact.

NOAH LEAVITT: Given that SARAH acronym and that H for hope, so what does make you hopeful going forward?

KATHERINE SEMRAU: It makes me hopeful that we can see change. That there are places around the world that have had huge impact on reducing maternal mortality, and neonatal mortality has come down, and it’s been slower. But we are seeing places where this is working. We look at Sri Lanka as an example. We look at other countries that have really dropped their mortality rates.

But it’s not just about the mortality. It’s about the experience of care during labor and delivery. I think I’m also hopeful that there is a community around global maternal and newborn health that is paying attention to this problem.

We’ve seen a rise in the United States of awareness around maternal mortality. It is a great concern in this country, because we have rates that are going up. And we need to do something about it. And that is true in other parts of the world as well is that we have more to do to actually make maternal and newborn health safe.

{***Noah***}

That was our conversation with Katherine Semrau about Ariadne Labs BetterBirth study.

If you want to learn more about the study, we’ll have much more information on our website, hsph.me/thisweekinhealth.

You can also learn more about the BetterBirth program by visiting ariadnelabs.org.

Just a note that we’ll be off next week while we’re producing a special episode for the School’s upcoming convocation on May 23.

We’ll be sharing a diverse range of student stories to celebrate graduation. and we’re really excited to bring that to you in a couple of weeks.