Using Different Types of Data to Improve Health in Mesoamerica

Data collected by IHME during its evaluation of the Salud Mesoamerica Initiative (SMI) is driving improvements in the health systems that serve some of the Americas’ poorest people.

While IHME is primarily known for its quantitative data collection and analyses, IHME researchers often combine qualitative data with quantitative data to evaluate health programs and health systems around the world. For instance, IHME is using both quantitative and qualitative data to evaluate SMI, a project focused on improving health care for the poorest 20% of the population in Central America and the state of Chiapas in Mexico.

The SMI approach is novel because of the foundational role of data collection and evaluation assistance from IHME: by collecting data on health infrastructure, population characteristics, and biological indicators, IHME, the ministries of health, and the funders can track country performance in predefined health outcomes. The initiative’s executing agency, the Inter-American Development Bank (IDB), can use the collected data to help ministries of health design and implement changes to their health systems. After judging performance, the countries can adjust their health systems’ practices to address any shortcomings. This process can be repeated many times, thereby driving continuous health system improvement.

The funding system for SMI is also innovative. Both countries and donors contribute funds to implement strategies for improving the coverage and quality of maternal and child health services, and if the countries reach performance targets tailored to their individual contexts, SMI’s funders (which include the Bill & Melinda Gates Foundation, the Carlos Slim Foundation, and the Spanish Agency for International Development Cooperation) will refund half of what the countries have spent on improvements. Countries that consistently reach their performance targets will also be permitted to extend their time in SMI while receiving further funding for health system improvements.

After three rounds of data collection and analysis by IHME, it is already clear that SMI is driving regional- and local-level improvements in the provision of health services to the poor. Starting with the next round of quantitative measurements this year, IHME will be able to assess whether those changes are also resulting in healthier populations.

One of the key regional-level issues already uncovered in the evaluation concerns the effective coverage of vaccines. When IHME tested dried blood spots collected from children in Chiapas in 2012 and 2013, the results showed that an astonishing number of children – 19% – who had received measles vaccinations were not protected by measles antibodies. Their vaccines had failed to make them immune to measles. There were also strong, localized variations in the levels of vaccine protection.

This map of Chiapas, Mexico, shows the proportion of children in each health jurisdiction who were vaccinated against measles but still lacked measles antibodies.

While this discovery was made through quantitative analysis, IHME’s qualitative evaluation of SMI has shown that policymakers and donors alike have embraced the potential for outside evaluation to prompt improvements in how vaccines are transported and administered. Changes like these can boost the vaccines’ effectiveness. “It’s a good thing to have an external evaluation for a project like this,” noted one official in Mexico’s Secretariat of Health. “It shows us how we’re doing – and in an impartial way, an unbiased way. We can use it to make any adjustments needed.”

Another analysis of dried blood spots will be performed in early 2018. That analysis should indicate whether the changes in policy and administration in Chiapas have led to increased effective coverage.

The evaluation of SMI has also produced local-level improvements in service provision and management in individual clinics and health jurisdictions. As part of the initial assessment, the investigators audited the equipment and drugs available at individual health facilities. It turned out that in Chiapas, for instance, over 90% of clinics were either missing elemental child medical care equipment, such as pediatric stethoscopes or functioning scales, or had recently been out of stock of vaccines or basic medications, such as zinc for the treatment of diarrhea. Having this simple information allowed health officials to target the areas and clinics missing those resources. On the day of the second follow-up assessment in health facilities, zero facilities were missing these essential items.

In addition, the qualitative element of IHME’s evaluation revealed changes in the work cultures of local clinics: they had augmented their own assessment and supply-monitoring efforts. Another Mexican Secretariat of Health official reported that “what local working teams did was create a self-assessment process. This allowed them to detect issues and find out in a more focused way where these issues were, whether it was a supply chain of the drugs, or if it was an administrative matter, or a different type of issue.”

Uncovering cultural and procedural changes such as these is one of the key tasks of qualitative research. When this qualitative information is combined with quantitative data it becomes possible to determine which procedures produced the best health results.

Combining quantitative and qualitative data is the key to robust evaluations that result in the clearest possible lessons for how to deliver improvements in health. Combining this approach to data with SMI’s innovative financing structure could provide an important model for future health improvement efforts, in which evaluation, data, and funding incentives are aligned.

SMI is now wrapping up its second round of interventions after the initial assessment, while IHME is getting ready to reassess population health and facility indicators in Chiapas and Central America. This process of intervention and reassessment will continue through SMI’s run, while the performance targets will evolve to place increasing emphasis on the quality of health services provided at SMI facilities. Over time, health officials will use the information uncovered by the evaluation to improve their health systems, leading to locally created, data-driven improvements in health for some of the Americas’ poorest, most underserved people.