Majority of countries over-report numbers when receiving funds based on performance
December 12, 2008 - A new study by the Institute for Health Metrics and Evaluation (IHME) at the University of Washington reveals troubling gaps between the number of children reported by countries to be immunized and numbers based on independent surveys in countries receiving aid money from the Global Alliance on Vaccines and Immunisations (GAVI) Immunisations Services Support (ISS) program. The study,Tracking progress towards universal childhood immunizations and the impact of global initiatives appears in the December 13th issue of The Lancet.
The study examines the number of children receiving diphtheria, tetanus, and pertussis (DTP3) immunizations in 193 countries from 1986 to 2006. The analysis shows that while there have been continual improvements globally in the proportion of children immunized against DTP3, this has increased only gradually over time and not to the level suggested by countries’ official reports. For example, since the launch of GAVI in 1999, officially reported estimates showed a 9% increase in DTP3 coverage from 81% to 90%, while survey-based estimates only showed a 4.9% increase in global coverage from 69% to 74% over the same period.
"An incentive to over-report progress, either intentionally or unintentionally will always exist with performance-based payments," said Dr. Christopher Murray, professor of global health and director of IHME at the University of Washington, and co-author of the study. "To counteract this problem requires not only independent monitoring, but also a system that is based on rigorous, empirical measurements using the best scientific methods available."
The study also found that the GAVI ISS program, which pays countries US$20 for each additional child that countries report to have immunized, leads to over-reporting in more than half of countries. Of 51 countries receiving ISS funding from GAVI, six countries overestimated the additional number of children immunized by four times; 10 countries overestimated their increase in coverage by more than double; 23 countries overestimated by less than double; and eight countries underestimated their increase in the number of children immunized.
The gap between country-reported data and independent survey data was particularly wide in Democratic Republic of the Congo, Guinea, Liberia, Mali, Niger, and Pakistan. In some of these countries, such as Niger and Mali, officially reported DTP3 coverage was almost 100%, while survey data for these countries showed that immunization coverage was closer to 50%.
"We don’t know exactly why there is such a striking gap between the survey data and the country-reported data when it comes to over reporting - or in some cases under reporting - the number of additional immunizations," said Stephen Lim, co-author of the study. "It could be due to incorrect or incomplete recording in health centers or pressure on health care workers to meet supervisors’ targets for immunizations. What we do know is that there is a clear correlation between when those gaps start to widen and when GAVI started funding these countries."
Overall, in the countries receiving ISS funding from GAVI, countries reported immunizing 13.9 million additional children, while independent surveys showed only 7.4 million additional children were immunized. At a rate of US$20 per additional child immunized, GAVI disbursed performance-based payments of $290 million, while the analysis showed survey-based estimated payments should have been $150 million; a difference of $140 million.
"With the additional resources that have become available to improve global health conditions, it is crucial that we accurately track how those resources are used," Murray said. "Independent, contestable measurement needs to become a condition of funding."