The Gavi FCE are mixed-methods evaluations, covering the full results framework from inputs to impact. The evaluations collect data and conduct analysis at the national and subnational levels (such as district or regional) where possible. This permits a more distinct examination of factors influencing successful program implementation and a more informed understanding of the factors that moderate the relationship between Gavi support and downstream indicators such as immunization coverage and child mortality. Information on subnational trends will provide better direction for planning and future resource allocation.
Study methods include the following:
Qualitative methods, including process tracking, stakeholder network analysis, and targeted studies
The process evaluation examines the interface between the Vaccine Alliance and countries as Gavi inputs (financing, technical assistance, information, instruments/guidelines, and personal interactions) are applied for, received, and implemented.
This evaluation will examine how Gavi’s process, products, and resources are working at the country level to influence immunization-related outcomes:
- To what extent is the process of providing Gavi Alliance support to countries improving over time? What has improved, has not improved, and why?
- What are the intended and unintended consequences of Gavi Alliance support across different levels of the immunization system and why have these consequences occurred?
- To what extent are the design of Gavi Alliance support and its implementation relevant to the country’s needs and aligned with the country’s priorities and systems?
The process evaluation will involve three different evaluation activities:
- Stakeholder network analysis: How are stakeholders at the local and international levels influencing the implementation of Gavi support?
- Process tracking: How does implementation of Gavi support compare to planned activities?
- Targeted studies including Milestone Event Studies and Contingent Studies: Flexible studies to focus on key topics of high priority to countries.
Resource tracking studies
Identifying how Gavi resources are used and how Gavi support, resources from other donors, and domestic resources interact
The resource tracking (RT) component of this evaluation aims to identify the flow and use of resources (financing, commodities, and technical assistance) for immunization programs. Adapting and applying existing RT tools like the National Health Accounts (NHA) framework and the Public Expenditure Tracking Survey (PETS) methodology, the RT component will investigate research questions of interest to both Gavi and in-country stakeholders. Specifically, it will answer the following research questions:
- What Gavi support (by type of support) is spent on immunization and other related activities, such as health system development?
- What is the financial contribution of other external donors for immunizations and related activities?
- What is the impact of Gavi and other external donor support on the national budget, i.e., domestic resources, for immunization and health systems?
Health facility surveys
Measuring constraints to immunization delivery from both the supply side (such as vaccine cold-chain integrity) and demand side (such as health care seeking), as well as barriers to immunization delivery, to identify whether resources are being directed where they are most needed
In order to examine the extent to which Gavi support has led to stronger health systems, it is essential to collect information on factors that affect a country’s ability to deliver immunizations. These include vaccine cold-chain functionality as well as location and accessibility of health services and availability of health workers.
In each FCE country, data collection is undertaken as part of the Access, Bottlenecks, Costs, and Equity (ABCE) Project, housed at IHME. The ABCE project includes a multi-country facility survey, conducted in facilities within a representative sample of districts determined using a multistage sampling frame. As the ABCE study is a general facility survey, we will incorporate additional questions that capture the frequency, timing, quality, efficiency, costs, and financing of immunization services.
Measuring population-based indicators, including vaccine antibodies, and geographically complementing health facility and resource tracking data to allow for cross-method analysis
Population-based household health surveys are a critical source of information, and – due to underdeveloped health information systems – are considered the gold standard for many of the indicators in this evaluation, including immunization coverage and child mortality. Household health surveys also provide an avenue for assessing and correcting routine health information system data; for example, household survey data can be used to examine and correct for inaccuracies that arise in administrative data estimates of immunization coverage. A key indicator in this evaluation is immunization coverage for the various antigens, including timely vaccine coverage.
The usual approach for measuring immunization coverage in surveys is to use vaccine card documentation and maternal recall when vaccine cards are not present. The median fraction of immunization coverage responses based on maternal recall is about 50%. Literature has shown, however, that maternal recall may have limited accuracy. Furthermore, even when a child has been documented on a vaccine card to have received a particular vaccine, this does not mean the child is effectively immunized. A breakdown in the cold chain, for example, may mean that the vaccine delivered is no longer efficacious.
Given that a core part of the assistance provided by Gavi is to strengthen systems and improve vaccine cold chains, it is critical to know if these investments translate to higher rates of immunization in the population. Low-cost biomarker-based approaches using dried blood spots (DBS) provide a way to answer this question and address the bias associated with maternal recall. If data collection is limited to a subsample, this can help to minimize the cost of data collection; statistical models can then be applied to correct card and maternal recall responses in the full survey sample to estimate population-based effective immunization coverage.
Vaccine effectiveness studies
Measuring true effectiveness of vaccines in the population using novel methods such as overlapping demographic and disease surveillance, nasopharyngeal carriage studies, and case-control studies
Another key aim of the evaluations is to understand the extent of reduction in vaccine-preventable mortality. It is likely, however, that none of the countries chosen will have sufficiently developed vital registration systems with medical certification to ascertain causes of death. As a result, the primary source of information on causes of death comes from verbal autopsy studies. IHME will supplement verbal autopsy data with secondary data from other sources such as hospital records and estimate corrections using previously established methods. Estimates of trends over time in causes of death will build upon the cause of death models that were developed for the Global Burden of Disease 2010 study (GBD 2010). Such data can provide an indication of trends in vaccine-preventable mortality, albeit with large uncertainty, which will be triangulated with the results from our other outcome and impact analyses on all-cause mortality and the vaccine effectiveness studies.
Novel outcome and impact evaluations
Analyzing primary and secondary data using novel statistical methods to estimate trends over time at the subnational level and using difference-in-differences analyses at the subnational level to examine causal attribution
The outcome and impact evaluation analysis involves these elements:
- National- and subnational-level estimation of trends in key indicators by combining all available data;
- Socioeconomic-related inequalities analysis;
- District-level difference-in-differences analysis; and
- Cost-effectiveness analysis