Limitations of methods for measuring out-of-pocket and catastrophic private health expenditures

Abstract

The financial burden of out-of-pocket health spending is hampered by inconsistent survey methods, research shows. The study, Limitations of methods for measuring out-of-pocket and catastrophic private health expenditures, found major limitations in the surveys that help produce estimates of global health expenditures, indicating that such estimates are extremely sensitive to survey design. The work was done in collaboration with scientists at Harvard Medical School.

Research findings

Researchers looked at the results from two health expenditure surveys and found significant technical challenges to the use of these surveys, including a lack of standardization in the wording, framing, recall periods, and number of questions. In many countries, when fewer questions were asked, the average estimate for health spending was lower than for countries where additional items were included. Questions with a shorter recall period, for example four weeks, yielded a larger estimate of private health spending than questions with a longer recall period, such as 12 months.

Researchers also noted that the same survey generated vastly different results when administered to different populations (a phenomenon known as “differential item functioning”), suggesting that cultural and contextual factors may influence survey results.

Analytical approach

Researchers analyzed results from two surveys, the World Health Organization’s World Health Survey and the World Bank’s Living Standards Measurement Study, which ask respondents about health expenditures in different ways. Data from the World Health Survey were used to compare estimates of average annual out-of-pocket spending on health care derived from a single-item and from an eight-item measure. This was done by calculating the ratio of the average obtained from the single item measure to that obtained from the eight-item measure. Estimates of catastrophic spending from the two measures were also compared. Data from the Living Standards Measurement Study from three countries (Bulgaria, Jamaica, and Nepal) with different recall periods and varying numbers of items in different modules were used to compare estimates of average annual out-of-pocket spending derived using various methods.

Research objective

This study investigates the effect of survey design, specifically the number of items and recall period, on estimates of household out-of-pocket and catastrophic expenditure on health. It is part of ongoing work by IHME to analyze health financing by studying development assistance for health, government health spending, private health expenditures, and the costs of services, such as hospitals and community health clinics. IHME researchers are dedicated to advancing the science of health measurement and evaluation by harnessing the value of existing data.

Recommendations for future work

Establishing a method to generate valid, reliable, and comparable information on national and international resource inputs for health is critical for developing health policies, managing program implementation, and evaluating efficiency and performance of health systems in developing countries. To accomplish this, progress is needed in three areas. First, new instruments that are less sensitive to the local cultural context and survey design are needed. Considerable effort and innovation are needed to create effective validation environments where new instruments can be developed, tested, and modified. Second, any effective new instruments for collecting information on out-of-pocket spending would need to be broadly adopted. Third, nonsurvey methods may also be helpful in tracking national out-of-pocket spending and other private spending on health (e.g., by non-governmental organizations or private enterprise). Further work on strengthening the basis for tracking catastrophic and impoverishing health payments is urgently needed.