The original version of this blog appeared on Development Pathways
PMT is a curse! Sisters, you all know that: inescapable,
debilitating, emotionally draining, a regular cause of extreme irritability!
But I refer here not to Pre-Menstrual Tension, but rather to
a new form of PMT that is sweeping the globe: Proxy Means Testing.
This variant of PMT is a method of selecting poor people to
become beneficiaries of social transfer programmes, currently being advocated
strongly by, among others, my good friends at the World Bank. Proxy means tests
generate a score for each applicant household, based on "fairly easy to
observe characteristics of the household such as the location and quality of
its dwelling, its ownership of durable goods, demographic structure of the
household, and the education and, possibly, the occupations of adult members"
(http://go.worldbank.org/SSMKS9WUT0). The specific indicators used in calculating
this score and their relative weights are derived from statistical analysis
(usually regression or principal components analysis) of data from detailed
household surveys.
PMT is touted as generating "impressive" results;
it claims to be based on "statistically rigorous methods"; in Chile
(where it all began), it exhibits an "excellent record" of targeting;
in Fiji, it has been pushed as being "highly reliable"; in Jamaica,
"leakage errors are less than 3 percent"; etc. As a result, the World
Bank claims that PMT is "objective", that it has fewer disincentive
effects than a true means test, and that it has been "proven to work
particularly well in countries with high levels of informality and where
personal and household income is difficult to verify with any degree of
precision". Overall, the advocates of PMT paint a happy picture of a
scientifically sound, technocratically robust and dispassionately objective
solution to poverty targeting.
But a recent paper, Targeting
the Poorest[i],
suggests that the reality is very different, and cautions policymakers strongly
against the dangers of being steamrollered into the adoption of PMT. It
suggests that the PMT approach is demonstrably deficient in five main areas.
First, the datasets on which PMT is based are not fit for
purpose. The household surveys on which
proxy means tests are modelled are designed to build an aggregate picture of
poverty at national, regional and - less often - district levels; they are not
appropriate for a detailed understanding of poverty in individual households. In
addition to the intrinsic sampling errors, household surveys also suffer from substantial
non-sampling errors, such as (i) lack of clarity on what constitutes a
household, (ii) incomplete coverage, and (iii) the reticence of households to provide
accurate information. These difficulties are further compounded by the fact
that household surveys typically measure only consumption and expenditure, not
income; that household surveys reflect only a single moment in time (often once
every five or ten years), while poverty at household level is highly dynamic;
and that data on asset-ownership (much used within PMT) is a reflection of past
income, not of present income, which would tend to penalise, for example, older
households, who have accumulated assets over a long period but whose current
income is diminishing.
Second,
PMT analysis can be unduly influenced by arbitrary statistical choices. The
paper looks at three specific examples:
a)
How equivalence is calculated. Some analysts do
not apply equivalence scales (ie they treat a child as having the same level of
consumption as an adult), whereas others treat children up to 12 (or 14 or 16)
as being equivalent to 0.5 (or 0.8) of an adult.
b)
How missing variables are interpreted. Inevitably
there are missing data in household surveys; and analysts must decide whether
these non-responses should be imputed or treated as absent. This decision
impacts on the estimated value of the coefficients in the regression, and hence
on the weights used in the PMT score.
c) How
sampling errors are treated. Household surveys only provide estimates based on
a sample; their precision therefore falls within a range, at a given level of
confidence. The paper tested two scenarios, one using the lower bound at the 95
percent confidence interval, and one using the upper bound.
The paper looks at the actual datasets for specific countries
where PMT is used, and demonstrates that each one of these three sets of
assumptions can arbitrarily change the eligibility status of some 10% of
households. Cumulatively, this could mean that the eligibility of over 30% of
households is determined not by its inherent poverty status, but by the statistical
whim of an analyst.
Third, the regressions used in PMT do not provide sufficient
clarity to distinguish between poor households. Clearly the choice of variables
to be included in the PMT influences the outcome: by selecting only a subset of
"fairly easy to observe" variables, the PMT model is inherently less
able to reflect the same degree of variation as the more comprehensive list in
the full consumption measure. Looking at examples in four countries where PMT
is being used, the paper shows that PMT regressions typically only explain
about 50% of the variation in consumption between households. What is worse is
that they are particularly weak at the poorer end of the scale, thus making it
especially difficult to distinguish between the poorest households. In other
words, PMT "performs the weakest at the point where it would be expected
to find the best correlation between assets and consumption". The paper's
analysis of data from four countries (Bangladesh, Rwanda, Sri Lanka and
Indonesia) shows that, depending on programme coverage, targeting error in PMT
programmes is typically between 35% and 43% at a 30% coverage level, between 44%
and 55% when 20% of the population is covered, and a staggering 57% to 71% at a
(more common in reality) 10% coverage level.
Interestingly enough, a separate study in Pakistan, this one by the
World Bank itself (Report No: 47288-PK,
May 8, 2009), reported even higher exclusion errors: of 61% at 20% coverage
and of 88% at 10% coverage. And remember that this is just the theoretical
error of the PMT regressions: it will inevitably be further compounded by
errors connected with the household data, with statistical analysis and with
implementation.
Fourth, PMT faces significant challenges at implementation,
and the paper cites a number of examples of this. There is the problem of
finding the beneficiaries, using either a census or on-demand method, with
examples of enumerators not wanting to enter urban slums because of security
concerns; male enumerators being barred from entering households where only females were
present; evangelical families refusing to take part in the enumeration process;
and nomadic groups, temporary migrants and remote communities being deliberately
excluded. Another issue is the objectivity of enumerators, often with excessive
demands placed on them, inadequate training and insufficient supervision:
examples have been documented of corruption, inadequate time per interview, and
deliberate changing of results where PMT was perceived to be wrong. Then there
is the question of the verifiability of the indicators: assets can be hidden;
ownership is hard to prove or disprove; education, occupation and even age can
be falsified; indeed the fear that proxies may be easily manipulated has led
advocates of PMT to suggest that proxies and weights should be kept secret -
not exactly an advertisement for "transparency"! Other documented
weaknesses include the fact that community verification is rarely effective;
there is evidence in many countries of political interference; recertification
is seldom sufficiently regular to capture the dynamics of poverty; and there is
often no effective appeals mechanism - indeed there is an inherent
irrationality in even introducing an appeals system against what is claimed to
be a fair, objective and transparent system of selection. In this regard, it is
notable that the World Bank’s social protection handbook (For Protection and Promotion: The Design and Implementation of
Effective Safety Nets) clearly states: “Proxy means tests are most
appropriately used where a country has reasonably high administrative capacity”
... which raises the question of why so many countries with relatively weak
administrative systems (e.g. Pakistan, Kenya, Nepal, Fiji, Niger) are being
encouraged to adopt the PMT methodology.
Finally, PMT does not avoid the social, moral, incentive or
political costs of targeting. In terms of social costs, the paper cites
qualitative research in Mexico, Nicaragua and Peru indicating that some
community members ascribe the omission of poor households to luck or God’s will,
describing the PMT methodology as similar to a lottery; the apparent unfairness
of selection leads to feelings of despair, frustration, resentment, anger and
envy, and there is evidence that this has resulted in a breakdown of community
cohesion and even conflict. Morally, there is clearly the issue, as noted by
Sen, that people may be rewarded for being deceitful and punished for being
honest, which may in time corrode the fabric of society. Nor is it clear why
incentive costs should be any less when using PMT: if potential beneficiaries
are aware of the proxies, such as possession of animals or farm implements,
they will be less likely to invest in them. And among the political costs of
PMT is that poverty-targeted programmes, especially when perceived as arbitrary,
tend to alienate the middle classes: evidence suggests that programmes using
PMT never command as significant a share of GDP as, for example, universal programmes
such as child grants and social pensions.
What
do these deficiencies mean in practice? All in all, the paper finds that -
despite all the grandiose claims of its proponents - PMT performs lamentably in
targeting the poor. It concludes that a
striking finding of the analysis was the consistency of the magnitude of errors
across countries, suggesting that such levels of error are to be expected using
PMT methodologies as currently employed. It counsels policymakers to bear in
mind "this combination of theoretical errors means a majority of eligible
poor households may be permanently excluded from social grant benefits as a
result of PMT scoring".
As I said, sisters, in another context, PMT appears to be "debilitating, emotionally draining, a regular cause of extreme irritability" ... but at least we can put a stop to this variant!
[i] AusAID,
Targeting the Poorest: An assessment of
the proxy means test methodology, September 2011
No comments:
Post a Comment