Performance Assessment

One of the responsibilities of government is to deliver high-quality, relevant services that meet the needs of citizens, communities, businesses, and other organizations. To do so, national governments have started to modernise their service offerings by introducing alternative system. This move has been accompanied by the development of sophisticated schemes to monitor and oversee how well the services are delivered. Given that in most industrialised economies, public sector current expenditure represents between 35 and 50 per cent (in the case of the UK 38.5 per cent) of GDP, and that prosperous states in particular have come under increasing fiscal pressure to cut their spending, the need of policy makers to be able to evaluate what the government gets for its money is evidently clear.

Deprivation may affect authority performance in many ways. Some service functions may be put under particular strain if large sections of the population suffer from low income, unemployment, poor health, or low educational attainment levels. Similarly, in areas with deprivation and large black economies, local tax collection tends to nosedive as citizens are keen to conceal their existence to their council, which ultimately feeds through to the relevant CPA indicators measuring tax collection performance.

Oversight has a long tradition in many countries and refers to scrutiny and steering from some point ‘above’ or ‘outside’ the individuals or organisations scrutinised. Traditionally, it has been effectuated by law courts or elected legislatures, but increasing use has been made of (‘quasi-independent’) reviewers, watchdogs, inspectors, regulators, auditors or monitors that are to some degree detached from executive government and line management.

The 1980s and 1990s have witnessed an increase in oversight and audit activity by governments that has led some authors to herald a ‘new age of inspection’ (Day and Klein 1990) and the advent of the ‘audit society’ (Power 1997).

the explosion of oversight has been far from uniform while the responsibilities and resources allocated to overseers has grown. In the case of the UK, for example, formal arms-length overseers doubled in size and real term resources during the 1980s and 1990s, at a time when UK civil service was cut by more than 30 per cent and local government by about 20 per cent (Hood et al. 1999).

In terms of mechanisms, oversight has shifted its emphasis, away from mere fiscal audits to value for money and performance audits. Since 2002, the UK government has assessed the delivery of public services provided by English local authorities through a regime called the Comprehensive Performance Assessment (CPA). Performance in six service blocks (benefits, social care, environment, libraries and leisure, use of resources, education and housing) is monitored through inspections and audits in order to determine if central governmental grant (of currently £120bn per annum (i.e., almost a quarter of UK public expenditure) is money spent effectively.

Andrews et al. (2005) investigated the extent to which success or failure in service provision is attributable to circumstances that are beyond the control of local managers and politicians. The explanatory variables used were: quantity of service need; age diversity; ethnic diversity; social class diversity; discretionary resources; lone parent households; population change; population; population density; and political disposition. The authors found that the ten constraint variables collectively explain around 35 percent of the inter-authority differences in service performance and 28 per cent of the differences in the ability to improve score. They concluded that these are “satisfactory levels of statistical explanation” (p. 650). More specifically, they found that higher ethnic and social class diversity appear to place additional burdens on service providers and thereby result in lower performance; that authorities with a high percentage of single parent households, which represented the authors’ proxy for measuring deprivation, found it more difficult to climb the CPA ladder (p. 651); that large authorities found it easier to achieve good CPA results; and that no differences exist between the four types of authorities. The authors then concluded that “the CPA process is flawed by the failure to take account of circumstances beyond the control of local policy makers.

We agree with Andrews’s conclusion that there appears to be an impact of deprivation on authority performance that needs exploring, and we reject the Audit Commission’s initial claim that no significant correlation can be detected. We also concur with Palmer and Kenway’s conclusion that in order to test this hypothesis in a statistical model, performance indicators are the wrong choice for the dependent variable because of their limited bearing on final CPA ratings. The selection of variables and data for the analysis, as explanatory variables, was a number of variables that controlled for external influences (e.g. political, economic, social and environmental).

CPA scores are more meaningful because, similar to the IMD 2004, they show the amount of variation across authorities. The conversion into percentages (of the maximum score possible), in turn, is required because 32 of the 148 authorities (viz., the ‘shire’ counties, which have ‘shire districts’ below them) are only assessed in four of the six service blocks and have therefore lower minimum and maximum scores (an approach already employed by Andrews et al 2005, p. 649).

Irrespective of the policy implications, we addressed the statistical issue by constructing a new deprivation domain for education that measures education deprivation of adults only. The new domain is based on indicators measuring the proportion of those aged under 21 not entering Higher Education (1999-1002), the secondary school absence rate (2001-2002), and the proportion of young people not staying in school level education above the age of 16 (2001). In order to test our choice, we later compared the statistical model using the modified domain with the model using the original domain and found that the resultant panel data estimates of all variables had the same sign, and the differences in their magnitude were so minor that both variables could be used interchangeably without affecting the resultant tables.
To conclude the explanation for our choice, and modification, of variables and data, the following list gives an overview of all explanatory and dependent variables used:
‐ Quantity of service need 2001 (measured in logarithms)
‐ Age diversity 2001 (log)
‐ Ethnic diversity 2001 (log)
‐ Social class diversity 2001 (log)
‐ Discretionary expenditure 2002, 2003, 2004 (log)
‐ Population size 2002, 2003, 2004 (log)
‐ Population Density 2002 (log)
‐ Overall Index of Multi Deprivation scores 2004 (log)
‐ Education, Skills and Training Deprivation 2004 (log) or Adults Education Deprivation 2004(log)
‐ Barriers to Housing and Services Deprivation 2004 (log)
‐ Crime Deprivation 2004 (log)

‐ Living Environment Deprivation 2004 (log)
‐ Income Deprivation 2004 (log)

‐ Type of Local Authority:
o County Councils,
o Inner London Boroughs,
o Metropolitan Districts,
o Outer London Boroughs, and
o Unitary Authorities.
‐ Political Control 2001, 2002, 2003:
o Labour,
o Conservative,
o Liberal,
o Independent, and
o No overall control

The comparative government literature shows that nations with proportional representation and (typically) coalition government have higher welfare spending, but worse fiscal discipline, than nations with plurality electoral rules and (typically) single-party government (Persson and Tabellini 2005, pp. 270-3).

During the elite interviews we conducted with auditors, auditees, and other stakeholders, repeated mention was made of the need to carry out comparative analyses between “like cases” of authorities. We followed this advice and observed interesting differences with regard to the effect of some of the explanatory variables, as evidenced by the data shown in table 2. For instance, quantity of service need has a negative effect on the CPA score in county councils, but a positive effect in the remaining types of authorities. Similarly, the effect of age diversity on CPA scores is positive when all authorities are grouped together, but negative in all types of authorities except inner London Boroughs.


the higher the level of deprivation, the lower the CPA performance score, a conclusion that contradicts, to a different extent, some earlier studies (Audit Commission 2003b, Palmer and Kenway 2004) but confirms others (Andrews 2004; Andrews et al. 2005).

This is not the end of the story, however, as this first high-level overview is only the beginning, and more detailed insights can be gained from a second stage, during which the different domains of deprivation are mapped onto the different types of authorities. This more extensive analysis is best approached with visual help.

The picture does not change much for the next set of deprivation domains to the right of the first, which shows that, if analysed separately, the 46 Unitary Authorities are relatively homogeneous and equally deprived across the seven domains. What is more, they do not deviate much from the national average.

However, divergences of some significance emerge within the third group comprising the 34 County Councils. As the first column on the left of this group indicates, they are the least deprived authorities in the country when measured across all seven deprivation domains through the composite IMD 2004. Yet, when split up into the individual domains, upward deviations surface for income deprivation and for barriers to housing and services, the latter of which is due presumably to the long distances prevalent in rural areas to reach the nearest shop, post office or hospital.

The metropolitan districts and, more strikingly, the inner and outer London boroughs display very drastic deviations between the individual deprivation domains. For instance, Inner London Boroughs are on average more deprived than the rest of the authorities in the deprivation domains of income and crime, whereas they are on average less deprived in barriers to housing and services (presumably for reasons to do with relatively good proximity in metropolitan areas) and adult education.

Deprivation in the domain of education has a negative effect on the overall CPA scores and in all CPA service blocks, except for social care (children) and libraries and leisure. In these two latter CPA service blocks, education has a positive but insignificant effect. Deprivation in the domain of barriers to housing and services has a significant negative effect in the CPA service blocks of social care (adults), environment, use of resources and benefits. Crime has a consistent negative effect on the overall CPA score and all its service blocks, except benefits where crime has a positive but insignificant effect on CPA. Similarly, deprivation in the domain of living environment has a negative effect on overall CPA and all its service blocks, except social care (children and adults) and environment. Only in these two latter CPA blocks is it that deprivation in living environment seems to have a positive and significant effect. Income deprivation has a negative and significant effect on CPA in the service blocks of education and social care (children), a positive and significant effect in the service block of housing, and a positive and not significant effect on the rest of service blocks.


Source:
Prof Iain McLean , The limits of performance assessments of public bodies: the case of deprivation as an environmental constraint on English local government, Nuffield College, University of Oxford, Public Services Programme,Nov 2006