Plenty of judgement, but not enough evidence

THE PAST FEW WEEKS have seen a high-profile debate about the effectiveness and benefits of a number of important transport policy measures, most specifically the London Ultra Low Emission Zone, and other Clean Air Zones; 20mph urban speed limits; and Low Traffic Neighbourhoods. Alongside this we have had the benefit of detaled analysis of the the Elizabeth Line which provides a best practice case study of monitoring and evaluation with important potential consequences for forecasting the responses to future schemes. 

There is much to reflect upon here for advocates of evidence-based policy. This approach requires two significant fundamentals. One is that a consistent supply of good quality technically defensible evidence needs to exist, and secondly that policy and decision-making frameworks recognise and respect the role of that evidence.

Evidence-based policy has been described as that in which decisions are grounded on, or influenced by, rigorously established credible, relevant and objective evidence. The concept presents a real contrast to policymaking predicated on ideology, opinion, theory, assertions of ‘common sense, or anecdotal or hoped-for outcomes, not to mention that based on moral or ethical absolutes.

The kind of evidence that’s most important these days is that related to the underlying factors which transport both affects, and is affected by, such as global warming and climate change. It is also that which recognises the consequences of individual transport policy initiatives and projects — including those impacts that go beyond the traditional user benefits and externalities, i.e. journey time savings, accessibility, and reduced accidents. Decision-making now is increasingly expected to embrace the social and economic distributive and demographic impacts of policies and projects.

In this context, the landscape of effective and robust decision-making needs more than a simple change from the ‘predict and provide’ thinking, which long served as the guiding basis of transport investment. A straight switch to a ‘decide and provide’ approach runs the risk of legitimising decisions without sufficient discussion of the basis of them, who they should be made by, and a proper analysis of their full potential outcomes.

It is perhaps, then, worth considering what the role of evidence should be in this new decision-making environment — and how close we are to meeting that objective.
For a considerable period of time a core evidential approach for a transport intervention was the forecast ‘need’ for a facility or service — embodied in the concept of predict and provide. Sensibly moving away from such an approach should not, however, mean that no evidence is required for taking appropriate decisions. They should sensibly still be exposed to the logic of whether what has been decided upon can and will actually be achieved within reasonable resource and delivery boundaries. And that there will not be unwelcome and unplanned-for outcomes.

A useful requirement would seem to be the important element of a clear statement of what the outcome is both desired and likely to be, so any proposals can be assessed as to whether they are actually realistic in achieving what is being sought. This means grounding decisions in a good understanding of both what is happening now, and what can reasonably be expected to happen next, with or without the proposed intervention, and this is where a quality evidence-base is required.

Two years ago the Institute for Government undertook an in-depth study looking at how governments use evidence to make transport policy. It reflected upon the fact that The Conservative Party’s 2019 manifesto promised an ambitious ‘transport revolution’, but the history of previous governments shows how they have often failed to deliver their transport promises. The report considered how the UK and comparable countries use evidence in designing transport policy, and offered recommendations for how the UK can make better policy in future.

The authors were Alistair Baldwin and Kelly Shuttleworth. Their report was based on 50 interviews with current and former officials, politicians, transport experts and academics in the UK and elsewhere.

The authors observe that a key issue in deploying evidence based policy is that decision-makers must understand the principles underlying the evidence they use.

They note that transport is a specialised field with often highly technical evidence. In contrast, decision-makers such as new ministers are often “thrown straight in”, with little time for training. While DfT ministers apparently receive an analytical and evidence pack on their second day in office, explaining the departmental approach to analysis and cost benefit analysis, it must be expected that levels of engagement and understanding will vary.

At a local level, it would be interesting to have more information about what kind of introductory and analytical briefing is provided to elected members taking on political responsibilities for transport. Indeed, there might be a role for the professional bodies in presenting such information, and even training on individual topics of current interest. The objective is surely to help ensure decision-makers are aware of the principles underlying the evidence they assess, and enhance their ability to understand its strengths and weaknesses and allow them to more effectively challenge evidence they receive and become more intelligent consumers of transport evidence and the various ways of balancing matters beyond just value for money.

The way evidence is used in decision making for transport is both complex and often opaque. Those not at the heart of the process can be left guessing as to the weight different forms of evidence and outcomes have been given in decisions, and business cases are rarely available to outside observers. In cost-benefit analysis, trade-offs are an integral part of decisions so transparency about how they are made is even more important.

It is often complained that UK transport modelling and economic appraisal is largely a complex ‘black box’ with limited ability outside of a narrow group of highly technical professionals to engage and the original forms of evidence receiving only limited external scrutiny and attention.

The bodies that supervise the validation and presentation of the evidence are often, themselves, unclear and though those like the National Audit Office are powerful in their scrutiny function, the breadth of their responsibilities and restricted remit mean that transport is subject to only a limited amount of their attention — particularly the consideration of matters outside the area of value for money.

While the DfT has significant analytical capability, a preponderance of economists in it and the Treasury can mean that other forms of evidence may be neglected in transport decision making – including social research and wider impact evaluation. This means that a genuine outside view should from time to time be invited. Such major reviews of underlying principles by suitably equipped panels are few and far between, but occasionally have undertaken highly-significant work.

The Advisory Committee on Trunk Road Assessment, established under Sir George Leitch and which reported in 1978, is much lauded for its detailed scrutiny of both process and content and arguably had a greater impact for coming at the subject from outside the institutional establishment. Although it became, for a while, a standing committee — hence SACTRA — being established for a particular purpose was perhaps its strength. Something similar is arguably required now to re-calibrate the basis of the post predict and provide era.

There are some obvious and concerning examples of evidence inadequacies in transport these days, coming just at a time when more and more of the debate is focusing on beliefs, convictions and crusades for particular political positions — even, or perhaps particularly — amongst many senior academics.
The current arguments over the ULEZ, speed limits and neighbourhood traffic management have become a political battleground where assertions and beliefs are dominating over clear evidence and thoughtful judgment about costs and benefits. Even data on matters like effectiveness in reducing emissions and enhancing safety are being challenged as flawed with no generally acknowledged authoritative and independent sources.

Unfortunately this is not just a UK problem. As with all countries the Institute for Government examined, evaluation was inconsistent and unevenly applied. Skills and capacity are not consistently present across national or local government and areas of good practice not propagated through the transport system. “This has led to investment and policies that are poorly monitored, leaving decision makers uninformed
about the effectiveness of their actions”, their study concludes.

Meanwhile the agencies who might once have taken a scientific and dispassionate look at observed evidence are long since absent, for example, the Transport Research Laboratory during its time as the ‘go-to’ body for in-depth analysis of technical matters. It was in fact, discussion of work at the Laboratory on the saturation level for car ownership and its consequences for highway provision, and challenges made at public inquiries, that effectively led to the establishment of ACTRA.

Alongside this absence of a body of technical excellence, it is ironic that in an era of supposedly limitless data unlocked by the digital revolution, detailed generally available knowledge of transport and travel patterns and impacts is remarkably poor, beyond the excellent work undertaken by Transport for London. For example, what is being done on the Elizabeth Line, by TfL itself and a study commissioned from Arup, and previously for the Jubilee line extension evaluation.

Even in publicly owned and funded organisations, it is pretty startling how limited the data collected and made publicly available, actually is — for example passenger volume data on National Rail and journey time data too. The failure to collect and analyse ‘before and after’ comparisons for major schemes is a matter of regret in recent years, especially the examples of Thameslink and Great Western main line electrification. To which could be added the commitment of considerable funds to Demand Responsive public transport projects without a clear understanding of their outcomes and cost-effectiveness.

Another contrast with the work of TfL is lack of equivalent data outside the Greater London area for impacts of the Elizabeth line due to the different availability of data from contactless and digital payment systems.

A recent area of debate has been of the so called ‘Transformational’ impacts of major transport schemes, and packages of them. The Elizabeth Line monitoring seems significant in this regard, suggesting greater levels of modal change and trip generation than previously expected.

Such experience may be extremely valuable in evidencing the usage impact of public transport service changes both in justifying enhancements and resisting calls for cuts in service levels post-Covid and to deal with financial pressures that could have significant effects on system use.

Away from passenger transport, criticism of ULEZ has been facilitated by a lack of clear evidence making it easy to accuse its proponents of having a pre-determined agenda, for example an unstated but implicit revenue-generating objective.

This to some extent mirrors what happened with speed cameras, which were quickly criticised as a money raising racket as their actual benefits in terms of accident reduction were little understood.

The distribution of responsibilities for evidence gathering on different kinds of projects is something worth greater discussion. Alongside the metropolitan transport authorities outside London, the emergence of new Sub-national transport bodies may offer an opportunity to match some of the expertise of TfL, though they will still be finding their feet on data collection capability and capacity, and indeed appropriate resource levels to generate good quality flows of insight and capture the lessons of specific projects. There may be benefits from a clearer framework for overall evidence sharing and comparison as only the largest authorities are probably able to have the necessary capability in an era of funding cuts for core resources within local authorities.

The Institute of Government study recommended that the UK government needs to provide a “sense of authoritative guidance” on the evidence many practitioners rely upon though it can never have a monopoly on expertise or experience. Bringing voices from outside government into the policy making process brings not only new perspectives but also new evidence.

Without an independent research institution, the UK is particularly reliant on its transport academic and consultancy sector to provide evidence, support and challenge. There is great benefit from getting a diverse set of voices who engage constructively on examining the best available knowledge, including those who may be most critical of government and the status quo.

In transport, with its tendency towards highly technical evidence, there is a potential for groupthink and engaging only those with conventional specialist knowledge.
The effective use of evidence is fundamental to the successful delivery of UK transport policies. That needs the right institutional set-up, the best people and expertise and the highest quality analysis. The government has avowed key priorities including delivering on net zero and levelling up. To achieve these, getting transport policy-making right will be crucial. It will need not only to respond to the challenges that currently face the transport system but to adapt to those of the future.

There is a role for policy and vision in transport policy, but proposed actions still need to be assessed for their benefits and costs. The evidence equation is fundamental to this. Perhaps the happiest arrangement is one where a clear transport strategy expressed politically is then tested for its effectiveness and value for money with the benefit of the latest insight into how system users are likely to respond to its different elements and service offers, and react to proposed price and regulatory messages.

Policies and infrastructure should also be tested against multiple futures, as has been notably done in New Zealand, to ensure they are more resilient to uncertainty and potential changing economic and social contexts, to ensure the best decisions are made. Transport is more likely to justify continued high levels of public investment with a comprehensive evidence base, and needs to embed a culture of evaluation alongside a robust approach to project appraisal.

Transport decision-makers also need to be more transparent in how they use evidence in their decisions, as doing otherwise feeds into a culture of suspicion that evidence has not been used, or has been misused. There may be no good reason to think this is the case, but in not being open such beliefs can grow.

The UK and its devolved nations continue to be highly centralised in their approaches to transport. In some ways this can be good for the use of evidence. However, empowering more local bodies to take decisions means that the best available evidence should be available for application everywhere, and that requires a clear approach and support to who is producing what and the gaps that need to be filled by additional studies and evaluation. Good quality evidence is an invaluable resource — but it needs to be planned for and well disseminated, and that deserves the right structures and mechanisms.

Peter Stonham is the Editorial Director of TAPAS Network

This article was first published in LTT magazine, LTT875, 5 September July 2023.