You are browsing the archive for DGU.

UK Departmental Government Spending – Improving the Quality of Reporting

- September 13, 2012 in DGU, Featured, Open Spending, Releases, Spending Stories

Continuing in their mission to make spending data more accessible and comprehensible, the Spending Stories team and the team of Data.Gov.Uk are releasing a reporting tool today that will help journalists and analysts to pick the freshest and best departmental spending data to work with when exploring the UK central government expenditure.

Spending data is juicy for journalists – why does it get neglected?

Many reasons. One key one is that the shelf-life of a spending dataset is pretty short from a journalist’s point of view, if they have to wait 6 months or even a year for spending data they need for a story to be released, then chances are – the sniff of the story they were wanting to write will probably have gone stale. Journalists, campaigners and activists need access to well-structured, machine readable and timely data from national as well as sub-national administrations. At OpenSpending, we’re often contacted by journalists with story ideas, or they approach us with a lead. The stumbling stone for them is either lack of information, or worse data that they can’t use because they are not sure of its completeness. The problem is thus the one of trees falling in a wood: If a transaction is missing from a list – does that mean there was no transaction for that amount on that date, or does it mean that the transaction simply was not reported? These distinctions are important for anyone trying to understand the data – and up to now they have been pretty tricky to answer. As an attempt to make this a little easier, today, we announce the availability of an automatic reporting tool for spending data (available both on and on OpenSpending), the result of a collaboration between and us in order to increase the visibility of the spend data and to increase the ease of browsing the substantial volume of datasets that make up the reporting of Government expenditure in The tool lists departments registered as data publishers on and details how precisely they have followed the HM Treasury reporting guidelines. It will also make the whole of the reported data available for search and analysis both on and on the OpenSpending site. The tool is useful to those both using the data, and those within government in assuring that departments are reporting on time. It helps to check:
  1. Quality of the data (i.e. adherence to HMT reporting guidelines, well-structured data)
  2. Status of reporting (i.e. how complete the reports are or if there is a reporting period missing)

Why was this possible?

Having all of these datasets organised under a single catalogue at Data.Gov.UK  in simple spreadsheet format combined with the team’s work in making the necessary metadata available enabled the OpenSpending team to create an extraction system to be set up to clean the data on a regular basis. The team then cleaned over 6000 column names to add compliance with HMT guidance.

How does it work?

The report generator then highlights in red departments who are registered as a publisher on but have failed to publish any information on their spending, in yellow those who have published data which cannot be interpreted as spending data (e.g. PDF format or not complying with the template provided by HMT) and green those departments whose records have been updated as regularly as demanded as per the publication requirements (latest data must have been published as recently as a month ago). The first stage of this release deals with central departments, who are obliged to report all spending over 25,000 GBP. Subsequent stages to follow soon after will monitor local councils and other governmental bodies, which have different reporting requirements. The interface will be useful both inside and out of government, to ensure transparency regulations are met and to better understand where gaps in data may alter the completeness of the picture offered by government data. Interested in more regular updates from the Spending Stories team? Join the discussion via the OpenSpending mailing list.

Opening up Government: publishes UK all central government spending data over 25k.

- June 16, 2011 in, DGU, Open Spending, OpenSpending

This post is by Friedrich Lindenberg, one of the core developers on the OpenSpending project. He describes some of the hurdles that had to be overcome to get to today’s online release of all UK central departmental spending data over £ 25k and some interesting questions stemming from the data. In November of last year, the UK government announced plans to publish central government spending data for all items with a value of more than £25,000. Seven months on, an impressive amount of this data has been released to the public: lists 557 distinct datasets from every government entity – from the NHS to the MOD. Despite this leap forward, it is still hard to get a general overview of the 3327 spreadsheets that have been made available: Questions remain unanswered: How much did a particular supplier get paid across government departments? Which are the biggest suppliers for all NHS outposts? Which companies are working to put on the London 2012 Olympic games and how much is each of each of them consuming? Interesting names and figures jump out: Who are the ‘Shadow Robot Company Ltd’ and what exactly are they doing with £25,586 of the UK’s money? To help finding answers to this question we set out to collect, clean up and present all central government spending data in OpenSpending.

Processing the Data

Once the data had been published, there was a lot of work to be done to make it useable in Open Spending. Having located all available spending releases in the index, the first step was creating a local cache of all the data and converting it to a common format. Even though government guidelines ask for the data to be published as CSV with a particular set of column headers, we had to correct both file format and column name for most of the available data. In some cases, even the content of the fields e.g. inverted dates (Month/Day vs. Day/Month) had to be corrected manually. Other departments had left out vital information such as the supplier VAT code or the government entity responsible for the spending. We also had to normalize many of the entities involved both companies and government departments. For companies we had the benefit of the excellent reconciliation service offered by, but unfortunately, for government departments and other entities no such service is available yet. As a workaround, a simple Google Document allowed us to map some of the used abbreviations and most blatant misspellings to their correct forms. After performing all these operations on a temporary SQLite database, we were able to generate a consolidated 450MB CSV file for all of the 25k spending with over 1.8m identifiable records as well as a list of error reports both for invalid files and individual records. These results are available on the UK Government 25k spending data package on CKAN and could now be easily loaded into and thence presented through an embeddable JavaScript in a convenient interface on The decision of the UK Government to publish this data represents a huge step towards more participatory governance, greater transparency and accountability in financial governance. Thanks to OpenSpending, government spending in the UK is searchable, categorizable and, most importantly, analysable by anyone interested in public spending. OpenSpending will continue to develop tools to allow ever more insightful analysis of the data and hopefully, many more governments will follow suit in opening up their public expenditure records.