You are browsing the archive for Neil Ashton.

How to create a budget data package

- October 15, 2014 in 特稿

This tutorial will show you how to create a budget data package from a (relatively clean) spreadsheet dataset by walking you through the process of converting the Armenian budget from the Open Budgets Portal. ## Getting started The Armenia BOOST government expenditure database contains planned, adjusted, and executed expenditures covering the years 2006 to 2012. […]

How to create a budget data package

- October 15, 2014 in Technical, tutorials

This tutorial will show you how to create a budget data package from a (relatively clean) spreadsheet dataset by walking you through the process of converting the Armenian budget from the Open Budgets Portal. Getting started The Armenia BOOST government expenditure database contains planned, adjusted, and executed expenditures covering the years 2006 to 2012. It […]

How to create a budget data package

- October 15, 2014 in Technical, tutorials

This tutorial will show you how to create a budget data package from a (relatively clean) spreadsheet dataset by walking you through the process of converting the Armenian budget from the Open Budgets Portal. Getting started The Armenia BOOST government expenditure database contains planned, adjusted, and executed expenditures covering the years 2006 to 2012. It […]

Deadly Environment

- April 22, 2014 in Data for CSOs

Deadly Environment: impunity

Global Witness‘s new report Deadly Environment documents the past decade’s shocking rise in killings of people defending environmental and land rights. As competition for natural resources intensifies, more and more ordinary people—particularly in indigenous communities—are being murdered for resisting land grabs, expulsion from their homes, and other abuses. These crimes are going almost entirely unpunished. School of Data worked with the Global Witness team to develop the interactive graphics that accompany the minisite of the report. We wanted to highlight the overall acceleration of the killings, their geographical distribution, and the horrifying extent of their impunity. Conscious of the moral gravity of the project, we also sought to bring the human qualities of the data to the surface. The result is a series of graphics developed with D3.js and presented sequentially using Owl Carousel. In the graphics that highlighted individual killings, a mouseover display of the case’s details was created using the d3-tip plugin. Geospatial data for the map graphic was taken from Natural Earth. These interactive graphics helped Global Witness’s report reach a wider audience, particularly on Twitter, where promotional tweets including screencaps of the graphics were very widely shared.

flattr this!

Mapping the Open Spending Data Community

- January 6, 2014 in Featured, Open Spending

Mapping the Open Spending Data Community

We’re pleased to announce the official release of “Mapping the Open Spending Data Community” by Anders Pedersen and Lucy Chambers, an in-depth look at how citizens, journalists, and civil society organisations around the world are using data on government finances to further their civic missions. The investigation began in 2012 with three goals:
  • To identify Civil Society organisations (CSOs) around the world who are interested in working with government financial data
  • To connect these CSOs with each other, with open data communities, and with other key stakeholders to exchange knowledge, experiences, and best practices in relation to spending data
  • To discover how CSOs currently work with spending data, how they would like to use it, and what they would like to achieve
This report is the result. It brings together key case studies from organisations who have done pioneering work in using technology to work with public finance data in each of budgets, spending, and procurements, and it presents a curated selection of tools and other advice in an appendix. As part of this research, we’ve also produced a four-part video series “Athens to Berlin“, which you can watch to meet some of the fascinating characters in the world of CSOs working with government spending data and to learn firsthand about their successes and their challenges. Originally Published on the Open Spending Blog Jan 3rd, 2014

Mapping the Open Spending Data Community

- January 3, 2014 in Releases

We’re pleased to announce the official release of “Mapping the Open Spending Data Community” by Anders Pedersen and Lucy Chambers, an in-depth look at how citizens, journalists, and civil society organisations around the world are using data on government finances to further their civic missions. The investigation began in 2012 with three goals: To identify […]

The Spending Data Handbook, revised and updated

- January 1, 2014 in Releases

The Spending Data Handbook is a user’s manual for government budget and spending data. If you want to learn what spending data is and how to work with it—whether you’re a journalist, a member of a civil society organization, or just an interested citizen—look no further than the Spending Data Handbook. In the Handbook, you’ll find […]

#dataroundup: help round up data news on Twitter

- December 12, 2013 in community

Photo credit: AgriLife Today

Do you like to keep tabs on new developments in data journalism, the latest in infographics, or on new tools and courses for data wrangling? Do you like to hang out on Twitter? If so, School of Data needs your help. Over the past year, we’ve experimented with different ways of running our regular Data Roundup column on the School of Data blog. Now we’d like to try something new: turning the Data Roundup over to the School of Data community. We invite all School of Data community members—whether mentors, learners, or just interested observers and well-wishers—to tweet about pieces of “data news” they find interesting with the hashtag #dataroundup. Every week, our Data Roundup editor (currently Marco Menchinella) will draw on the pool of tweets to create that week’s Roundup, giving credit to the user who found the story. Suggested topics for the Roundup include:
  • data-related tools or programming language libraries
  • tutorials and other learning materials
  • announcements of workshops, conferences, and other events
  • data visualizations and pieces of data-driven or quantitative journalism (especially from less-publicized sources)
  • data portals, open datasets, and other sources of data
But these are just suggestions—the content of the Roundup is up to you. Whatever it is that you, the School of Data community, find interesting in the world of data, we want to hear about it and to help spread the news. Join us on Twitter under the hashtag #dataroundup! flattr this!

Joined Up Data: first steps towards connecting transparency initiatives

- November 1, 2013 in Transparency

Unprecedented amounts of information on the financial resources available to fight poverty are now being released through the efforts of multi-stakeholder transparency initiatives in sectors like aid, construction, contracting, and extractives. Unleashing this data’s potential will mean joining it up, following the money by merging and comparing datasets from different sources. Transparency initiatives have the opportunity to make this possible by identifying the shared building blocks of their data standards that enable comparisons With this opportunity in mind, Development Initiatives (DevInit), the technical leads of the International Aid Transparency Initiative (IATI) and designers of the IATI standard for resource flow data, have initiated the Joined Up Data project. With this project, they aim to build a new culture of collaboration among multi-stakeholder transparency initiatives and to begin work on developing a shared set of building blocks for their data standards. Joined Up Data has begun with a scoping study developed for DevInit by the Open Knowledge Foundation and debuted at a workshop at the Open Government Partnership summit. The study identifies opportunities for collaboration between five transparency initiatives in different sectors: aid (IATI), construction (Construction Sector Transparency Initiative; CoST), contracting (Open Contracting), extractives (Extractive Industries Transparency Initiative; EITI), and fiscal transparency generally (Global Initiative for Fiscal Transparency). It assesses the coverage of each initiative’s data collection requirements and identifies areas of overlap that could become fruitful sites of collaboration, and it discusses the implementation challenges faced by each initiative and the governance processes underlying the creation of each initiative’s disclosure requirements. The scoping study demonstrates that there is much to discuss and a great deal of work to be done. The five initiatives covered in the study would all benefit from the development of building blocks covering organisational identifiers for government entities, geospatial data for sub-national administrative boundaries, and data standards for contracting information. The initiatives also have much to learn from one another’s structure: bottom-up initiatives like EITI and CoST can share the view from “on the ground”, whereas more top-down initiatives like IATI can contribute technical guidance. The Joined Up Data project is just taking its first steps, with future iterations promising to both deepen and broaden its coverage. The project also needs you. Improving the experience of using data is the most important goal of the Joined Up Data project. If you work with transparency initiative data (or want to), the organizers of Joined Up Data will surely want to hear how you use data and how it can be made more useful to you as they plan the next steps of the project. Contact us on the Open Spending list to get involved.  

Geocoding in Google Docs: GeoJSON boundaries with Koordinates

- October 31, 2013 in HowTo

GeoJSON with Koordinates In our geocoding recipe, you learned how to use Google Sheets formulas to automatically convert place names into coordinates in your spreadsheet data. In this tutorial, you’ll learn how to take geocoding a step further, converting coordinates to GeoJSON boundaries—machine-readable descriptions of the boundaries of geographical regions like countries or provinces—using the Koordinates API. The Koordinates API takes latitude and longitude coordinates and returns boundary shapes that are on or near those coordinates. In the geocoding recipe, you used built-in functions like ImportXML, CONCATENATE, and JOIN to get coordinates from place names using formulas like this:
=JOIN(",", ImportXML(CONCATENATE("http://open.mapquestapi.com/nominatim/v1/search?format=xml&q=",A2), "//place[1]/@lat | //place[1]/@lon"))
This formula is possible because the MapQuest API used in that tutorial returns XML data, which you can query using XPath strings like //place[1]/@lat with the built-in function ImportXML. But Koordinates, the geospatial data API we’re using in this tutorial, returns its results in JSON rather than XML, and Google Docs doesn’t have any built-in functions to traverse JSON objects. Hence we’ll have to define our own. To get started, you need to create an account with Koordinates and get an API key. To do create a key, log in, click on your name at the top of the page, select APIs and Web Services, and then click Accept terms and create key. Make a copy of the key that is generated—you’ll be using it soon. Now go your spreadsheet, access the Tools menu, and select Script editor. Click Close to get rid of the dialogue box that appears. Once you’re in the script editor, delete the text from the edit box and then save your file, giving it a memorable name (perhaps Koordinates API function). Now enter the following into the empty edit box, inserting your API key at the indicated spot:
function koords (coordsString) {
 /* Block 1. Formats lat-long coordinates for Koordinates API call. */
 coordsString = coordsString.replace(/\s/, "") ;
 var coords = coordsString.split(",") ;
 var xy = "&x=" + coords[1] + "&y=" + coords[0] ;

 /* Block 2. Formats API call, makes API call, parses the result,
    and stores the resulting object in "json". */
 var key = "YOUR API KEY GOES HERE, INSIDE THESE QUOTATION MARKS" ;
 var url = "http://api.koordinates.com/api/vectorQuery.json/?key=" + key + "&layer=1103&geometry=true" ;
 var data = UrlFetchApp.fetch(url + xy) ;
 var text = data.getContentText() ;
 var json = JSON.parse(text) ;

 /* Block 3. Traverses "json" to find boundary data and returns it. */
 var result = json["vectorQuery"]["layers"]["1103"]["features"][0]["geometry"] ;
 return JSON.stringify(result) ;

} ;
Let’s go through this code block by block. The first block of code converts a string representation of latitude-longitude coordinates into the format Koordinates expects, with longitude in first place. It throws out any spaces in the string (replace), splits the string on the character ",", and glues it back together in the reverse order. It also gets it set up to be inserted into the Koordinates API call by placing the longitude value after "&y=" and the longitude after "&x=". The next block sets up the URL for the API call and then makes the call. It asks for map layer 1103, which is a layer for country boundaries. It also requests “geometry”, meaning actual polygons (rather than just metadata, the default). The resulting JSON string is parsed into a JavaScript object with JSON.parse and put into the variable json. The JSON returned by the API looks like this: Koordinates API call result The polygon content we want, the “geometry”, is buried away deep inside the object. To get at it, we need to dig down through several layers of attributes. This is what the last block of code does, grabbing the "geometry" within the first item (item number 0) in the "features list of map objects returned for the query. It also turns the result into a string with JSON.stringify and returns it as the function’s value. Try calling your new custom function, giving it the name of a cell that contains lat-long coordinates (here, F4):
=koords(F4)
You should get in return a blob of JSON data—the rich geometric representation of the country containing the point in F4. You can now use koords like an ordinary Google Apps function. For example, you can wrap up the formula from the previous lesson to create a new formula that goes straight from place names to GeoJSON boundaries.
=koords(JOIN(",", ImportXML(CONCATENATE("http://open.mapquestapi.com/nominatim/v1/search?format=xml&q=",A2), "//place[1]/@lat | //place[1]/@lon")))
In this example, we used map layer 1103, which contains country boundaries. But Koordinates has many other layers which you might find useful. Check out the layers list to see what else you can do with the Koordinates API. flattr this!