You are browsing the archive for Sean Hammond.

New Template for CKAN Extensions

- November 21, 2014 in Extensions

We’ve just merged a new template for CKAN extensions. Whenever you create a new CKAN extension using the paster --plugin=ckan create -t ckanext ... command (as documented in the writing extensions tutorial) it’ll now use the new template, which gives you:
  • PyPI integration – and files are automatically generated for your extension, ready for publishing to PyPI
  • A tests directory including stub tests for you to get started writing tests for your extension
  • Travis CI integration – automatically run your tests in a clean environment each time you push a new commit to GitHub. A .travis.yml file and build and run scripts are automatically generated for your extension, you still need to log in to Travis and click the switch to turn on Travis for your extension though.
  • integration – track the code coverage of your tests. A .coveragerc file is automatically generated for your extension. Again, you still need to login to Coveralls and turn it on.
  • A .gitignore file
  • A LICENSE file (uses the GNU AGPL by default)
  • A reStructuredText README file with a skeleton documentation structure including generated installation and configuration instructions, how to run the tests, etc
  • Travis, Coveralls and README badges! Show the world that you have continuous integration, good test coverage, PyPI downloads, and your extension’s supported Python version, development status and license.
Screenshot from 2014-11-21 16:26:14 For an example of an extension built using this template, look at ckanext-deadoralive. What we’re trying to do with this new template is:
  1. Save ourselves time, by not having to manually create all of this boilerplate every time we roll a new CKAN extension
  2. Help improve the quality of CKAN extensions by encouraging developers to write good tests and documentation, and to use services PyPI, Travis and Coveralls
More to come. If you have any ideas for things to add to the CKAN extension template, let us know on ckan-dev

Export Datasets from CKAN to Excel

- November 9, 2014 in Uncategorized

ckanapi-exporter is a new API script that we’ve developed for exporting dataset metadata from CKAN to Excel-compatible CSV files. Check out the short presentation below, and visit ckanapi-exporter for more details:

New Broken Link Checker Plugin for CKAN

- October 21, 2014 in Extensions, Presentations

deadoralive is a new broken-link checker service that works with CKAN and other sites, and ckanext-deadoralive is a CKAN plugin that you can install to integrate your CKAN site with the link checker. The pair have been developed by Open Knowledge as part of our work on the new version of Öppnadata, the Swedish national open data portal. This quick presentation gives an overview of the link checker’s features and design: For more details, see these blog posts:

Senior Python web developer wanted for world leading open source data portal

- July 20, 2014 in Jobs

Working in the fast-growing area of open data, we build open source tools to drive transparency, accountability and re-use. Our flagship product CKAN runs the official national data portals from the UK to Brazil, US to Australia and many others. We also build data tools and OpenSpending browsers. We’re looking for someone passionate about the technical challenges of building software that is used as the infrastructure for open data around the world, so come join our growing team to shape the future of the open data ecosystem!

Key Skills

  • Python, JavaScript, HTML, CSS
  • Python web frameworks (we use Pylons and Flask)
  • PostgreSQL and SQLAlchemy
  • Self motivated, self-starter, able to manage your own time
Extra bonus points for:
  • Open source projects/contributions
  • Front end skills, particularly in data-vis
  • You’ve written an app using open data before
  • Experience working in a distributed team

How to apply

Email, with the subject line “Python Developer – CKAN/Services”. Please include:
  1. Your CV
  2. A link to your GitHub (or similar) profile
  3. A cover letter

More about the Job

You will be working as part of a small, dynamic team in a modern, open-source development environment. This role is full-time and we are very happy with remote-working. We generally work remotely (with strong contingents in London and Berlin), using asynchronous communication (email, IRC, GitHub) but with standups, developer meetings and demos most days (Skype, Google Hangout) and real-world gatherings more than twice a year including at the Open Knowledge Festival. We also try and ensure our developers can attend at least one open source conference a year. At each level of our software stack we use best-in-class open source software, including Python, Nose, Travis CI and Coveralls, Sphinx and Read the Docs, Flask, Jinja2, Solr, PostgreSQL and SQLAlchemy, JavaScript and jQuery, Bootstrap, Git and Github, and Transifex. We iterate quickly, and publish working, open-source code early and often. All of our code is on github – it is open to public scrutiny, and we encourage contributions from third-party developers. This means that we have to write exceptionally clear, readable, well-tested code with excellent documentation. All code contributions, whether from internal or external developers, are made with GitHub pull requests and we do code reviews in the open on GitHub. We are engaged with a large and active community of users, developers and translators of our open source software, via our mailing lists, GitHub issues and pull requests, public developer meetings, Stack Overflow and Transifex. We support users in getting started with our software, encourage and mentor new developers, and take on feedback and suggestions for the next releases.

About the Open Knowledge Foundation

The Open Knowledge Foundation (OKF) is an internationally recognized non-profit working to open knowledge and see it used to empower and improve the lives of citizens around the world. We build tools, provide advice and develop communities in the area of open knowledge: data, content and information which can be freely shared and used. We believe that by creating an open knowledge commons we can make a significant contribution to improving governance, research and the economy. The last two years have seen rapid growth in our activities, increasing our annual revenue to £2m and our team to over 35 across four continents. We are a virtual organisation with the whole team working remotely, although we have informal clusters in London, Cambridge and Berlin. The OKF is an international leader in its field and has extensive experience in building open source tools and communities around open material. The Foundation’s software development work includes some of the most innovative and widely acclaimed projects in the area. For example, its CKAN project is the world’s leading open source data portal platform – used by,, the European Commission’s open data portal, and numerous national, regional and local portals from Austria to Brazil. The award winning OpenSpending project enables users to explore over 13 million government spending transactions from around the world. It has an active global network which includes Working Groups and Local Groups in dozens of countries – including groups, ambassadors and partners in 21 of Europe’s 27 Member States. We’re changing the world by promoting a global shift towards more open ways of working in government, arts, sciences and much more.

The Open Knowledge Data Packager

- June 9, 2014 in Releases

Today we’re launching the Open Knowledge Data Packager! Data Packager is a web app for quickly creating and publishing Tabular Data Packages from collections of CSV files on your computer. You can register for a free user account and start creating data packages now, or take a look at a sample data package. With Data Packager’s simple interface you can create a data package, upload CSV files to it, enter some metadata, and get a web page where users can explore and download your data package. When you login, you’ll be taken to your dashboard, where you’ll see a list of any packages you’ve created so far and Add package button:
Data Packager dashboard screenshot

My Data Packager dashboard

Click the Add package button to create a new data package and you’ll be taken to a form where you can enter the title and other metadata for your package:
Creating a new data package

Creating a new data package

Click on Next: Add CSV files and you’ll be taken to a form where you can upload one or more CSV files to your data package:
Uploading CSV files to a new data package

Uploading CSV files to a new data package

Finally, click on Finish to create your data package. You’ll be taken to your data package’s page:
Browsing your newly created data package

Browsing your newly created data package

You can publish the URL of this page, or share it with anyone who you want to share your data package with.

Why Tabular Data Packages?

Tabular Data Packages (defined by the Tabular Data Package spec) are a simple and easy-to-use data publishing and sharing format for the web. A Tabular Data Package is a collection of CSV files with a datapackage.json file. The datapackage.json file contains metadata about the package (title of the package, description, keywords, license, etc.) and schemas for each of the package’s CSV files. The format is a good compromise between CSV and Excel, providing the simplicity and ease-of-use of CSV with some of the expressivity of full-blown spreadsheets. The schemas for the CSV files use the JSON Table Schema format, a simple format for tabular data schemas. It includes metadata for each of the CSV file’s columns (column name, type, description, etc.) and optional primary and foreign keys for the file.

Data Packager Features

After you’ve created your data package and uploaded some CSV files to it, Data Packager has a few nice features for you…

Download data packages

The Download Data Package button on your data package’s page will download a ZIP file including all of your package’s CSV files and the datapackage.json file containing the metadata you entered for your package and files, plus schemas for each of your files: Screenshot of Download Data Package button

Schema browser

Data Packager automatically generates a JSON Table Schema for each CSV file that you upload. The generated schema includes:
  • Column names for each of the file’s columns (taken from the CSV file’s header row, if it has one)
  • The type of the data in each column (string, number, date…), inferred from the values in the columns
  • Some descriptive statistics calculated for numerical columns (minimum and maximum values, mean, standard deviation…)
  • Temporal extents (earliest and latest dates) for date and time columns
By clicking on one of the CSV files on your data package’s page, you can browse the file’s schema using the schema browser. Each file’s page shows a preview of the CSV file contents, and by clicking on the columns in the preview you can inspect the schema for each column:
The schema browser

The schema browser

Schema editor

By clicking the Edit button on one of your CSV file’s pages, you can edit the file’s JSON Table Schema and add your own custom attributes. Data Packager validates all the changes that you make and gives helpful error messages if you try to save an invalid schema.
The schema editor

The schema editor

Primary and foreign keys

If you add primary and foreign keys to a CSV file’s schema, they’ll also be shown on the file’s page.
Primary and foreign keys

Primary and foreign keys


All of Data Packager’s features can also be used via its JSON API.

Open Source

Data Packager is 100% open source. You can:

Built with CKAN

Data Packager is built using CKAN, the highly-customisable open source data portal platform. All Data Packager features are implemented by a CKAN extension, ckanext-datapackager.

Translators needed!

- March 11, 2013 in Releases

We’re preparing to release CKAN 2.0 in a few weeks time, and we need help with translations. CKAN’s user interface is available in over 20 languages. The new version introduces over 450 new words and strings in the interface, and they all need translating before it’s released. If you speak one of the relevant languages, you can join that language team to help with the translation effort. If you are a speaker of a language where CKAN isn’t available yet, this would also be a great time to start a translation project for it! If you can help with translating the strings and keeping CKAN open to people across the world, please sign up here to contribute.