You are browsing the archive for Feature.

Join us for CKANCon 2016!

- July 14, 2016 in Association, community, Events, Feature, Featured

Join us in Madrid, Spain on October 4th, for CKANCon 2016, one of the official International Open Data Conference Pre-Events. CKANCon is a day packed with talks and discussions showcasing the incredible work people are doing with CKAN. This includes topics ranging from uses and best CKAN practices to technical services and new extensions. New, long-standing, and future CKAN users are encouraged to attend. Full details, including speakers and breakout sessions, will be announced soon. If you’re interested in showcasing your CKAN work, please email ckancon@ckan.org! We are looking for speakers to give short talks about upcoming features, extensions, integrations and anything else CKAN. CKAN-Gif_v4

Join us for CKANCon 2016!

- July 14, 2016 in Association, community, Events, Feature, Featured

Join us in Madrid, Spain on October 4th, for CKANCon 2016, one of the official International Open Data Conference Pre-Events. UPDATE: We are happy to announce that registration for the event is now open! You can register today for both in person and online participation! CKANCon is a day packed with talks and discussions showcasing the incredible work people are doing with CKAN. This includes topics ranging from uses and best CKAN practices to technical services and new extensions. New, long-standing, and future CKAN users are encouraged to attend. Full details, including speakers and breakout sessions, will be announced soon. If you’re interested in showcasing your CKAN work, please email ckancon@ckan.org! We are looking for speakers to give short talks about upcoming features, extensions, integrations and anything else CKAN. CKAN-Gif_v4

Showcase your data

- September 21, 2015 in Extensions, Feature, Featured

We all know CKAN is great for publishing and managing data, and it has powerful visualisation tools to provide instant insights and analysis. But it’s also useful and inspiring to see examples of how open data is being used. CKAN has previously provided for this with the ‘Related Items’ feature (also known as ‘Apps & Ideas’). We wanted to enhance this feature to address some of its shortcomings, packaged up as an extension to easily replace and migrate from Related Items. So we developed the Showcase extension!
Showcase Example

A Showcase details page. This Showcase example is originally from http://data.beta.nyc/showcase/illegal-hotels-inspections

Separating out useful, but under-loved features from CKAN core to extensions like this:
  • makes core CKAN a leaner and a more focused codebase
  • gives these additional features a home, with more dedicated ownership and support
  • means updates and fixes for an extension don’t have to wait until the next release of CKAN
Some improvements made in Showcase include:
  • each showcase has its own details page
  • more than one dataset can be linked to a showcase
  • a new role of Showcase Admin to help manage showcases
  • free tagging of showcases, instead of a predefined list of ‘types’
  • showcase discovery by search and filtering by tag
This was my first contribution to the CKAN project and I wanted to ensure the established voices from the CKAN developer community were able to contribute guidance and feedback. Remote collaboration can be hard, so I looked at the tools we already use as a team, to lower the barrier to participation. I wanted something that was versioned, allowed commenting and collaboration, and provided notification to interested parties as the specification developed. We use GitHub to collect ideas for new features in a repository as Issues, so it seemed like a natural extension to take these loose issues (ideas) and turn them into pull requests (proposals). The proposal and supporting documents can be committed as simple MarkDown files, and discussed within the Pull Request. This provides line-by-line commentary tools enabling quick iteration based on the feedback. If a proposal is accepted and implemented, the pull request can be merge, if the proposal is unsuccessful, it can be closed. The Pull Request for the Showcase specification has 22 commits, and 57 comments from nine participants. Their contributions were invaluable and helped to quickly establish what and how the extension was going to be built. Their insights helped me get up to speed with CKAN and its extension framework and prevented me from straying too far in the wrong direction. So, by developing the specification and coding in the open, we’ve managed to take an unloved feature of CKAN and give it a bit of polish and hopefully a new lease of life. I’d love to hear how you’re using it!

Showcase your data

- September 21, 2015 in Extensions, Feature, Featured

We all know CKAN is great for publishing and managing data, and it has powerful visualisation tools to provide instant insights and analysis. But it’s also useful and inspiring to see examples of how open data is being used. CKAN has previously provided for this with the ‘Related Items’ feature (also known as ‘Apps & Ideas’). We wanted to enhance this feature to address some of its shortcomings, packaged up as an extension to easily replace and migrate from Related Items. So we developed the Showcase extension!
Showcase Example

A Showcase details page. This Showcase example is originally from http://data.beta.nyc/showcase/illegal-hotels-inspections

Separating out useful, but under-loved features from CKAN core to extensions like this:
  • makes core CKAN a leaner and a more focused codebase
  • gives these additional features a home, with more dedicated ownership and support
  • means updates and fixes for an extension don’t have to wait until the next release of CKAN
Some improvements made in Showcase include:
  • each showcase has its own details page
  • more than one dataset can be linked to a showcase
  • a new role of Showcase Admin to help manage showcases
  • free tagging of showcases, instead of a predefined list of ‘types’
  • showcase discovery by search and filtering by tag
This was my first contribution to the CKAN project and I wanted to ensure the established voices from the CKAN developer community were able to contribute guidance and feedback. Remote collaboration can be hard, so I looked at the tools we already use as a team, to lower the barrier to participation. I wanted something that was versioned, allowed commenting and collaboration, and provided notification to interested parties as the specification developed. We use GitHub to collect ideas for new features in a repository as Issues, so it seemed like a natural extension to take these loose issues (ideas) and turn them into pull requests (proposals). The proposal and supporting documents can be committed as simple MarkDown files, and discussed within the Pull Request. This provides line-by-line commentary tools enabling quick iteration based on the feedback. If a proposal is accepted and implemented, the pull request can be merge, if the proposal is unsuccessful, it can be closed. The Pull Request for the Showcase specification has 22 commits, and 57 comments from nine participants. Their contributions were invaluable and helped to quickly establish what and how the extension was going to be built. Their insights helped me get up to speed with CKAN and its extension framework and prevented me from straying too far in the wrong direction. So, by developing the specification and coding in the open, we’ve managed to take an unloved feature of CKAN and give it a bit of polish and hopefully a new lease of life. I’d love to hear how you’re using it!

Implementing VectorTiles Preview of Geodata on HDX

- September 16, 2015 in Extensions, Feature, visualization

This post is modified version of a post on the HDX blog.  It is modified here to highlight information of most interest to the CKAN community.  You can see the original post here. Humanitarian data is almost always inherently geographic. Even the data in a simple CSV file will generally correspond to some piece of geography: a country, a district, a town, a bridge, or a hospital, for example. HDX has built on CKAN’s preview capabilities with the ability to preview large (up to 500MB) vector geographic datasets in a variety of formats.  Resources uploaded (or linked) to HDX with the format strings ‘geojson’, ‘zipped shapefile’, or ‘kml’ will trigger the creation of a geo preview. Here is an example showing administrative boundaries for Colombia:
image00
To minimize bandwidth in the interest of often poorly-connected field locations, we built the preview from vector tiles. This means that details are removed at small scales but will reappear as you zoom in. The preview is created only for the first layer it encounters in a resource. If the resource contains multiple layers, the others will not show up. For those cases, you can create separate resources for each layer and they will be available in the preview. Multiple geometry types (polygon + line, for example) in kml or geojson are not yet supported.

Implementation

It’s a common problem in interactive mapping: to preview the whole geographic dataset, we would need to send all of the data to the browser, but that can require a long download or even crash the browser. The classic solution is to use a set of pre-rendered map tiles — static map images made for different zoom levels and cut into tiny pieces called tiles.  The browser has to load only a few of these pieces for any given view of the map. However, because they are just raster images, the user cannot interact with them in any advanced way. We wanted to maintain interactivity with the data, eventually having hover effects or allowing users to customize styling, so we knew that we needed a different approach. We reached out to our friends at Geonode who pointed us to the recently developed Vector Tiles Specification. The vector tile solution is a similar approach to traditional map tiles, but instead of creating static image tiles, it involves cutting the geodata layer into small tiles of vector data. Each zoom level receives a simplification (level of detail, or LoD) pass, which reduces the number of vertices displayed, similar to the way that 3D video games or simulators reduce the number of polygons in distant objects to improve performance. This means that for any given zoom level and location, the browser needs to download only the vertices necessary to fill the map.  You can learn more about how vector tiles work in this helpful FOSS4G NA talk from earlier this year. Because vector tiles are a somewhat-new technology, there wasn’t any off-the-shelf framework to let us integrate them with our CKAN instance. Instead, we built a custom solution from several existing components (along with our own integration code): Our architecture looks like this: image03 The GISRestLayer orchestrates the entire process by notifying each component when there is a task to do. It then informs CKAN when the task is complete, and a dataset has a geo preview available.  It can take a minute or longer to generate the preview, so the asynchronous approach — managed through Redis Queue (RQ) — was essential to let our users continue to work while the process is running. A special HDX team member, Geodata Preview Bot, is used to make the changes to CKAN. This makes the nature of the activity on the dataset clear to our users.

Future development

This approach gives HDX a good foundation for adding new geodata features in the future. We will be conducting research to understand what users think is important to add next. Here are some initial new-feature ideas:
  • Automatically generate additional download formats so that every geodataset is available in zipped shapefile, GeoJSON, KML, etc.
  • Allow the contributing user to specify the order of the resources in the map legend (and therefore which one appears by default).
  • Allow users to preview multiple datasets on the same map for comparison.
  • Automatically apply different symbol colors to different resources in the same dataset.
  • Allow users to style the geographic data, changing colors and symbols.
  • Allow users to configure and embed maps of their data in their organization or crisis pages.
  • Provide OGC-compliant web services of contributed datasets (WFS, WMS, etc.).
  • Allow external geographic data services (WMS, WFS, etc) to be added to a map preview.
  • Make our vector tiles available as a web service.
If any of these enhancements sound useful or you have new ideas, send us an email at hdx.feedback@gmail.com. If you have geodata to share with the HDX community, start adding your data here. We would like to say a special thanks to Jeffrey Johnson who pointed us toward the vector tiles solution and to the contributors of all the open source projects listed above! In addition to GISRestLayer, you’ll find the rest of our code here.

Implementing VectorTiles Preview of Geodata on HDX

- September 16, 2015 in Extensions, Feature, visualization

This post is modified version of a post on the HDX blog.  It is modified here to highlight information of most interest to the CKAN community.  You can see the original post here. Humanitarian data is almost always inherently geographic. Even the data in a simple CSV file will generally correspond to some piece of geography: a country, a district, a town, a bridge, or a hospital, for example. HDX has built on CKAN’s preview capabilities with the ability to preview large (up to 500MB) vector geographic datasets in a variety of formats.  Resources uploaded (or linked) to HDX with the format strings ‘geojson’, ‘zipped shapefile’, or ‘kml’ will trigger the creation of a geo preview. Here is an example showing administrative boundaries for Colombia: image00 To minimize bandwidth in the interest of often poorly-connected field locations, we built the preview from vector tiles. This means that details are removed at small scales but will reappear as you zoom in. The preview is created only for the first layer it encounters in a resource. If the resource contains multiple layers, the others will not show up. For those cases, you can create separate resources for each layer and they will be available in the preview. Multiple geometry types (polygon + line, for example) in kml or geojson are not yet supported.

Implementation

It’s a common problem in interactive mapping: to preview the whole geographic dataset, we would need to send all of the data to the browser, but that can require a long download or even crash the browser. The classic solution is to use a set of pre-rendered map tiles — static map images made for different zoom levels and cut into tiny pieces called tiles.  The browser has to load only a few of these pieces for any given view of the map. However, because they are just raster images, the user cannot interact with them in any advanced way. We wanted to maintain interactivity with the data, eventually having hover effects or allowing users to customize styling, so we knew that we needed a different approach. We reached out to our friends at Geonode who pointed us to the recently developed Vector Tiles Specification. The vector tile solution is a similar approach to traditional map tiles, but instead of creating static image tiles, it involves cutting the geodata layer into small tiles of vector data. Each zoom level receives a simplification (level of detail, or LoD) pass, which reduces the number of vertices displayed, similar to the way that 3D video games or simulators reduce the number of polygons in distant objects to improve performance. This means that for any given zoom level and location, the browser needs to download only the vertices necessary to fill the map.  You can learn more about how vector tiles work in this helpful FOSS4G NA talk from earlier this year. Because vector tiles are a somewhat-new technology, there wasn’t any off-the-shelf framework to let us integrate them with our CKAN instance. Instead, we built a custom solution from several existing components (along with our own integration code): Our architecture looks like this: image03 The GISRestLayer orchestrates the entire process by notifying each component when there is a task to do. It then informs CKAN when the task is complete, and a dataset has a geo preview available.  It can take a minute or longer to generate the preview, so the asynchronous approach — managed through Redis Queue (RQ) — was essential to let our users continue to work while the process is running. A special HDX team member, Geodata Preview Bot, is used to make the changes to CKAN. This makes the nature of the activity on the dataset clear to our users.

Future development

This approach gives HDX a good foundation for adding new geodata features in the future. We will be conducting research to understand what users think is important to add next. Here are some initial new-feature ideas:
  • Automatically generate additional download formats so that every geodataset is available in zipped shapefile, GeoJSON, KML, etc.
  • Allow the contributing user to specify the order of the resources in the map legend (and therefore which one appears by default).
  • Allow users to preview multiple datasets on the same map for comparison.
  • Automatically apply different symbol colors to different resources in the same dataset.
  • Allow users to style the geographic data, changing colors and symbols.
  • Allow users to configure and embed maps of their data in their organization or crisis pages.
  • Provide OGC-compliant web services of contributed datasets (WFS, WMS, etc.).
  • Allow external geographic data services (WMS, WFS, etc) to be added to a map preview.
  • Make our vector tiles available as a web service.
If any of these enhancements sound useful or you have new ideas, send us an email at hdx.feedback@gmail.com. If you have geodata to share with the HDX community, start adding your data here. We would like to say a special thanks to Jeffrey Johnson who pointed us toward the vector tiles solution and to the contributors of all the open source projects listed above! In addition to GISRestLayer, you’ll find the rest of our code here.

Presenting public finance just got easier

- March 20, 2015 in Extensions, Feature, Featured, Releases, visualization

mexico_ckan_openspending CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new features of this release is the ability to create extensions that get called before and after a new file is uploaded, updated, or deleted on a CKAN instance. This may not sound like a major improvement  but it creates a lot of new opportunities. Now it’s possible to analyse the files (which are called resources in CKAN) and take them to new uses based on that analysis. To showcase how this works, Open Knowledge in collaboration with the Mexican government, the World Bank (via Partnership for Open Data), and the OpenSpending project have created a new CKAN extension which uses this new feature. It’s actually two extensions. One, called ckanext-budgets listens for creation and updates of resources (i.e. files) in CKAN and when that happens the extension analyses the resource to see if it conforms to the data file part of the Budget Data Package specification. The budget data package specification is a relatively new specification for budget publications, designed for comparability, flexibility, and simplicity. It’s similar to data packages in that it provides metadata around simple tabular files, like a csv file. If the csv file (a resource in CKAN) conforms to the specification (i.e. the columns have the correct titles), then the extension automatically creates the Budget Data Package metadata based on the CKAN resource data and makes the complete Budget Data Package available. It might sound very technical, but it really is very simple. You add or update a csv file resource in CKAN and it automatically checks if it contains budget data in order to publish it on a standardised form. In other words, CKAN can now automatically produce standardised budget resources which make integration with other systems a lot easier. The second extension, called ckanext-openspending, shows how easy such an integration around standardised data is. The extension takes the published Budget Data Packages and automatically sends it to OpenSpending. From there OpenSpending does its own thing, analyses the data, aggregates it and makes it very easy to use for those who use OpenSpending’s visualisation library. So thanks to a perhaps seemingly insignificant extension feature in CKAN 2.3, getting beautiful and understandable visualisations of budget spreadsheets is now only an upload to a CKAN instance away (and can only get easier as the two extensions improve). To learn even more, see this report about the CKAN and OpenSpending integration efforts.

International Open Data Day 2015 – Open Knowledge Philippines

- March 6, 2015 in Feature, HackODD2015, ODD2015, OKFN-PH, Open Data

In the Philippines, Open Knowledge and School of Data local group celebrated the International Open Data Day 2015 with back to back events on February 20-21, 2015. The event was officially opened on 20 February 2015 at Mapua Institute of Technology, Makati campus. Mapua is a higher education institution in the Philippines offering engineering and sciences, […]

International Open Data Day 2015 – Open Knowledge Philippines

- March 6, 2015 in Feature, HackODD2015, ODD2015, OKFN-PH, Open Data

In the Philippines, Open Knowledge and School of Data local group celebrated the International Open Data Day 2015 with back to back events on February 20-21, 2015. The event was officially opened on 20 February 2015 at Mapua Institute of Technology, Makati campus. Mapua is a higher education institution in the Philippines offering engineering and sciences, […]

International Open Data Day 2015 – Open Knowledge Philippines

- March 6, 2015 in Feature, HackODD2015, ODD2015, OKFN-PH

In the Philippines, Open Knowledge and School of Data local group celebrated the International Open Data Day 2015 with back to back events on February 20-21, 2015. The event was officially opened on 20 February 2015 at Mapua Institute of Technology, Makati campus. Mapua is a higher education institution in the Philippines offering engineering and sciences, […]