You are browsing the archive for Adrià Mercader.

Validation for Open Data Portals: a Frictionless Data Case Study

- December 18, 2017 in case study, ckan, Data Quality, Frictionless Data, goodtables

The Frictionless Data project is about making it effortless to transport high quality data among different tools and platforms for further analysis. We are doing this by developing a set of software, specifications, and best practices for publishing data. The heart of Frictionless Data is the Data Package specification, a containerization format for any kind of data based on existing practices for publishing open-source software. Through its pilots, Frictionless Data is working directly with organisations to solve real problems managing data. The University of Pittsburgh’s Center for Urban and Social Research is one such organisation. One of the main goals of the Frictionless Data project is to help improve data quality by providing easy to integrate libraries and services for data validation. We have integrated data validation seamlessly with different backends like GitHub and Amazon S3 via the online service goodtables.io, but we also wanted to explore closer integrations with other platforms. An obvious choice for that are Open Data portals. They are still one of the main forms of dissemination of Open Data, especially for governments and other organizations. They provide a single entry point to data relating to a particular region or thematic area and provide users with tools to discover and access different datasets. On the backend, publishers also have tools available for the validation and publication of datasets. Data quality varies widely across different portals, reflecting the publication processes and requirements of the hosting organizations. In general, it is difficult for users to assess the quality of the data and there is a lack of descriptors for the actual data fields. At the publisher level, while strong emphasis has been put in metadata standards and interoperability, publishers don’t generally have the same help or guidance when dealing with data quality or description. We believe that data quality in Open Data portals can have a central place on both these fronts, user-centric and publisher-centric, and we started this pilot to showcase a possible implementation. To field test our implementation we chose the Western Pennsylvania Regional Data Center (WPRDC), managed by the University of Pittsburgh Center for Urban and Social Research. WPRDC is a great example of a well managed Open Data portal, where datasets are actively maintained and the portal itself is just one component of a wider Open Data strategy. It also provides a good variety of publishers, including public sector agencies, academic institutions, and nonprofit organizations. The portal software that we are using for this pilot is CKAN, the world leading open source software for Open Data portals (source). Open Knowledge International initially fostered the CKAN project and is now a member of the CKAN Association. We created ckanext-validation, a CKAN extension that provides a low level API and readily available features for data validation and reporting that can be added to any CKAN instance. This is powered by goodtables, a library developed by Open Knowledge International to support the validation of tabular datasets. The ckanext-validation extension allows users to perform data validation against any tabular resource, such as  CSV or Excel files. This generates a report that is stored against a particular resource, describing issues found with the data, both at the structural level, such as missing headers and blank rows,  and at the data schema level, such as wrong data types and  out of range values. Read the technical details about this pilot study, our learnings and areas we have identified for further work in the coming days here on the Frictionless Data website.

CKAN 2.6.0 released, patch versions for 2.3.x, 2.4.x and 2.5.x available

- November 2, 2016 in Releases

We are happy to announce that CKAN 2.6.0 is now released. In addition, new patch releases for older versions of CKAN are now available to download and install. CKAN 2.6 The 2.6.0 release includes improvements on how private datasets are shown on search results, as well as several other minor improvements and over 50 bug fixes. You can check all individual changes on the CHANGELOG. Thank you very much to all the almost 30 community members that have submitted patches since the last release. If you have customizations or extensions, we suggest you trial the upgrade first in a test environment and refer to the changes in the changelog. Upgrade instructions are below. As there aren’t many major changes since the last version upgrading should be relatively straight-forward. Note that as previously announced, starting from this version, CKAN requires at least Python 2.7 and Postgres 9.2. CKAN patch releases These new patch releases for CKAN 2.3.x, 2.4.x and 2.5.x (2.3.5, 2.4.4 and 2.5.3) fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using. For a list of the fixes included you can check the CHANGELOG. Upgrading For details on how to upgrade, see the following links depending on your install method: Upgrading CKAN. If you find any issue, you can let the technical team know in the mailing list or the IRC channel.  

CKAN patch releases 2.3.4, 2.4.3 and 2.5.2 now available

- March 31, 2016 in Uncategorized

The CKAN team is happy to announce that the new patch releases for CKAN 2.3.x, 2.4.x and 2.5.x are now available to download and install. These patch releases fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using. Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes or involve any change in the requirements, database or Solr schema. As stated in the releases policy, the latest patch release is the only one officially supported. For details on how to upgrade, see the following links depending on your install method: If you find any problem, let us know in the mailing list or the IRC channel.

CKAN patch releases 2.3.4, 2.4.3 and 2.5.2 now available

- March 31, 2016 in Uncategorized

The CKAN team is happy to announce that the new patch releases for CKAN 2.3.x, 2.4.x and 2.5.x are now available to download and install. These patch releases fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using. Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes or involve any change in the requirements, database or Solr schema. As stated in the releases policy, the latest patch release is the only one officially supported. For details on how to upgrade, see the following links depending on your install method: If you find any problem, let us know in the mailing list or the IRC channel.

CKAN patch releases 2.3.4, 2.4.3 and 2.5.2 now available

- March 31, 2016 in Uncategorized

The CKAN team is happy to announce that the new patch releases for CKAN 2.3.x, 2.4.x and 2.5.x are now available to download and install. These patch releases fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using. Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes or involve any change in the requirements, database or Solr schema. As stated in the releases policy, the latest patch release is the only one officially supported. For details on how to upgrade, see the following links depending on your install method: If you find any problem, let us know in the mailing list or the IRC channel.

Code of Conduct

- January 27, 2016 in Association, community

As the CKAN community grows and includes more people from various backgrounds it seems like a good time to adopt a Code of Conduct that will ensure it remains a welcoming place for everybody. The Code of Conduct can be accessed on the main CKAN repository: https://github.com/ckan/ckan/blob/master/CONDUCT.rst Rather than trying to come up with a useful one ourselves we have adopted one based on The Open Code of Conduct. As stated on the code, if you feel this has been breached you can contact conduct at ckan.org. This currently forwards to the members of the tech team. As ever, feel free to send us any comments or feedback.

Code of Conduct

- January 27, 2016 in Featured, oer

As the CKAN community grows and includes more people from various backgrounds it seems like a good time to adopt a Code of Conduct that will ensure it remains a welcoming place for everybody. The Code of Conduct can be accessed on the main CKAN repository: https://github.com/ckan/ckan/blob/master/CONDUCT.rst Rather than trying to come up with a useful one ourselves we have
adopted one based on The Open Code of Conduct. As stated on the code, if you feel this has been breached you can
contact conduct at ckan.org. This currently forwards to the members of
the tech team. As ever, feel free to send us any comments or feedback.

Code of Conduct

- January 27, 2016 in Association, community

As the CKAN community grows and includes more people from various backgrounds it seems like a good time to adopt a Code of Conduct that will ensure it remains a welcoming place for everybody. The Code of Conduct can be accessed on the main CKAN repository: https://github.com/ckan/ckan/blob/master/CONDUCT.rst Rather than trying to come up with a useful one ourselves we have
adopted one based on The Open Code of Conduct. As stated on the code, if you feel this has been breached you can
contact conduct at ckan.org. This currently forwards to the members of
the tech team. As ever, feel free to send us any comments or feedback.

CKAN patch releases 2.0.5, 2.1.3 and 2.2.1 now available

- October 15, 2014 in Releases

We are happy to announce that new patch releases for CKAN 2.0.x, 2.1.x and 2.2.x are now available to download and install. These patch releases fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using. For a list of the fixes included you can check the CHANGELOG: Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes or involve any change in the requirements, database or Solr schema. As stated in the release policy, the latest patch release is the only one officially supported. For details on how to upgrade, see the following links depending on your install method: If you find any issue, you can let the technical team know in the mailing list or the IRC channel.  

CKAN Census 2014

- May 12, 2014 in community, Deployments, Featured

Census!   Update: We have now also published a Deployment Survey to learn more about how CKAN is deployed and installed: /deployment-survey/
CKAN is used by hundreds of organizations across the world to publish their Open Data on-line. More and more CKAN sites are going live and CKAN is being used in many new ways, integrating with other tools and being extended with new features. We want to get a clearer picture of the current CKAN ecosystem to better understand how CKAN is being used and help scope the project roadmap. And if at the same time we can bring the ckan.org instances page up to date all the better! So if you are developing or managing a CKAN site or know someone who does, can you spend 5 minutes filling out this quick survey? /census Any help in spreading out the link will be much appreciated as well. The results will make a big difference, and of course, they will be made available to the community later on. This is the first in a series of surveys that will focus on different aspects of maintaining and using CKAN. In the following weeks we’ll send another one around focused on Deployment and Installation. Let’s map CKAN usage across the world!   Photo by suziesparkle