You are browsing the archive for tools.

Data Curator – share usable open data

- March 14, 2019 in Frictionless Data, tools

Data Curator is a simple desktop editor to help describe, validate, and share usable open data.

Open data producers are increasingly focusing on improving open data so it can be easily used to create insight and drive positive change. Open data is more likely to be used if data consumers can:

  • understand the structure and quality of the data
  • understand why and how the data was collected
  • look up the meaning of codes used in the data
  • access the data in an open machine-readable format
  • know how the data is licensed and how it can be reused

Data Curator enables open data producers to define all this information, and validate the data, prior to publishing it on the Internet. The data is published as a Tabular Data Package following the Frictionless Data specification. This allows open data consumers to read the data using Frictionless Data applications and software libraries.

“We need to make it easy to manage data throughout its lifecycle and ensure it can be easily and reliably retrieved by people who want to reuse and repurpose it. We developed Data Curator to help publishers define certain characteristics to improve data and metadata quality” – Dallas Stower, Assistant Director-General, Digital Platforms and Data, Queensland Government – Project Sponsor

< p class="part" data-startline="21" data-endline="21">Data Curator allows you to create data from scratch or open an Excel or CSV file. Data Curator requires that each column of data is given a type (e.g. text, number). Data can be defined further using a format (e.g. text may be a URL or email). Constraints can be applied to data values (e.g. required, unique, minimum value, etc.). This definition process can be accelerated by using the Guess feature, that guesses the data types and formats for all columns.

Data can be validated against the column type, format and constraints to identify and correct errors. If it’s not appropriate to correct the errors, they can be added to the provenance information to help people understand why and how the data was collected and determine if it is fit for their purpose.

Data Curator screenshot

Often a set of codes used in the data is defined in another table. Data Curator lets you validate data across tables. This is really useful if you want to share a set of standard codes across different datasets or organisations.

Data Curator lets you save data as a comma, semicolon, or tab separated value file. After you’ve applied an open license to the data, you can export a data package containing the data, its description, and provenance information. The data package can then be published to the Internet. Some open data platforms support uploading, displaying, and downloading data packages. Open data consumers can then confidently access and use quality open data.

Get Started

Download Data Curator for Windows or macOS.

Learn more about Data Curator and Frictionless Data.

Who made Data Curator?

Data Curator was made possible with funding and guidance from the Queensland Government.

The project was led by Stephen Gates from the ODI Australian Network. Software development made possible by Gavin Kennedy and Matt Mulholland from the Queensland Cyber Infrastructure Foundation (QCIF).

Data Curator uses the Frictionless Data software libraries maintained by Open Knowledge International. Data Curator started life as Comma Chameleon an experiment by the Open Data Institute.

How to deal with IP rights of growing projects and products at hackathons – experience from the Japanese context

- February 28, 2019 in hackathon, IP, japan, OK Japan, Open Data Day, tools

Ahead of Open Data Day on Saturday 2nd March 2019, Tomoaki Watanabe from our Japanese chapter Open Knowledge Japan shares more information on a tool they developed to facilitate dealing with intellectual property (IP) rights during hackathon events. This year, many Open Data Day events are planned in Japan: Open Knowledge Japan created an overview at https://odd.okfn.jp/. The main idea is simple: a tool to facilitate consensus formation on the intellectual property (IP) rights in the materials generated out of a hackathon. That is so that interested participants, or even others, know that they can develop a promising project further after the event is over. You would see many gems generated at a hackathon: interesting projects, promising products and services, inspiring ideas, sketches, mockups and so on. The excitement and the passion it produces may fade away over time. And sometimes, you may be disappointed by the fact that a seemingly promising project does not get anywhere, even when there are some people who want to develop it further. It is just that the one who wants to carry on cannot figure out if doing so is okay. Another problem related to the rights and licenses issue happens when a for profit company hosts a hackathon and claims all the IP rights – this may make some participants feel surprised and exploited. In Japan, a research project related to digital fabrication <http://coi.sfc.keio.ac.jp/> saw this problem and came up with a rather simple set of templates, called “a participation agreement,” <https://github.com/IAMAS/makeathon_agreement> to facilitate and document participants’ consensus on the rights issues around hackathons or makeathons. (Disclosure: I am currently a member of the research project, although I am not one of the members who developed it.) It has been developed with the help of a lawyer to ensure proper legal force. The main developer helped introduce them to well over a dozen hackathons and similar events, and a small study found that all private and public sector organizers knew about the tool (Kobayashi & Mizuno, 2016). The tool is published under an open license (CC-BY-SA 4.0), and meant to be customized by the organizer(s) of the event adopting the tool.

Legal hack group brainstorms solutions to stakeholder challenge by MC Legal Hackathon and Franklin Graves, CC0

How to Use and Major Features

The tool is a set of documents. The Participation Consent Form is like a terms of service for the event, so that the participants know, prior to the event, what happens to his rights. The Post-Event Confirmation Form is to be agreed after the event. The “default settings” of the documents are such that participants’ intellectual property rights stay with participants. The exception is when a participant becomes unreachable, in which case his rights are deemed waived. It means that, at least, participants can explore what to do with the unfinished project and negotiate over the terms. Additionally, it has a clause stating that contributing ideas alone would not result in any ground for an IP right. These settings are in the pre-participation document, and the post-participation document is for writing down an agreement among the contributing group members over their rights, permissions, terms, etc. so that they know what they can do with the output of their team. One important “default setting” there is the statement that those who do not participate into commercialization after the event will waive all their IP rights. Some organizers may think that some revenue sharing scheme or acknowledgement of contribution (attribution / crediting) should be adopted as a default. In order to customize the documents, a good starting point would be to change the very first part – from title, organizer’s name, and to Art. 1, which is about the purpose of the event. Another section to customize may be the Art. 2, where the IP rights are discussed. If a Creative Commons Attribution license (CC-BY) should be adopted as a baseline for copyrights in participants’ contributions, for example, this would be a good place to state that. Art. 2 of the post-event document has the “default setting” mentioned above on non-participating members of post-event commercialization waiving the IP rights. One additional thing to note is that because the tool was originally developed for Japanese context, they contain references to the Japanese copyright law. There is a repository at GitHub so that people can report suggestions, bugs or questions. Forking the tool is also welcome, according to a Japanese material by the developers. To recap, these documents (templates) are good for avoiding the avoidable rights-clearance issues arising out of collaborative events like a hackathon.   Reference: Kobayashi & Mizuno (2016). “Making rules regarding intellectual property rights at co-creation events such as hack-a-thons: proposal of a participation agreement.” Digital Practice, v.7, n.2, pp.128-135. (in Japanese)  

Visual gateways into science: Why it’s time to change the way we discover research

- November 14, 2017 in open knowledge maps, Open Science, Open Source, tools

Have you ever noticed that it is really hard to get an overview of a research field that you know nothing about? Let’s assume for a minute that a family member or a loved one of yours has fallen ill and unfortunately, the standard treatment isn’t working. Like many other people, you now want to get into the research on the illness to better understand what’s going on. You proceed to type the name of the disease into PubMed or Google Scholar – and you are confronted with thousands of results, more than you could ever read. It’s hard to determine where to start, because you don’t understand the terminology in the field, you don’t know what the main areas are, and it’s hard to identify important papers, journals, and authors just by looking at the results list. With time and patience you could probably get there. However, this is time that you do not have, because decisions need to be made. Decisions that may have grave implications for the patient. If you have ever had a similar experience, you are not alone. We are all swamped with the literature, and even experts struggle with this problem. In the Zika epidemic in 2015 for example, many people scrambled to get an overview of what was until then an obscure research topic. This included researchers, but also practitioners and public health officials. And it’s not just medicine; almost all areas of research have become so specialized that they’re almost impenetrable from the outside. But the thing is, there are many people on the outside that could benefit from scientific knowledge. Think about journalists, fact checkers, policy makers or students. They all have the same problem – they don’t have a way in. Reuse of scientific knowledge within academia is already limited, but when we’re looking at transfer to practice, the gap is even wider. Even in application-oriented disciplines, only a small percentage of research findings ever influence practice – and even if they do so, often with a considerable delay. At Open Knowledge Maps, a non-profit organization dedicated to improving the visibility of scientific knowledge for science and society, it is our mission to change that. We want to provide visual gateways into research – because we think that it is important that we do not only provide access to research findings, but also to enable discoverability of scientific knowledge. At the moment, there is a missing link between accessibility and discoverability – and we want to provide that link. Imagine a world, where you can get an overview of any research field at a glance, meaning you can easily determine the main areas and relevant concepts in the field. In addition, you can instantly identify a set of papers that are relevant for your information need. We call such overviews knowledge maps. You can find an example for the field of heart diseases below. The bubbles represent the main areas and relevant papers are already attached to each of the areas. Now imagine that each of these maps is adapted to the needs of different types of users, researchers, students, journalists or patients. And not only that: they are all structured and connected and they contain annotated pathways through the literature as to what to read first, and how to proceed afterwards. This is the vision that we’ve have been working on for the past 1.5 years as a growing community of designers, developers, communicators, advisors, partners, and users. On our website, we are offering an openly accessible service, which allows you to create a knowledge map for any discipline. Users can choose between two databases: Bielefeld Academic Search Engine (BASE) with more than 110 million scientific documents from all disciplines, and PubMed, the large biomedical database with 26 million references. We use the 100 most relevant results for a search term as reported by the respective database as a starting point for our knowledge maps. We use text similarity to create the knowledge maps. The algorithm groups those papers together that have many words in common. See below for an example map of digital education. We have received a lot of positive feedback on this service from the community. We are honored and humbled by hundreds of enthusiastic posts in blogs, and on Facebook and Twitter. The service has also been featured on the front pages of reddit and HackerNews, and recently, we won the Open Minds Award, the Austrian Open Source Award. Since the first launch of the service in May 2016, we have had more than 200,000 visits on Open Knowledge Maps. Currently, more than 20,000 users leverage Open Knowledge Maps for their research, work, and studies per month. The “Open” in Open Knowledge Maps does not only stand for open access articles – we want to go the whole way of open science and create a public good. This means that all of our software is developed open source. You can also find our development roadmap on Github and leave comments by opening an issue. The knowledge maps themselves are licensed under a Creative Commons Attribution license and can be freely shared and modified. We will also openly share the underlying data, for example as Linked Open Data. This way, we want to contribute to the open science ecosystem that our partners, including Open Knowledge Austria, rOpenSci, ContentMine, the Internet Archive Labs and Wikimedia are creating. Open Knowledge International has played a crucial role in incubating the idea of an open discovery platform, by way of a Panton Fellowship where the first prototype of the search service was created. Since then, the Open Knowledge Network has enthusiastically supported the project, in particular the Austrian chapter as well as Open Knowledge International, Open Knowledge Germany and other regional organisations. Members of the international Open Knowledge community have become indispensable for Open Knowledge Maps, be it as team members, advisors or active supporters. A big shout-out and thank you to you! As a next step, we want to work on structuring and connecting these maps – and we want to turn discovery into a collaborative process. Because someone has already gone that way before and they have all the overview and the insights. We want to enable people to communicate this knowledge so that we can start laying pathways through science for each other. We have created a short video to illustrate this idea:

Visual gateways into science: Why it’s time to change the way we discover research

- November 14, 2017 in open knowledge maps, Open Science, Open Source, tools

Have you ever noticed that it is really hard to get an overview of a research field that you know nothing about? Let’s assume for a minute that a family member or a loved one of yours has fallen ill and unfortunately, the standard treatment isn’t working. Like many other people, you now want to get into the research on the illness to better understand what’s going on. You proceed to type the name of the disease into PubMed or Google Scholar – and you are confronted with thousands of results, more than you could ever read. It’s hard to determine where to start, because you don’t understand the terminology in the field, you don’t know what the main areas are, and it’s hard to identify important papers, journals, and authors just by looking at the results list. With time and patience you could probably get there. However, this is time that you do not have, because decisions need to be made. Decisions that may have grave implications for the patient. If you have ever had a similar experience, you are not alone. We are all swamped with the literature, and even experts struggle with this problem. In the Zika epidemic in 2015 for example, many people scrambled to get an overview of what was until then an obscure research topic. This included researchers, but also practitioners and public health officials. And it’s not just medicine; almost all areas of research have become so specialized that they’re almost impenetrable from the outside. But the thing is, there are many people on the outside that could benefit from scientific knowledge. Think about journalists, fact checkers, policy makers or students. They all have the same problem – they don’t have a way in. Reuse of scientific knowledge within academia is already limited, but when we’re looking at transfer to practice, the gap is even wider. Even in application-oriented disciplines, only a small percentage of research findings ever influence practice – and even if they do so, often with a considerable delay. At Open Knowledge Maps, a non-profit organization dedicated to improving the visibility of scientific knowledge for science and society, it is our mission to change that. We want to provide visual gateways into research – because we think that it is important that we do not only provide access to research findings, but also to enable discoverability of scientific knowledge. At the moment, there is a missing link between accessibility and discoverability – and we want to provide that link. Imagine a world, where you can get an overview of any research field at a glance, meaning you can easily determine the main areas and relevant concepts in the field. In addition, you can instantly identify a set of papers that are relevant for your information need. We call such overviews knowledge maps. You can find an example for the field of heart diseases below. The bubbles represent the main areas and relevant papers are already attached to each of the areas. Now imagine that each of these maps is adapted to the needs of different types of users, researchers, students, journalists or patients. And not only that: they are all structured and connected and they contain annotated pathways through the literature as to what to read first, and how to proceed afterwards. This is the vision that we’ve have been working on for the past 1.5 years as a growing community of designers, developers, communicators, advisors, partners, and users. On our website, we are offering an openly accessible service, which allows you to create a knowledge map for any discipline. Users can choose between two databases: Bielefeld Academic Search Engine (BASE) with more than 110 million scientific documents from all disciplines, and PubMed, the large biomedical database with 26 million references. We use the 100 most relevant results for a search term as reported by the respective database as a starting point for our knowledge maps. We use text similarity to create the knowledge maps. The algorithm groups those papers together that have many words in common. See below for an example map of digital education. We have received a lot of positive feedback on this service from the community. We are honored and humbled by hundreds of enthusiastic posts in blogs, and on Facebook and Twitter. The service has also been featured on the front pages of reddit and HackerNews, and recently, we won the Open Minds Award, the Austrian Open Source Award. Since the first launch of the service in May 2016, we have had more than 200,000 visits on Open Knowledge Maps. Currently, more than 20,000 users leverage Open Knowledge Maps for their research, work, and studies per month. The “Open” in Open Knowledge Maps does not only stand for open access articles – we want to go the whole way of open science and create a public good. This means that all of our software is developed open source. You can also find our development roadmap on Github and leave comments by opening an issue. The knowledge maps themselves are licensed under a Creative Commons Attribution license and can be freely shared and modified. We will also openly share the underlying data, for example as Linked Open Data. This way, we want to contribute to the open science ecosystem that our partners, including Open Knowledge Austria, rOpenSci, ContentMine, the Internet Archive Labs and Wikimedia are creating. Open Knowledge International has played a crucial role in incubating the idea of an open discovery platform, by way of a Panton Fellowship where the first prototype of the search service was created. Since then, the Open Knowledge Network has enthusiastically supported the project, in particular the Austrian chapter as well as Open Knowledge International, Open Knowledge Germany and other regional organisations. Members of the international Open Knowledge community have become indispensable for Open Knowledge Maps, be it as team members, advisors or active supporters. A big shout-out and thank you to you! As a next step, we want to work on structuring and connecting these maps – and we want to turn discovery into a collaborative process. Because someone has already gone that way before and they have all the overview and the insights. We want to enable people to communicate this knowledge so that we can start laying pathways through science for each other. We have created a short video to illustrate this idea:

Presenting the Open Content Exchange Platform

- July 29, 2015 in eSpace, Featured, tools

Last year Open Knowledge joined the eSpace (Europeana Space) project to cooperate on the work for the Content Space, one of the spaces of possibility for the creative reuse of digital cultural content which this project is developing. Recently this Content Space went live, including the first version of the Open Content Exchange Platform, a resource […]

أساسيات جدول البيانات

- July 16, 2015 in Learn, tools

جداول البيانات مهمة لصحفيي البيانات وغيرهم ممن يريد التعامل مع جداول الأرقام وإجراء عمليات حسابية وإحصائية عليها بشكل سريع وقابل للمشاركة، في هذا الدليل نوضح أساسيات إستخدام جداول البيانات للمبتدئين

image02

رابط دليل جداول البيانات هنا

أساسيات جدول البيانات

- July 16, 2015 in Learn, tools

جداول البيانات مهمة لصحفيي البيانات وغيرهم ممن يريد التعامل مع جداول الأرقام وإجراء عمليات حسابية وإحصائية عليها بشكل سريع وقابل للمشاركة، في هذا الدليل نوضح أساسيات إستخدام جداول البيانات للمبتدئين

image02

رابط دليل جداول البيانات هنا

أساسيات جدول البيانات

- July 16, 2015 in Learn, tools

جداول البيانات مهمة لصحفيي البيانات وغيرهم ممن يريد التعامل مع جداول الأرقام وإجراء عمليات حسابية وإحصائية عليها بشكل سريع وقابل للمشاركة، في هذا الدليل نوضح أساسيات إستخدام جداول البيانات للمبتدئين

image02

رابط دليل جداول البيانات هنا

أساسيات جدول البيانات

- July 16, 2015 in Learn, tools

جداول البيانات مهمة لصحفيي البيانات وغيرهم ممن يريد التعامل مع جداول الأرقام وإجراء عمليات حسابية وإحصائية عليها بشكل سريع وقابل للمشاركة، في هذا الدليل نوضح أساسيات إستخدام جداول البيانات للمبتدئين

image02

رابط دليل جداول البيانات هنا

طريقة تصميم الخرائط الشجرية باستخدام Google Drive

- July 4, 2015 in DataViz, tools, treemaps