You are browsing the archive for code.

Cryptography: or the History, Principles, and Practice of Cipher-Writing (1898)

- February 14, 2018 in code, Cryptography, cyphers, language

An exploration of covert communication through history by the artist, naturalist, antiquarian, and writer Frederick Edward Hulme.

Cryptography: or the History, Principles, and Practice of Cipher-Writing (1898)

- February 14, 2018 in code, Cryptography, cyphers, language

An exploration of covert communication through history by the artist, naturalist, antiquarian, and writer Frederick Edward Hulme.

Cryptography: or the History, Principles, and Practice of Cipher-Writing (1898)

- February 14, 2018 in code, Cryptography, cyphers, language

An exploration of covert communication through history by the artist, naturalist, antiquarian, and writer Frederick Edward Hulme.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

- January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

- January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.

Barnard’s Universal Criminal Cipher Code (1895)

- September 6, 2016 in cipher, code, encryption, police

A book of codes to help disguise internal police telegrams in what amounted to some kind of 19th-century version of the encrypted email.

Barnard’s Universal Criminal Cipher Code (1895)

- September 6, 2016 in cipher, code, encryption, police

A book of codes to help disguise internal police telegrams in what amounted to some kind of 19th-century version of the encrypted email.

Diplohack in Brussels – The first hack in the Council of the European Union

- April 5, 2016 in code, coder, coding, Featured, india, India Open Data Census, Open City Census, Open Knowledge, Open Knowledge India

For the first time in history, we can hack from inside the Council of the European Union building! Join us at #Diplohack in Brussels in the Council of the European Union on the 29-30 of April. logo-diplohack We invite everyone to take part, whether you’re a diplomat, developer, designer, citizen, student, journalist or activist. We will connect different profiles together in teams to use European data for good. The idea is that you create a prototype or MVP (minimum viable product) with this data in just 24 hours that is focused on transparency and decision-making. We will support you in any way possible, explain the data and help you get started. Diplohack, as the hackathon is called, forms part of the Dutch Presidency of the Council of the European Union transparency strategy. The Brussels diplohack will run for 24 hours straight and is part of the several Diplohacks across Europe. Those hackathons intend to make the EU more transparent. Tech developers, EU diplomats, journalists, citizen activists, social entrepreneurs, data experts and many more will join forces and think of transparency applications to make decision making in the EU searchable and understandable. Everybody interested in the EU data can enter the hackathon. The winners of the diplohack will be invited to compete in a European final in Amsterdam during the TransparencyCamp Europe Unconference. The Diplohack event is organised the Council of the European Union, the Dutch EU Presidency and Open Knowledge Belgium. Get your free ticket for the #Diplohack! The Diplohack will be preceded by the Webinar with EU data experts to explain more about the data. You can join even if you don’t participate in the Diplohack itself. Register here.
Check http://diplohack.brussels/ or the discuss forum thread more info on the programme and the Eventbrite page for more practical information.
pic_3

India Open Data Census and Other Updates

- June 5, 2014 in code, coder, coding, Featured, india, India Open Data Census, Open City Census, Open Knowledge, Open Knowledge India

The India City Open Data Census is an ongoing, crowd-sourced measure of the current state of access to a selected group of datasets in municipalities across India. Any community member contribute an assessment of these datasets in their municipality at any time. Census content will be peer-reviewed periodically by a volunteer team of Open Data Census Librarians led by Open Knowledge India.

The first step in making data actionable is to make sure the data is easily accessible. Many cities, whether they have an open data policy in place or not, have work to do in terms of making datasets open and available online. Do an evaluation of where your city stands on releasing our landscape of datasets openly and work with your municipal partners to come up with a plan for making all of them open and available in 2014.

Once you have an inventory together, work with your city partners to figure out what barriers stand in the way of making any missing datasets open and accessible online and discuss solutions to overcoming those barriers. Work with government to create a timeline tool and/or alerts for when data will be released. If your city is making all this data available, now is the time to start thinking about what questions can be answered or problems addressed with these datasets. Take a look at what other cities are doing with key datasets. Are there lessons to be learned for your city?

To take this project a step further, you can pick an issue area of particular concern to your city (crime or blight, for example) and do an inventory of all datasets related to that issue. Then work with issue-area experts from the community to determine what potential value those datasets might have for addressing the problem, or what datasets are missing that would be particularly valuable.

Present State

Presently, the census has been conducted for 7 major cities in India — New Delhi, Mumbai, Kolkata, Chennai, Bangalore, Hyderabad and Ahmedabad. The total number of existing datasets is 105. However, none of these is open and the Percentage of Open Dtatasets found is ZERO therefore. What this means is that our governments have a lot of work to do and a lot of improvement is to be made to realize the potential of Open Governance in India. In many cities, like Kolkata, Delhi and Bangalore and Mumbai, many of the existing datasets can be made open simply by presenting them in a machine-readable format. In other cities, the governments need to work harder, as the existing datasets for these cities lag behind on various parameters. In fact, many of the datasets DO NOT EVEN EXIST!

You can find more information regarding the census at: India City Open Data Census

Other Updates

We are planning to start a new platform for coders shortly. In this platform, anybody would be able to code freely to produce Open Source products. India has a substantial volume of young coders, who can benefit from such a platform. However, building such a platform would require a lot of work and the plans are still in nascent stage. Please look for updates on Open Knowledge India’s pages.

India Open Data Census and Other Updates

- June 5, 2014 in code, coder, coding, Featured, india, India Open Data Census, Open City Census, Open Knowledge, Open Knowledge India

The India City Open Data Census is an ongoing, crowd-sourced measure of the current state of access to a selected group of datasets in municipalities across India. Any community member contribute an assessment of these datasets in their municipality at any time. Census content will be peer-reviewed periodically by a volunteer team of Open Data Census Librarians led by Open Knowledge India.

The first step in making data actionable is to make sure the data is easily accessible. Many cities, whether they have an open data policy in place or not, have work to do in terms of making datasets open and available online. Do an evaluation of where your city stands on releasing our landscape of datasets openly and work with your municipal partners to come up with a plan for making all of them open and available in 2014.

Once you have an inventory together, work with your city partners to figure out what barriers stand in the way of making any missing datasets open and accessible online and discuss solutions to overcoming those barriers. Work with government to create a timeline tool and/or alerts for when data will be released. If your city is making all this data available, now is the time to start thinking about what questions can be answered or problems addressed with these datasets. Take a look at what other cities are doing with key datasets. Are there lessons to be learned for your city?

To take this project a step further, you can pick an issue area of particular concern to your city (crime or blight, for example) and do an inventory of all datasets related to that issue. Then work with issue-area experts from the community to determine what potential value those datasets might have for addressing the problem, or what datasets are missing that would be particularly valuable.

Present State

Presently, the census has been conducted for 7 major cities in India — New Delhi, Mumbai, Kolkata, Chennai, Bangalore, Hyderabad and Ahmedabad. The total number of existing datasets is 105. However, none of these is open and the Percentage of Open Dtatasets found is ZERO therefore. What this means is that our governments have a lot of work to do and a lot of improvement is to be made to realize the potential of Open Governance in India. In many cities, like Kolkata, Delhi and Bangalore and Mumbai, many of the existing datasets can be made open simply by presenting them in a machine-readable format. In other cities, the governments need to work harder, as the existing datasets for these cities lag behind on various parameters. In fact, many of the datasets DO NOT EVEN EXIST!

You can find more information regarding the census at: India City Open Data Census

Other Updates

We are planning to start a new platform for coders shortly. In this platform, anybody would be able to code freely to produce Open Source products. India has a substantial volume of young coders, who can benefit from such a platform. However, building such a platform would require a lot of work and the plans are still in nascent stage. Please look for updates on Open Knowledge India’s pages.