You are browsing the archive for godi.

What is the difference between budget, spending and procurement data?

Danny Lämmerhirt - May 18, 2017 in Global Open Data Index, godi, OpenSpending

Fiscal data is a complex topic. It comes in all different kind of formats and languages, its’ availability cannot be taken for granted and complexity around fiscal data needs special skills and knowledge to unlock and fully understand it. The Global Open Data Index (GODI) assesses three fiscal areas of national government: budgets, spending, and procurement. Repeatedly our team receives questions why some countries rank low in budgets, public procurement or spending, even though fiscal data is openly communicated. The quick answer: often we find information that is related to this data but does not exactly describe it in accordance with the described GODI data requirements. It appears to us that a clarification is needed between different fiscal data. This blogpost is dedicated to shed light on some of these questions. As part of our public dialogue phase, we also want to address our experts in the community. How should we continue to measure the status of these three key datasets in the future? Your input counts! Should we set the bar lower for GODI and avoid measuring transactional spending data at all? Is our assessment of transactional spending useful for you? You can leave us your feedback or join the discussion on this topic in our forum.

The different types of fiscal data

A government budget year produces different fiscal data types. Budgeting is the process where a government body sets its priorities as to how it intends to spend an amount of money over a specific time period (usually annually or semi-annually). Throughout the budgeting cycle  (the process of defining the budget), an initial budget can undergo revisions to result in a revised budget. Spending is the process of giving away money. This mean, the money might be given as a subsidy, a contract, refundable tax credit, pension or salary. Procurement is the process of selecting services from a supplier who fits best the need. That might involve selecting vendors, establishing payment terms, some strategic tender or other vetting mechanism meant to prevent corruption. Not only are the processes linked to each other, the data describing these processes can be linked too (e.g. in cases where identifiers exist linking spending to government budgets and public contracts). For laypersons, it might be difficult to tell the difference when they are confronted with a spending or procurement dataset: Is the money I see in a dataset spending, or part of contracting? The following paragraphs explain the differences.

Budget

As mentioned above, budgeting is called the process where a government body decides how to spend money over a certain time period. The amount is broken into smaller amounts (budget items) which can be classified as follows:
  • Administrative (which government sub-unit gets the money)
  • Functional (what the money is going to be used for)
  • Economic (how the money is going to be used, e.g., procurement, subsidies, salaries etc.)
  • Financing source (where the money should come from).
After the budget period ends, we know how much money was actually spent on each item – in theory. The Global Open Data Index assesses budget information at the highest administrative level (e.g. national government, federal government), which is broken down in one of these classifications. Here is an example of some fully open budget data of Argentina’s national government.

Example of Argentina’s national government budget 2017 (table shortened and cleaned)

The image shows the government entity, and expenditures split into economic classification (how the money is used). At the far right, we can see a column describing the total amount of money effectively spent on a planned budget expenditure. It basically compares allocated and paid money. This column must not be mixed with spending information on a transactional level (which displays each single transaction from a government unit to a recipient).

Spending

The Spending Data Handbook describes spending as “data relating to the specific expenditure of funds from the government”. Money might be given as a subsidy, as payment for a provided service, a salary (although salaries will seldom be published on a transactional level), a pension fund payment, a contract or a loan, to name just a few.
GODI focusses on transactions of service payments (often resulting from a prior procurement process). Monetary transactions are our baseline for spending data. GODI assesses the following information:
  • The amount that was transferred
  • The recipient (an entity external to the government unit)
  • When the transaction took place
  • Government office paying the transaction
  • Data split into individual transactions
GODI exclusively looks for single payment transfers. The reason why we are looking at this type of data is that spending patterns can be detected, and fraud or corruption uncovered. Some of the questions one might be able to address include: Who received what amount of money? Could government get its services from a cheaper service provider? Is government contracting to a cluster of related companies (supporting cartels)? GODI’s definition of spending data, even though ambitious in scope, does not consider the entire spectrum of transactional spending data. Being produced by many agencies, spending data is scattered  across different places online. We usually pick samples of specific spending data such as government payments to external suppliers (e.g. the single payments through a procurement process). Other types of payment, such as grants, loans or subsidies are then left aside. Our assessment is also ‘generous’ because we accept spending data that is only published above a certain threshold. The British Cabinet Office, a forerunner in disclosing spending data, only publishes data above £25,000. GODI accepts this as valid, even though we are aware that spending data below this amount remains opaque. There are also many more ways to expand GODI’s definition of spending data. For instance, we could ask if each transaction can be linked to a budget item or procurement contract so that we understand the spending context better.

Example image of British Spending data (Cabinet Office spending over £25,000)

Above is an example image of Great Britain’s Cabinet Office spending. You can see the date and the amount paid by government entity. Using the supplier name, we can track how much money was paid to the supplier. However, in this data no contract ID or contract name is provided that could allow to fully understand as part of what contracts these payments have been made.

Procurement

When purchasing goods and services from an external source, government units require a certain process for choosing the supplier who fits best the need. This process is called procurement and includes planning, tendering, awarding, contracting and implementation. Goals are to enable a fair competition among service providers and to prevent corruption. Many data traces enable to shed light on each procurement stage. For example one might want to understand from which budget a service is gonna be paid, or what amount of money has been awarded (with some negotiation possible) or finally contracted to a supplier. This blogpost by the Open Contracting Partnership illustrates how each of the procurement stages can be understood through different data. GODI focuses on two essential stages, that are considered to be a good proxy to understand procurement. These however do not display all information. Tender phase
  • Tenders per government office
  • Tender name
  • Tender description
  • Tender status
Award phase
  • Awards per government office
  • Award title
  • Award description
  • Value of the award
  • Supplier’s name
Any payment resulting out of government contracts with external suppliers (sometimes only one, sometimes more) has to  be captured in government spending. For example, there might a construction contractor that is being paid by milestone, or an office supplies dealer which is chosen as a supplier. Then each spending transaction is for a specific item purchased through a procurement process. Below you can see a procurement database of Thailand. It displays procurement phases, but does not display individual transactions following from these. This particular database does not represent actual spending data (monetary transactions), but preceding stages of the contracting process. Despite this the platform is misleadingly called “Thailand Government Spending”.

Procurement database in Thailand

Another example is a procurement database indicating how much money has been spent on a contract:

Example for the procurement website ‘Cuánto y a quién se contrató’ (Colombia)

The road ahead – how to measure spending data in the future

Overall, there is slow but steady progress around the openness of fiscal data. Increasingly, budget and procurement data is provided in machine-readable formats or openly licensed, sometimes presented on interactive government portals or as raw data (more detail see for example in the most recent blogpost of the Open Contracting Partnership around open procurement data). Yet, there is a long way to go for transactional spending data. Governments do first laudable steps by creating budget or procurement websites which demonstrate how much money will or has been spent in total. These may be confusingly named ‘spending’ portals because in fact they are linked to other government processes such as budgeting (e.g. how much money should be spent) or procurement (how much money has been decided to pay for an external service). The actual spending in form of single monetary transactions is missing. And to date there is no coherent standard or specification that would facilitate to document transactional spending. We want to address our experts in the community. How should we continue to measure the status of these three key datasets in the future? Your input counts!  You can leave us your feedback and discuss this topic in our forum.   This blog was jointly written by Danny Lämmerhirt and Diana Krebs (Project Manager for Fiscal Projects at Open Knowledge International)

How to Read the Global Open Data Index Results

Danny Lämmerhirt - May 2, 2017 in Global Open Data Index, godi, Open Government Data

The Global Open Data Index (GODI) is a tool to educate civil society and governments about open government data publication. We do so through presenting different information, including places scores, ranking, and scores for each data category per place, and comments by our submitters and reviewers. Even though we try to make this assessment coherent and transparent as possible, interpreting the results is not always straightforward. While open data has a very strict definition, scoring of any index is a discretional action. In real life, you can’t be partly open – either the data fit the criteria, or they do not.  This blog post will help GODI user to understand the following:  – What does the final score mean?  – How to interpret scores that vary between 0%, 40% or 70%?  – What does a score of 0% mean? For a more thorough explanation on how to read the results, go to index.okfn.org/interpretation/ 

What does the score mean?

Our scoring (ranging from 0% open to 100% open) does not necessarily show a gradual improvement towards open data. In fact, we assess very different degrees of data openness – which is why any score below 100 percent only indicates that a dataset is partially open. These levels of openness include public data, access-controlled data, as well as data gaps (See GODI methodology). To understand the differences we highly recommend reading each score together with our openness icon bar (see image below). For instance: a score of 70% can say that we found access-controlled, machine-readable data, that cannot be downloaded in bulk. Any score below 100% means “no access”, “closed access” or “public access”. Here we explain what each of them means, and how the data for each category look in practice.

Public Access Data

Data is publicly accessible if the public can see it online without any access controls. It does not imply that data can be downloaded, or that it is freely reusable. Often it means that data is presented in HTML on a website. The image above shows a search interface of a company register. It allows for targeted searches for individual companies but does not enable to retrieve all data at once. Individual search results (non-bulk) are displayed in HTML format and can then be downloaded as PDF (not machine-readable). Therefore the score is 70% and visualised as follow openness icon bar in our ranking:

Access-controlled data

Data is access-controlled if a provider regulates who, when, and how data can be accessed. Access control includes: * Registration/identification/authentication * Data request forms, data sharing agreement (stipulating use cases), * Ordering/purchasing data.   There are many reasons for establishing access controlled data including website traffic management, or to maintain control over how data is used. It is debatable whether some registration/authentication mechanisms reduce the openness of data (especially when registration is automated). Data request forms, on the other hand, are clearly not open data.   This image shows a data request form. The dataset is entirely hidden behind a “paywall”. This often prevents our research team from assessing the data at all. In this case, we could not verify in which format the data will be provided, and neither whether the data are actually weather forecast data (the particular category we look at). Therefore this access-controlled data gained 0 points and counts as 0% open. By contrast,  access-controlled data often score very high, up to 85% (because we subtract 15 out of 100 points for access-controls like registration requirements). 
 

How to read a score of 0%?

The are many reasons why datasets will score 0%. We tried to address the reasons in the reviewer or submitter’s comments as well. See here for the main reasons: Data gaps A data gap can mean that governments do not produce any data in a given category. Sometimes, if GODI shows a zero percent score, we see data gaps. For instance the case for Western African countries that lack air quality monitoring systems, or countries that have no established postcode system. Data gaps indicate that the government information systems are not ready to produce open data, sometimes because resources are missing, at times because it is not a priority of government.

Exist, but only to governmental use

Sometimes government has the data, but for many reasons choose not to open it to the public at all.  

Not granular

Since our criteria look for particular granularity, we considered all datasets that didn’t reach this granularity levels as not granular, and therefore they were regarded as not available. For example – Great Britain has published elections results, but not on poll station level, which is a crucial level to detect voter fraud. Therefore, while there is some data for UK elections, it is not at the right level and considered as non-existent.

 Do not fit our criteria

We are looking for particular datasets in GODI. When they don’t have all the characteristics we are looking for, we consider them as not available. For the full explanation on how to read the results see – index.okfn.org/interpretation/

Introducing the 4th Global Open Data Index – Advancing the State of Open Data Through Dialogue

Open Knowledge International - May 1, 2017 in Featured, Global Open Data Index, godi, Open Government Data

We are pleased to present the 4th edition of the Global Open Data Index (GODI), a global assessment of open government data publication. GODI compares national government in 94 places across the 15 key datasets that have been assessed by our community as the most useful for solving social challenges. For this edition, we received 1410 submitted of datasets, but only 10% of these are open according to the Open Definition. At Open Knowledge International (OKI), we believe it’s important to look further than just the numbers. GODI is not just a benchmark, it can be and should be used as a tool to improve open data publication and make data findable, useful and impactful. This is why we include a new phase, the dialogue phase for this edition. Measuring the publication of open government data is a constant challenge and while open data has a fixed definition, practices for data publishing that vary widely from one government to government, and even within each government department. The GODI community has faced a number of difficult questions that make assessment harder. ‘Which government agency publishes the data?’ or ‘Is the data published on a portal or a department web page?’ are just two examples. For this edition of the GODI, we gave attention to the development of the methodology to help us address some of these challenges in the assessment process. Not only did we incorporate feedback from the past GODIs, but we engaged in systematic and detailed in a consultation with the community. In addition, we had a more extensive review phase including an expanded quality assurance stage. We believe that these changes are necessary, and makes our assessment better than ever. In past years we published GODI as a snapshot in time and didn’t make any change to it after publication. Like any assessment tool, we are not perfect, and we did find ourselves publishing an errata section to accommodate errors. This year we want to be responsive to feedback from the community and government, and give both the opportunity to debate the results. We encourage every user to contest the results for a dataset by publishing a comment about it on our forum. We will be accepting feedback for the next 30 days, and on June 2nd we will re-evaluate the index, correct errors before closing it for submissions until the next edition. What are we looking to achieve from the dialogue stage?  
  1. Feedback from the full spectrum of stakeholders including government publishers to data users. This helps to improve future editions of the index.
  2. Help government to understand where the gaps are in their publication of data.
  3. Help citizens to find the data sets they are looking for.
  4. Help publishers of data to understand the difficulties of users accessing the data
Different challenges arise from open data publication today. Over the next couple of weeks, we will be publishing more of our insights and thoughts on these difficulties. Our main findings are:
  1. GODI helps to identify “data gaps.”
  2. Open data is not easy to find, and governments frequently don’t publish on their portals but across different government websites or split across many pages on one website.
  3. There is data online, but users find it difficult to access and work with
  4. Licenses remain an issue in open data publication
You can read the more about GODI findings on the insights page Learn how to read the results, and download the data for your own research Please take a look at the new edition of GODI and challenge us by letting us know what you think on the forum!  

Introducing the 4th Global Open Data Index – Advancing the State of Open Data Through Dialogue

Open Knowledge International - May 1, 2017 in Featured, Global Open Data Index, godi, Open Government Data

We are pleased to present the 4th edition of the Global Open Data Index (GODI), a global assessment of open government data publication. GODI compares national government in 94 places across the 15 key datasets that have been assessed by our community as the most useful for solving social challenges. For this edition, we received 1410 submitted of datasets, but only 10% of these are open according to the Open Definition. At Open Knowledge International (OKI), we believe it’s important to look further than just the numbers. GODI is not just a benchmark, it can be and should be used as a tool to improve open data publication and make data findable, useful and impactful. This is why we include a new phase, the dialogue phase for this edition. Measuring the publication of open government data is a constant challenge and while open data has a fixed definition, practices for data publishing that vary widely from one government to government, and even within each government department. The GODI community has faced a number of difficult questions that make assessment harder. ‘Which government agency publishes the data?’ or ‘Is the data published on a portal or a department web page?’ are just two examples. For this edition of the GODI, we gave attention to the development of the methodology to help us address some of these challenges in the assessment process. Not only did we incorporate feedback from the past GODIs, but we engaged in systematic and detailed in a consultation with the community. In addition, we had a more extensive review phase including an expanded quality assurance stage. We believe that these changes are necessary, and makes our assessment better than ever. In past years we published GODI as a snapshot in time and didn’t make any change to it after publication. Like any assessment tool, we are not perfect, and we did find ourselves publishing an errata section to accommodate errors. This year we want to be responsive to feedback from the community and government, and give both the opportunity to debate the results. We encourage every user to contest the results for a dataset by publishing a comment about it on our forum. We will be accepting feedback for the next 30 days, and on June 2nd we will re-evaluate the index, correct errors before closing it for submissions until the next edition. What are we looking to achieve from the dialogue stage?  
  1. Feedback from the full spectrum of stakeholders including government publishers to data users. This helps to improve future editions of the index.
  2. Help government to understand where the gaps are in their publication of data.
  3. Help citizens to find the data sets they are looking for.
  4. Help publishers of data to understand the difficulties of users accessing the data
Different challenges arise from open data publication today. Over the next couple of weeks, we will be publishing more of our insights and thoughts on these difficulties. Our main findings are:
  1. GODI helps to identify “data gaps.”
  2. Open data is not easy to find, and governments frequently don’t publish on their portals but across different government websites or split across many pages on one website.
  3. There is data online, but users find it difficult to access and work with
  4. Licenses remain an issue in open data publication
You can read the more about GODI findings on the insights page Learn how to read the results, and download the data for your own research Please take a look at the new edition of GODI and challenge us by letting us know what you think on the forum!  

Network update from OK Japan: Corporate transparency and taxpayers’ money ahead of the 2020 Tokyo Olympics

Open Knowledge Japan - January 19, 2017 in Chapters, Chapters updates, godi, japan, network, OK Japan, Open Spending

This blog post is part of our on-going Network series featuring updates from chapters across the Open Knowledge Network and was written by the Open Knowledge Japan team. The OK Japan chapter has been active in the open data space in activities such as the promotion of open data use and policy discussions. Since we formed the team in 2012, our members have been instrumental in promoting International Open Data Day in Japan and OpenSpending/ Where Does My Money Go. We published use cases and other notable developments in the space through our blog. Our members also took part in many different government boards, advised or worked with municipalities and departments on open data implementation. Below is some news about us and open data developments in Japan.

Transparency discussed

Late October, Open Knowledge Japan has co-organized, with OpenCorporates Japan an event discussing corporate ID and transparency issues, including the Panama Papers. The keynote talk was given by Chris Taggart, CEO and founder of OpenCorporates, who was visiting Tokyo that time.

okjapanChris Taggart and Japanese experts discussing transparency issues in Tokyo

Work meeting held for Global Open Data Census

We hosted an informal meeting inviting key government officials to work on the Global Open Data Census. The Census scores and Japan’s ranking have been discussed in the open data policy circle.

Relevance of open knowledge for Japan

Aside from what we did lately, there are recent news reports that make open knowledge issues very relevant in the country. Related to the 2020 Tokyo Olympics, we have been learning about many allegations of shady processes. For example, some large sums of tax money reported going from our government to an unnamed “consultant” so that Tokyo could become the host city for the 2020 Olympics. Tokyo has also been involved in other transparency issues – the governor resigned this year after criticisms related to his spending and lack of clear explanations on those, and was given a vote of no-confidence. The new governor uncovered additional problems with the ongoing project of relocating the Tsukiji market, the largest fish market in Tokyo, including potential underground water contamination.

tokyo_tower_special_lightup_invitation_for_2020_olympic_games_on_march_2013Image Credit: Tokyo Tower Special Lightup <Invitation for 2020 Olympic Games> (Shibakouen, Tokyo, Japan) (CC BY)

In the early part of 2017, we will be working towards International Open Data Day 2017. Japan has been one of the most active countries in terms of the number of localities participating in IODD in the past few years (with more than 60 cities participating in 2016!). Some of the issues we will be discussing through this and other occasions include the above-mentioned data plans that the national and prefectural governments will create, as well wider use of Open Spending Next that some of our members have started learning.

Network update from OK Japan: Corporate transparency and taxpayers’ money ahead of the 2020 Tokyo Olympics

Open Knowledge Japan - January 19, 2017 in Chapters, Chapters updates, godi, japan, network, OK Japan, Open Spending

This blog post is part of our on-going Network series featuring updates from chapters across the Open Knowledge Network and was written by the Open Knowledge Japan team. The OK Japan chapter has been active in the open data space in activities such as the promotion of open data use and policy discussions. Since we formed the team in 2012, our members have been instrumental in promoting International Open Data Day in Japan and OpenSpending/ Where Does My Money Go. We published use cases and other notable developments in the space through our blog. Our members also took part in many different government boards, advised or worked with municipalities and departments on open data implementation. Below is some news about us and open data developments in Japan.

Transparency discussed

Late October, Open Knowledge Japan has co-organized, with OpenCorporates Japan an event discussing corporate ID and transparency issues, including the Panama Papers. The keynote talk was given by Chris Taggart, CEO and founder of OpenCorporates, who was visiting Tokyo that time.

okjapanChris Taggart and Japanese experts discussing transparency issues in Tokyo

Work meeting held for Global Open Data Census

We hosted an informal meeting inviting key government officials to work on the Global Open Data Census. The Census scores and Japan’s ranking have been discussed in the open data policy circle.

Relevance of open knowledge for Japan

Aside from what we did lately, there are recent news reports that make open knowledge issues very relevant in the country. Related to the 2020 Tokyo Olympics, we have been learning about many allegations of shady processes. For example, some large sums of tax money reported going from our government to an unnamed “consultant” so that Tokyo could become the host city for the 2020 Olympics. Tokyo has also been involved in other transparency issues – the governor resigned this year after criticisms related to his spending and lack of clear explanations on those, and was given a vote of no-confidence. The new governor uncovered additional problems with the ongoing project of relocating the Tsukiji market, the largest fish market in Tokyo, including potential underground water contamination.

tokyo_tower_special_lightup_invitation_for_2020_olympic_games_on_march_2013Image Credit: by t-mizo Tokyo Tower Special Lightup <Invitation for 2020 Olympic Games> (Shibakouen, Tokyo, Japan) (CC BY 2.0)

In the early part of 2017, we will be working towards International Open Data Day 2017. Japan has been one of the most active countries in terms of the number of localities participating in IODD in the past few years (with more than 60 cities participating in 2016!). Some of the issues we will be discussing through this and other occasions include the above-mentioned data plans that the national and prefectural governments will create, as well wider use of Open Spending Next that some of our members have started learning.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

Brook Elgie - January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

Brook Elgie - January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.

Office of Company Registrar Nepal joined the momentum

Nikesh Balami - August 21, 2016 in Featured, godi, Index, nepal, OCR Nepal, Open Data, ranking

Sometimes small key points makes you happy and boost your determination. My recent visit to the Office of the Company Registrar Nepal website makes me feel that way. The reason for my happiness  with the website is pretty straightforward – I was happy because I found a page named “Data” in the site with whole bunch of data, and more importantly the data were in an Open Format. Although the momentum of Open Data is gaining momentum all over the country, these kinds of government initiative are very rare and countable. Kudos goes to Nepal Government, their partners and CSO for pushing this momentum forward: http://www.ocr.gov.np/index.php/en/data OCR Nepal   The Office of Company Registrar Nepal website now consists data regarding company registration from 2002 till 2072 BS and the data is available for everyone to download and reuse in CSV and XML format. They have embarked this OCR Open Government Data (OGD) initiative to increase access to company data and wants to create ecosystem of  OGD to increase usability of data and realize the benefits. As we (Open Knowledge Nepal) discuss lots of times regarding the roles played by central government of Nepal for adopting open data as a policy, and the commitments shown by them through some projects such as the Aid Management Platform, Election Data in our blogs, events, projects and researches. This initiative of Office of the Company Registrar Nepal can now be a perfect examples for others public bodies of Nepal. It will also help Nepal to improve its rankings in the global Open Data surveys like Global Open Data Index and Open Data Barometer. I still remember those disappointing moments when Company Register gain only 35% open in Global Open Data Index 2015, all because the data wasn’t available in bulk and wasn’t machine readable. Now slowly but steadily we can mark those changes and gain 100% open. GODI Nepal This is a good start of the long Open Data journey from Nepal Government. Still, we can do much better. This kinds of start from our Government also boost the enthusiasm and determination of all CSO who are working in the field of Open Data at Nepal.  

Office of Company Registrar Nepal joined the momentum

Nikesh Balami - August 21, 2016 in Featured, godi, Index, nepal, OCR Nepal, Open Data, ranking

Sometimes small key points makes you happy and boost your determination. My recent visit to the Office of the Company Registrar Nepal website makes me feel that way. The reason for my happiness  with the website is pretty straightforward – I was happy because I found a page named “Data” in the site with whole bunch of data, and more importantly the data were in an Open Format. Although the momentum of Open Data is gaining momentum all over the country, these kinds of government initiative are very rare and countable. Kudos goes to Nepal Government, their partners and CSO for pushing this momentum forward: http://www.ocr.gov.np/index.php/en/data OCR Nepal   The Office of Company Registrar Nepal website now consists data regarding company registration from 2002 till 2072 BS and the data is available for everyone to download and reuse in CSV and XML format. They have embarked this OCR Open Government Data (OGD) initiative to increase access to company data and wants to create ecosystem of  OGD to increase usability of data and realize the benefits. As we (Open Knowledge Nepal) discuss lots of times regarding the roles played by central government of Nepal for adopting open data as a policy, and the commitments shown by them through some projects such as the Aid Management Platform, Election Data in our blogs, events, projects and researches. This initiative of Office of the Company Registrar Nepal can now be a perfect examples for others public bodies of Nepal. It will also help Nepal to improve its rankings in the global Open Data surveys like Global Open Data Index and Open Data Barometer. I still remember those disappointing moments when Company Register gain only 35% open in Global Open Data Index 2015, all because the data wasn’t available in bulk and wasn’t machine readable. Now slowly but steadily we can mark those changes and gain 100% open. GODI Nepal This is a good start of the long Open Data journey from Nepal Government. Still, we can do much better. This kinds of start from our Government also boost the enthusiasm and determination of all CSO who are working in the field of Open Data at Nepal.