You are browsing the archive for open data survey.

What we learnt from Boundary hunting

- December 10, 2018 in #mysociety, Open Data Census, open data survey, open politics, politics

You may remember that in August this year, mySociety and Open Knowledge International launched a survey, looking for the sources of digital files that hold electoral boundaries… for every country in the world. Well, we are still looking! There is a good reason for this hunt: the files are integral for people who want to make online tools to help citizens contact their local politicians, who need to be able to match users to the right representative. From mySociety’s site TheyWorkForYou to Surfers against Sewage’s Plastic Free Parliament campaign, to Call your Rep in the US, all these tools required boundary data before they could be built.

Photo by Chase Clark on Unsplash

We know that finding this data openly licensed is still a real challenge for many countries, which is of course why we launched the survey. We encourage people to continue to submit links to the survey, and we would love if people experienced in electoral boundary data, could help by reviewing submissions: if you are able to offer a few hours of help, please email democracy@mysociety.org The EveryBoundary survey FAQs tell you everything you need to know about what to look for when boundary hunting. But we also wanted to share some top tips that we have learnt through our own experiences. Do
  • Start the search by looking at authoritative sources first: electoral commissions, national mapping agencies, national statistics bodies, government data portals.
  • Look for data formats (.shp, .geojson, kml etc), and not just a PDF.
  • Ask around if you can’t find the data: if a map is published digitally, then the data behind it exists somewhere!
Don’t
  • Confuse administrative boundaries with electoral boundaries — they can be the same, but they often aren’t (even when they share a name).
  • Assume boundaries stay the same — check for redistricting, and make sure your data is current.
If you get stuck
  • Electoral boundaries are normally defined in legislation; sometimes this takes the form of lists of the administrative subdivisions which make up the electoral districts. If you can get the boundaries for the subdivisions you can build up the electoral districts with this information.
  • Make FOI requests to get hold of the data.
  • If needed, escalate the matter. We have heard of groups writing to their representatives, explaining the need for the data . And don’t forget: building tools that strengthen democracy is a worthwhile cause.  
mySociety is asking people to share electoral boundary data as part of efforts to make information on every politician in the world freely available to all, and support the creation of a Democratic Commons.  Electoral boundary files are an essential part of the data infrastructure of a Democratic Commons. A directory of electoral boundary sources is a potential benefit to many people and organisations  — so let’s keep up the search!

Help us find the world’s electoral boundaries!

- August 7, 2018 in #mysociety, Open Data Census, open data survey, open politics, politics

mySociety and Open Knowledge International are looking for the digital files that hold electoral boundaries, for every country in the world — and you can help.  Yeah, we know — never let it be said we don’t know how to party. But seriously, there’s a very good reason for this request. When people make online tools to help citizens contact their local politicians, they need to be able to match users to the right representatives. So head on over to the Every Boundary survey and see how you can help — or read on for a bit more detail.

Image credit: Sam Poullain

Data for tools that empower citizens

If you’ve used mySociety’s sites TheyWorkForYou — or any of the other parliamentary monitoring sites we’ve helped others to run around the world — you’ll have seen this matching in action. Electoral boundary data is also integral in campaigning and political accountability,  from Surfers against Sewage’s ‘Plastic Free Parliament’ campaign, to Call your Rep in the US. These sites all work on the precept that while people may not know the names of all their representatives at every level — well, do you? — people do tend to know their own postcode or equivalent. Since postcodes fall within boundaries, once both those pieces of information are known, it’s simple to present the user with their correct constituency or representative. So the boundaries of electoral districts are an essential piece of the data needed for such online tools.  As part of mySociety’s commitment to the Democratic Commons project, we’d like to be able to provide a single place where anyone planning to run a politician-contacting site can find these boundary files easily.

And here’s why we need you

Electoral boundaries are the lines that demarcate where constituencies begin and end. In the old days, they’d have been painstakingly plotted on a paper map, possibly accessible to the common citizen only by appointment. These days, they tend to be available as digital files, available via the web. Big step forward, right? But, as with every other type of political data, the story is not quite so simple. There’s a great variety of organisations responsible for maintaining electoral boundary files across different countries, and as a result, there’s little standardisation in where and how they are published.

How you can help

We need the boundary files for 231 countries (or as we more accurately — but less intuitively — refer to them, ‘places’), and for each place we need the boundaries for constituencies at national, regional and city levels. So there’s plenty to collect. As we so often realise when running this sort of project, it’s far easier for many people to find a few files each than it would be for our small team to try to track them all down. And that, of course, is where you come in. Whether you’ve got knowledge of your own country’s boundary files and where to find them online, or you’re willing to spend a bit of time searching around, we’d be so grateful for your help. Fortunately, there’s a tool we can use to help collect these files — and we didn’t even have to make it ourselves! The Open Data Survey, first created by Open Knowledge International to assess and display just how much governmental information around the world is freely available as open data, has gone on to aid many projects as they collect data for their own campaigns and research. Now we’ve used this same tool to provide a place where you can let us know where to find that electoral boundary data we need. Start here  — and please feel free to get in touch if anything isn’t quite clear, or you have any general questions. You might want to check the FAQs first though! Thanks for your help — it will go on to improve citizen empowerment and politician accountability throughout the world. And that is not something everyone can say they’ve done.

New report: Governing by rankings – How the Global Open Data Index helps advance the open data agenda

- November 29, 2017 in Featured, Global Open Data Index, godi, GODI16, open data survey, research

This blogpost was jointly written by Danny Lämmerhirt and Mária Žuffová (University of Strathclyde). We are pleased to announce our latest report Governing by rankings – How the Global Open Data Index helps advance the open data agenda. The Global Open Data Index (GODI) is one of the largest worldwide assessments of how well governments publish open data, coordinated by Open Knowledge International since 2013. Over the years we observed how GODI is used to monitor open data publication. But to date, less was known how​ ​GODI​ ​may​ ​translate​ ​into​ ​open​ ​data​ ​policies​ ​and publication​. How does GODI mobilise support for open data? Which actors are mobilised? Which aspects of GODI are useful, and which are not? Our latest report provides insights to these questions.

Why does this research matter?

Global governance indices like GODI enjoy great popularity due to their capacity to count, calculate, and compare what is otherwise hardly comparable. A wealth of research – from science and technology studies to sociology of quantification and international policy – shows that the effects of governance indicators are complex (our report provides an extensive reading list). Different audiences can take up indices to different (unintended) ends. It is therefore paramount to trace the effects of governance indicators to inform their future design. The report argues that there are multiple ways of looking for ‘impacts’ depending on different audiences, and how they put GODI into practice. Does a comparative open data ranking like GODI help mobilise high-level policy commitments? Does it incentivise individual government agencies to adjust and improve the publication of open data? Does it open up spaces for discussion and deliberation between government and civil society? This thinking builds on an earlier report by Open Knowledge International arguing that indicators have different audiences, with different lived experiences, needs, and agendas. While any form of measurement needs to align with these needs to become actionable (which affects how the impact of indicators will take shape), it also needs to retain comparability.

Our findings

We used Argentina, United Kingdom and Ukraine as case studies to represent different degrees of open data publication, economic development and political set-up. Our report, drawing from a series of twelve interviews and document analysis, suggests that GODI drives change primarily from within government. We assume this finding is partly due to our limited sample size. While key actors in the government are easy to identify, as open data publication is often one of their job responsibilities,  further research is needed to identify more civil society actors and how they engage with GODI. Below we describe nine ways how GODI influences open data policy and publication.
  1. Getting international visibility and achieving progress in country rankings or generally high ranking may incentivise and maintain high-level political support for open data, despite non-comparability of results across years.  
  2. In the absence of open data legislation, GODI has been used by Argentinian government as a soft policy tool to pressure other government agencies to publish data.
  3. Government agencies tasked with implementing open data used GODI to reward and point out progress made by other agencies, but also flag blockages to high-level politicians.  
  4. GODI sets standards what datasets to publish and sets a baseline for improvement. Outcomes are debatable around categories where the central government does not have easy political levers to publish data.
  5. GODI may be confounded with broader commitments to open government and used as an argument to reduce investment in other aspects of open government agenda. In the past, some high-level politicians presented  high ranking in GODI as evidence of government transparency and obsoletion of other ways of providing government information.  
  6. This effect may possibly be exacerbated by superficial media coverage that reports on the ranking without engaging with broader information and transparency policies. An analysis of Google News results suggests that journalists tend to reproduce (mostly politicians’) misconceptions and confound a good ranking in GODI with a high degree of government transparency and openness.
  7. Our findings suggest that individuals and organisations working around transparency and anti-corruption make little use of GODI due to a lack of detail and a misalignment with their specialised work tasks. For instance, Transparency International Ukraine uses the Transparent Public Procurement Rating to evaluate the legal framework, aside from the publication of open data.
  8. On the other hand, academics show interest to GODI to develop new governance indicators. They also often use country scores as a proxy for measuring open data availability.
  9. GODI has a potential for use in data journalism. Data journalism trainers may use it as a source of government data during their trainings.  

What we learned and the road ahead

Our research suggests that governments in all analysed countries pay attention to GODI.  With a few exceptions, they use it mostly to support open data publication and pave the way for new open data policies. While this is a promising finding, it has important implications for GODI and its design. If GODI sets standards in open data publication, as some interviewees from the government suggest, it needs to make sure to represent different data demands in the assessment and to encourage the implementation of sound policies. The challenge is to support policy development, which is often a lengthy process as opposed to short-lived rank-seeking. Some interviewees suggested valuable avenues for GODI’s design. For instance, assessing progress in open data publication perpetually rather than once a year over a limited timespan would require a long-term commitment to open data publication and better opportunities for civic engagement, as it would prevent governments from updating datasets once a year before GODI’s deadline only. Another route forward is discussed in another recent research by OKI, highlighting the potential to adjust an open data index to align it more closely to specific needs of topical expert organisations. Beyond engaging via GODI, civil society and academia might also participate in the development of new data monitoring instruments such as the Open Data Survey, that are relevant for their mission.    

How do open data measurements help water advocates to advance their mission?

- November 23, 2017 in Global Open Data Index, godi, GODI16, open data survey, WASH, water quality

This blogpost was jointly written by Danny Lämmerhirt and Nisha Thompson (DataMeet). Since its creation, the open data community has been at the heart of the Global Open Data Index (GODI). By teaming up with expert civil society organisations we define key datasets that should be opened by government to align with civil society’s priorities. We assumed that GODI also teaches our community to become more literate about government institutions, regulatory systems and management procedures that create data in the first place – making GODI an engagement tool with government.

Tracing the publics of water data

Over the past few months we have reevaluated these assumptions. How do different members of civil society perceive the data assessed by GODI? Is the data usable to advance their mission? How can GODI be improved to accommodate and reflect the needs of civil society? How would we go about developing user-centric open data measurements and would it be worth to run more local and contextual assessments? As part of this user research, OKI and DataMeet (a community of data science and open data enthusiasts in India) teamed up to investigate the needs of civic organisations in the water, sanitation and health (WASH) sector. GODI assesses whether governments release information on water quality, that is pollution levels, per water source. In detail this means that we check whether water data is available at potentially a household level or  at each untreated public water source such as a lake or river. The research was conducted by DataMeet and supervised by OKI, and included interviews and workshops with fifteen different organisations. In this blogpost we share insights on how law firms, NGOs, academic institutions, funding and research organisations perceive the usefulness of GODI for their work. Our research focussed on the South Asian countries India, Pakistan, Nepal, and Bangladesh. All countries face similar issues with ensuring safe water to their populations because of an over-reliance on groundwater, geogenic pollutants like arsenic, and high pollutants from industry, urbanisation, farming, and poor sanitation.

According to the latest GODI results, openness of water quality data remains low worldwide.

What kinds of water data matter to organisations in the water sector?

Whilst all interviewed organisations have a stake in access to clean water for citizens, they have very different motivations to use water quality data. Governmental water quality data is needed to
  1. Monitor government activities and highlight general issues with water management (for advocacy groups).
  2. Create a baseline to compare against civil society data (for organisations implementing water management systems)
  3. Detect geographic target areas of under-provision as well as specific water management problems to guide investment choices (for funding agencies and decision-makers)
Each use case requires data with different quality. Some advocacy interviewees told us that government data, despite a potential poor reliability, is enough to make the case that water quality is severely affected across their country. In contrast, researchers have a need for data that is provided continuously and at short updating cycles. Such data may not be provided by government. Government data is seen as support for their background research, but not a primary source of information. Funders and other decision-makers use water quality data largely for monitoring and evaluation – mostly to make sure their money is being used and is impactful. They will sometimes use their own water quality data to make the point that government data is not adequate. Funders push for data collection at a project level not continuous monitoring which can lead to gaps in understanding. GODI’s definition of water quality data is output-oriented and of general usefulness. It enables finding the answer to whether the water that people can access is clean or not. Yet, organisations on the ground need other data – some of which is process-oriented – to understand how water management services are regulated and governed or what laboratory is tasked to collect data. A major issue for meaningful engagement with water-related data is the complexity of water management systems. In the context of South Asia, managing, tracking, and safeguarding water resources for use today and in the future is complex. Water management systems, from domestic to industrial to agricultural ones, are diverse and hard to examine and keep accountable. Where water is coming from, how much of it is being used and for what, and then how waste is being disposed of are all crucial questions to these systems. Yet there is very little data available to address all these questions.

How do organisations in the WASH sector perceive the GODI interface?

GODI has an obvious drawback for the interviewed organisations: transparency is not a goal for organisations working on the ground and does not in itself provoke an increase in access to safe water or environmental conservation. GODI measures the publication of water quality data, but is not seen to stimulate improved outcomes. It also does not interact with the corresponding government agency. One part of GODI’s theory of change is that civil society becomes literate about government institutions and can engage with government via the publication of government data. Our interviews suggest that our theory of change needs to be reconsidered or new survey tools need to be developed that can enhance engagement between civil society and government. Below we share some ideas for future scenarios.

Our learnings and the road ahead

Adding questions to GODI

Interviews show that GODI’s current definition of water quality data does not always align with the needs of organisations on the ground. If GODI wants to be useful to organisations in the WASH sector, new questions can be added to the survey and be used as a jumping off point for outreach to groups. Some examples include:
  1. Add a question regarding metadata and methodology documentation to capture quality and provenance water data, but also where we found and selected data.
  2. Add a question regarding who did the data collection government or partner organisation. This allows community members to trace the data producers and engage with them.
  3. Assess transparency of water reports. Reports should be considered since they are an important source of information for civil society.

Customising the Open Data Survey for regional and local assessments

Many interviewees showed an interest in assessing water quality data at the regional and hyperlocal level. DataMeet is planning to customise the Open Data Survey and to team up with local WASH organisations to develop and maintain a prototype for a regional assessment of water quality. India will be our test case since there is local data for the whole country available at varying degrees across states. This may include to also assess quality of data and access to metadata. Highest transparency would mean to have water data from each individual lab were the samples are sent. Another use case of the Open Data Survey would include to measure the transparency of water laboratories. Bringing more transparency and accountability to labs would be the most valuable for ground groups sending samples to labs across the country.

Map of high (> 30 mg/l) fluoride values from 2013–14. From: The Gonda Water Data story

Storytelling through data

Whilst some interviewees saw little use in governmental water quality data, its usefulness can be greatly enhanced when combined with other information. As discussed earlier, governmental water data gives snapshots and may provide baseline values that serve NGOs as rough orientation for their work. Data visualisations could present river and water basin quality and tell stories about the ecological and health effects. Behavior change is a big issue when adapting to sanitation and hygiene interventions. Water quality and health data can be combined to educate people. If you got sick, have you checked your water? Do you use a public toilet? Are you washing your hands? This type of narration does not require granular accurate data.

Comparing water quality standards

Different countries and organisations have different standards for what counts as high water pollution levels. Another project could assess how the needs of South Asian countries are being served by a comparing pollution levels with different standards. For instance, fluorosis is an issue in certain parts of India: not just from high fluoride levels but also because of poor nutrition in those areas. Should fluoride affected areas have lower permissible amounts in poorer countries? These questions could be used to make water quality data actionable to advocacy  groups.

The Open Data Survey: Measuring what matters to you

- November 21, 2017 in godi, Open Data Census, Open Data Index, open data survey

I once heard a brilliant government official say that in government you only measure what matters to you. This resonated with me back when I was a public servant and it makes even more sense now that I have participated over the last few years in the Global Open Data Index (GODI), one of Open Knowledge International (OKI)’s main projects. We developed GODI to address the question how much government information is published as open data. The index uses a league table that ranks countries from most to least open, based on their results for fifteen key datasets. In addition the survey compares the openness of key datasets worldwide, and lists which countries have, for example, the most open budgets, company registers or election data. But national open government data is only one aspect of the open data ecosystem. The Open Data Survey, the tool that powers GODI, allows to collect information on any aspect of open data.  Many organisations have repurposed the survey throughout the years to foreground information that these organisations find important, urgent or that help them reach their goals. In this blogpost I will highlight a few use cases of the Open Data Survey. In a follow-up post I will explain how you can start using the survey to measure what is important for you, whether with OKI hosting an instance for you or by deploying your own survey.

The Open Data Census

Our first example is the Open Data Census, a tool usually run by our local groups and chapters to understand how their local governments are performing in data publication. We have a record of about 40 different censuses assessing local and regional open data in many different countries. The census is the only tool to assess open data on a city level.

Example of a city-level census, comparing Argentinian cities with one another

Users of the Open Data Census include Open Knowledge Argentina, Open Knowledge Brazil and the Sunlight Foundation who assessed the openness of U.S. cities as part of its Open Cities programme. The Census results did not only highlight top performing cities in the United States, but also enabled Sunlight Foundation to do further policy analysis and understand why some cities perform better than others. Similar to the Global Open Data Index, the census measures the openness of around fifteen different datasets. But it is also fully customisable, allowing any organisation to assess various aspects of open data – from open data policies, through to publication of good quality data, or whether a local government engages with citizens to identify and publish the most relevant data.  

Code for America’s digital services survey

Code for America repurposed our Open Data Survey to assess the state of digital services in U.S. cities. As part of their Digital Front Door initiative, Code for America used a fork of the survey to assess if the government websites were “meeting Code for America’s criteria for good digital services, and prioritize opportunities for improvements”. With more than 40 cities assessed, this was probably one of the biggest alternative uses for the survey and a great example how to assess aspects such as design of websites (which is an important element for open data publication).

Assessing WASH resilience

Sheena Carmel Opulencia-Calub, a 2015 School of Data fellow, used the survey to produce a public local information resource centred around Water, Sanitation and Hygiene (WASH) needs, to guide policy-making in the Philippines and advance disaster risk preparedness of local authorities. The website is currently a proof-of-concept, designed for information managers from government and CSOs who take care of data gathering and sharing during emergency situations. The website also visualises the evolution of key indicators related to water and sanitation, helping local authorities and information managers to make better-informed decisions.

Interface of the WASH Resilience survey

It uses two types of league tables, including a ranking of WASH data quality and availability of the 225 cities and provinces of the Philippines. The contents are now outdated but this example shows how the Open Data Survey can be repurposed to not only assess the availability of open data in a specific sector, but also to link this assessment to follow-up actions with government.

Measure what matters to you

To conclude, the Open Data Survey is a versatile tool and can be used to rank, compare, and highlight very different aspects of (open) data. I hope abovementioned use cases sparked your interest and ideas how to use the Open Data Survey. Stay tuned – in a follow-up blogpost we will explain how to customise the survey in order to make it fit your needs.  

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

- January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

- January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.