You are browsing the archive for Tryggvi Björgvinsson.

OKI Agile: Kanban – the dashboard of doing

- August 3, 2017 in agile, kanban, Our Work

This is the fourth in a series of blogs on how we are using the Agile methodology at Open Knowledge International. Originating from software development, the Agile manifesto describes a set of principles that prioritise agility in work processes: for example through continuous development, self-organised teams with frequent interactions and quick responses to change (http://agilemanifesto.org). In this blogging series we explore the different ways Agile can be used to work better in teams and to create more efficiency in how to deliver projects. Previous posts dealt with user storiesmethodologies and the use of scrum and sprints in open data: this time we go into Kanban.  Picking a methodology has to take into account both the team doing the work but also the project. Bigger teams call for a bigger methodology, and more critical projects call for more methodology density (higher ceremony or publicly visible correctness). Bigger methodology adds larger amount of cost to the project. So picking a big methodology, with high ceremony, for a small team and less critical projects is a waste of time, money and effort. Scrum is an example of a relatively big methodology with high ceremony (in the agile universe). There are daily standups, sprint backlogs and commitments, product owner, scrum masters etc. Always going for Scrum may make us look like we’re organised, but we might be organised in the wrong way.

Less can lead to more

The opposite of Scrum in the agile world is perhaps Kanban. It’s designed to not interfere in any way with how the team already works. It may even be an addition to Scrum. It’s a form of what has been called Just-In-Time delivery/development. In Scrum the team creates a sprint backlog of what the team commits to and what will be worked on during the sprint. During the sprint, which typically ranges from 2-4 weeks, nobody can touch the backlog: nothing can be added, nothing can be modified. Items in the sprint backlog can only be closed. This is done to allow the team to focus on delivery, not requirements discussions. Kanban works on a different level. In Japanese Kan stands for card and ban means for signal. Kanban could therefore be translated as Card signals and that comes pretty close because Kanban is about using cards or some equivalent of cards to signal progress. It’s a card dashboard showing what’s being done where. There is no sprint that denotes work units, it’s just continuous work and planning when needed, or just in time.

Spreadsheet planning

In its essence, Kanban is progress visualised in a structured spreadsheet. You have columns that show different stages of progress or work. The different stages can include:
  • Incoming work
  • Brainstorming/Specifications
  • Implementation/development
  • Ready for review
  • In tests/proofreading
  • Ready for deployment
  • Deployed
This is not a be all, end all list. This is an example of what columns might be included. Kanban doesn’t tell us what columns we should use. Kanban is designed to be an addition on top of our current processes. The rows in our spreadsheet is used to track different aspects of the project. For example, if we have different members of the team working in different areas: Designers, content writers, software developers, a row would represent each area. This could also be used to track features of the project if the team is homogeneous (all team members work together in the same area, e.g. all software developers). The cells of the spreadsheet will then include requirements, for example user stories, that are in each stage for each feature or team. If we remember the requirements iceberg, this this is the 1-2 day tasks that we perform.

Visualising progress

As work progresses on each requirement item, it moves between the different columns. This creates the Kanban dashboard, visualising progress of the project. Kanban encourages autonomy and leadership on all levels. Every team member can come in each day, have a look at what’s needed and just do some work and then once done, move it between columns. That’s part of the just-in-time attitude. We don’t need to plan everything in beforehand, things will be done just in time. That means for managers, who normally push schedules for work to team members, they now have to sit back and give the team autonomy over the process. Likewise, team members who are used to being told what to do, now have to take matters into their own hands and pull the work that needs to be done. What about prioritisation you may ask? What about team members just picking low-hanging fruit or easy tasks and leave all the boring tasks until last even if they are the highest priority? Kanban puts limits on each column. How many is up to the team, but it is not allowed to have more than a prefixed amount of requirements in each column at any given moment. So there might never be more than 8 incoming work items, 2 items being drafted, 3 being implemented etc. The only exception to this can be the last column, delivered which collects everything that has been done, but is usually not a part of the Kanban dashboard or if it is, then it’s regularly purged. So if the team wants to implement a drafted feature but the amount of items in the implementation column has already reached the maximum they need to sort that out first, before they can take on more implementation work. The project manager or client representative will typically manage the incoming work column but that one is also limited by the maximum amount and progress. Common sense should of course be applied to these limitations. If the team is heterogeneous, i.e. each row in the “pretend spreadsheet” is a different area instead of a different feature, the limitations apply to each of the cells in the column (e.g. we can’t have the content team limit the software developers just because they have more to do).

Principles

Kanban is really simple, it’s all about visualising the work flow and not overwhelming the team. It’s built around four principles:
  • It should work with whatever process is currently used by the team
  • It helps the team make small incremental changes towards improvement
  • It respects roles and responsibilities of each team member
  • To each her/his own, team members are autonomous and leaders over their own work
It should be really simple to pick up Kanban and add it on top of whatever you’re doing today. Kanban works really well for whiteboards but the biggest obstacle for a remote organisation is finding a good digital tool for the workflow visualisation; Trello might work if if you don’t care too much about the rows. Google spreadsheet might work if you don’t care about the extra effort of moving things around. If you find a good way to manage a Kanban board, please share it through the comments below or on our forum!  

OKI Agile: Picking/Designing a Methodology

- May 22, 2017 in agile, Our Work

This is the seond in a series of blogs on how we are using the Agile methodology at Open Knowledge International. Originating from software development, the Agile manifesto describes a set of principles that prioritise agility in work processes: for example through continuous development, self-organised teams with frequent interactions and quick responses to change (http://agilemanifesto.org). In this blogging series we go into the different ways Agile can be used to work better in teams and to create more efficiency in how to deliver projects. The first post dealt with user stories: this time we go into methodologies. More efficiency in project delivery is the name of the game. Working together in teams or with other people requires us to put in places methods and methodologies to accomplish that. A shared set of methods that allows team members to walk into a project and start delivering as soon as possible.

Glossary

  • Method – A systematic procedure.
  • Methodology – A series of related methods or techniques.
  • Methodology size – Number of methods/techniques used in the methodology.
  • Methodology density – Amount of precision/checkpoints needed. More density is a higher ceremony methodology (more formal one)
  • Criticality – The nature of damage of undetected defects (impact if we forget something). Higher criticality means worse impact.

What is a methodology

A methodology consists of 10 elements, of which one, the team values, permeates all of them. Let’s first put them in a pretty picture and then list them out with descriptions:
  • Team values – What the team strives for, how they’d like to communicate and work together. The values affect each element of the methodology so different values create different methodologies.
  • Roles – It’s best to think of this as the job descriptions we’d put into the ads when you need to hire more staff.
  • Skills – The skills needed for the roles we need.
  • Team – This is the group of people that will tackle a project and what roles they have.
  • Tools – The tools people use either within a technique/method or to produce a deliverable according to the standard.
  • Techniques – The methods used to get stuff done (generate work product), this can be everything from work changing techniques like breaking up into sprints (time blocks for work), to games played to achieve a specific output like planning poker (for estimates), to just a description of what is done to make things happen like “write a blog post”.
  • Activities – The meetings, reviews, milestones and other things people do or attend. The most obvious activity is “create deliverable” or something like that, but the more interesting activities are the events that take place.
  • Standards – Description of what is permitted and not permitted in the work product. These can be standards such as what programming language or security measures to use, how management is handled or how decisions get made (e.g. RASCI) and other project conventions.
  • Work Products – Not only the final product, this is also the internal products, i.e. what each person or team hands over to another person or team, something like user stories or mockups.
  • Quality – Often not considered explicitly but these are the rules and concerns that need to be tracked for each deliverable (work product). This could be a part of activities but it’s so important that it’s better to split it out.

Principles for picking/designing methodologies

There is no one size fits all methodology. There are methods one reuses between them (the shared set of methods/techniques people know about) and probably a default set one uses in absence of something more fitting. However, in general one needs to think about and pick/design methodologies based on two things:
  1. The project itself
  2. The number of people on the project team
The nature of the project calls for different methodologies, some projects can get away with a very lightweight methodology, where it won’t be the end of the world if something is forgotten, while others call for more publicly visible correctness where bad things happen if things are forgotten. For example, paying salaries requires a methodology with more visible checkpoints and correctness than responding to Tweets. People might lose their houses if they don’t get paid, but nobody will be out on the street for missing a possibility to retweet. A lot changes based on the number of people involved. Communications become harder to do and the effectiveness of individuals decreases as a result the methodology must get bigger to tackle all of these: Picking/designing a methodology has to be based on these four principles (they have to be kept in mind so because they aren’t all achievable always, notably number 4 in our case):
  1. Bigger teams call for a bigger methodology
  2. More critical projects call for more methodology density (publicly visible correctness)
  3. Cost comes with weight (a small increase in methodology adds large amount of cost of the project)
  4. The most effective communication is face to face and interactive (everyone participates instead of just doing a broadcast).

Agile development

We have agreed on adopting the agile values (slightly adapted from the agile software values) as our team values where we can. That means we value:
  • Individuals and interactions over processes and tools
  • Working stuff over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan
We value everything that’s mentioned above, we just value the things on the left (the bold) more than the things on the right. There are also 12 principles we try to follow as much as we can:
  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable things.
  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  3. Deliver working stuff frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  4. Business people and implementers must work together daily throughout the project.
  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  6. The most efficient and effective method of conveying information to and within a team is face-to-face conversation.
  7. Working stuff is the primary measure of progress.
  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  9. Continuous attention to excellence and good design enhances agility.
  10. Simplicity–the art of maximizing the amount of work not done–is essential.
  11. The best architectures, requirements, and designs emerge from self-organizing teams.
  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
As an encouragement and because you were just bombarded with a long list of things and a discussion about planning how we plan work, here’s something from XKCD to keep in mind as we dive into methodologies:

OKI Agile: How to create and manage user stories

- April 26, 2017 in agile, Our Work

This is the first in a series of blogs on how we are using the Agile methodology at Open Knowledge International. Originating from software development, the Agile manifesto describes a set of principles that prioritise agility in work processes: for example through continuous development, self-organised teams with frequent interactions and quick responses to change (http://agilemanifesto.org). In this blogging series we go into the different ways Agile can be used to work better in teams and to create more efficiency in how to deliver projects. The first blog is dedicated to user stories, a popular agile technique.  User stories are a pretty nifty way of gathering requirements in an agile environment, where one of the key values is responding to change over following a plan. They are a good anchor for conversation that can then take place at the right time.

What is a user story?

A user story is a short sentence that encapsulates three things:
  1. Who?
  2. What?
  3. Why?
Notice that this does not include “How?” The “How?” is left to the team delivering the requirement. After all, the team consists of the professionals. They know how to deliver the best solution.  The most common way to encapsulate a user story is to use the template:
  • As a [WHO] I want [WHAT] so that [WHY]
Be careful not to sneak in any Hows into that template. That usually happens in the What so stay focussed! Words like by, using or with should be avoided like the plague because they usually result in a HowBasically avoid anything that has to do with the actual implementation.

Bad user stories

    • As a government official I want a Range Rover so that I can get from A to B quickly
  • Problem: A Range Rover is an actual implementation, it might not be what is needed even though it is what’s believed to be desired.
      • As a visitor to a website I want to be able to read a landing page using my spiffy MacBook Air and have the content presented in the Lato typeface, size 14, and with good white space between paragraphs so that I can know what the website is all about
    • Problem: A whole lot! What about GNU/Linux and Windows users? What if there is a better typeface out there? What about language of the content? The Why isn’t really a why. The list goes on. Don’t go into detail. It’s bad practice and creates more problems than it solves.

    Good user stories

    • As a government official I want means of transportation so that I can get from A to B quickly
    • As a website visitor I want to know what the website is about so that I can see how it can help me

    Why shouldn’t we go into details?

    It’s really quite simple. We expect the requirements to change and we’d just waste a lot of time going into the details of something that might change or get thrown out. We’re trying to be efficient while still giving the team an understanding of the broader picture. An extreme example would be that between project start and time when the team is going to tackle a user story the world might have moved to virtual governments that don’t need transportation any more (technology moves fast). The team also consists of experts so they know what works best (if not, why are they tasked to deliver?). The customers are the domain experts so they know best what is needed. In the website visitor example above, the team would know the best way of showing what a website is about (could be a landing page) but the customer knows what the customer is going to offer through the website and how they help people. We also value interactions and individuals over processes and tools. In an ever changing requirements environment we want non-details which can when the time comes be the basis for a conversation about the actual implementation. The team familiarises itself with the requirement at the appropriate time. So when starting work on the transportation user story, the team might discuss with the customer and ask questions like:
    • “How fast is quickly?”,
    • “Are A and B in the same city, country, on Earth?”,
    • “Are there any policies we need to be aware of?” etc.

    Acceptance of user stories

    Surely the customer would still want to be able to have a say in how things get implemented. That’s where acceptance criteria comes in. The customer would create a checklist for each user story when the time comes in a joint meeting, based on discussion. That’s the key thing. It comes out of a discussion. This criteria tells the team in a bit more detail what they need to fulfill to deliver the requirement (user story). For the government in need of transport this might be things like:
    • Main area of interest/focus is London area
    • Applicable to/usable in other cities as well
    • Allows preparations for a meeting while in transit
    • Very predictable so travel time can be planned in detail
    • Doesn’t create a distance between me and the people I serve
    Then the implementation team might even pick public transportation to solve this requirement. A Range Rover wasn’t really needed in the end (albeit this would probably go against the “satisfy the customer” principle but hey! I’m teaching you about user stories here! Stay focussed!).

    How is this managed?

    One key thing we want to get out of user stories is to not scope the requirement in detail until it becomes clear that it’s definitely going to be implemented. How then do you know what you’ll be doing in the future? User stories can be of different sizes; From very coarse to detailed stories. The very coarse ones don’t even need to be written as user stories. They’re often referred to as epics. Many break requirements into three stages. The releases or the projects or whatever the team works on. Then each of these can be broken up into features and each feature can be broken up into tasks. It’s up to the team to decide when it’s best to formulate these as user stories and it really depends on the team and the project. Some might have epics as the big long term project, break that up into user stories, and then break each user story up into tasks. Others might have a single product, with the releases (what you want to achieve in each release: “The geospatial release”) at the top and then have features as sentences (epics) underneath the release and then transform the sentences into user stories you work on. Whatever way you do, this is the general guideline of granularity:
    • Coarsest: Long-term plans of what you’ll be doing
    • Mid-range: Delivery in a given time period (e.g. before deadlines)
    • Finest: What team will deliver in a day or two
    The reason the finest level is in a day or two is to give the team a sense of progress and avoid getting stuck at: “I’m still doing the guildhall thing” which is very demoralizing and inefficient (and not really helpful for others who might be able to help). There is a notion of the requirements iceberg or pyramid which tries to visualise the three stages. The bottom stage is larger and bigger items (the coarse stuff), mid range is what you’re delivering in a time period, and the finest is the smallest blocks of work. That’s what’s going to be “above” surface for the core team. That’s still just a fraction of the big picture.


    When should who be involved?

    So the core team has to decide at what stage of the iceberg they want to write the user stories, and that kind of depends on the project, the customer, and the customer’s involvement. So we need to better understand “the team”. The core team should always be present and work together. Who is in the core team then? If that’s not clear, there’s a story/joke, about the pig and the chicken, that can guide us: A pig and a chicken decided to open up a restaurant. They were discussing what name to give the restaurant when the chicken proposed the name: Ham & Eggs. The pig sneered its nose and said: “That’s unfair, I’d be committed but you’d only be involved!” That’s the critical distinction between the core team and others. The core team is the pigs. Everyone else who is only involved to make the project happen is a chicken. The pigs run the show. The chickens are there to make sure the pigs can deliver. Chickens come in various sizes and shapes. It can be team managers (planning persons), unit heads, project managers, biz-dev people, and even customers. The term customer is pretty vague. You usually don’t have all your customers involved. Usually you only have a single representative. For bespoke/custom development (work done at the request of someone else), that person is usually the contact person for the client you’re working for. At other times the single customer representative is an internal person. That internal individual is sometimes referred to as the product owner (comes from Scrum) and is a dedicated role put in place when there is no single customer, e.g. the product is being developed in-house. That person then represents all customers and has in-depth knowledge about all customers or has access to a focus group or something. This individual representative is the contact point for the team. The one who’s there for the team to help them deliver the right thing. More specifically this individual:
    • Creates initial user stories (and drives creation of other user stories)
    • Helps the team prioritise requirements (user stories)
    • Accepts stories (or rejects) when the team delivers
    • Is available to answer any questions the team might have
    So the representative’s role is to provide the implementers with enough domain knowledge to proceed and deliver the right thing. This individual should not have any say in how the core team will implement it. That’s why the team was hired/tasked with delivering it, because they know how to do it. That’s also why user stories do not focus on the how. The core team, the pigs, need to decide at what intersections in the iceberg they want to have this representative present (where discussions between the core team and the representative will happen). When they go from coarsest to mid-range or from mid-range to finest. So in a weird sense, the core team decides when the customer representative decides what will be done. As a rule of thumb: the user stories feed into the stage above the intersection where representative is present. So if the representative helps the team go from coarse to mid-range, the user stories are created for the mid-range stage. If the representative is there for mid-range to finest, the user stories are going to be very fine-grained. As a side note, because the chickens are there to make sure the pigs can deliver, they will always have to be available to answer questions. Many have picked up the standup activity from the Scrum process to discuss blockers, and in those cases it’s important that everyone involved, both pigs and chickens, is there so the chickens can act quickly to unblock the pigs. Now go and have fun with user stories. They shouldn’t be a burden. They should make your life easier… or at least help you talk to chickens.

    Beauty behind the scenes

    - August 5, 2015 in ckan, OKF Sweden, Open Data, Open Knowledge

    Good things can often go unnoticed, especially if they’re not immediately visible. Last month the government of Sweden, through Vinnova, released a revamped version of their open data portal, Öppnadata.se. The portal still runs on CKAN, the open data management system. It even has the same visual feeling but the principles behind the portal are completely different. The main idea behind the new version of Öppnadata.se is automation. Open Knowledge teamed up with the Swedish company Metasolutions to build and deliver an automated open data portal.

    Responsive design

    In modern web development, one aspect of website automation called responsive design has become very popular. With this technique the website automatically adjusts the presentation depending on the screen size. That is, it knows how best to present the content given different screen sizes. Öppnadata.se got a slight facelift in terms of tweaks to its appearance, but the big news on that front is that it now has a responsive design. The portal looks different if you access it on mobile phones or if you visit it on desktops, but the content is still the same. These changes were contributed to CKAN. They are now a part of the CKAN core web application as of version 2.3. This means everyone can now have responsive data portals as long as they use a recent version of CKAN.
    New Öppnadata.se

    New Öppnadata.se

    Old Öppnadata.se

    Old Öppnadata.se

    Data catalogs

    Perhaps the biggest innovation of Öppnadata.se is how the automation process works for adding new datasets to the catalog. Normally with CKAN, data publishers log in and create or update their datasets on the CKAN site. CKAN has for a long time also supported something called harvesting, where an instance of CKAN goes out and fetches new datasets and makes them available. That’s a form of automation, but it’s dependent on specific software being used or special harvesters for each source. So harvesting from one CKAN instance to another is simple. Harvesting from a specific geospatial data source is simple. Automatically harvesting from something you don’t know and doesn’t exist yet is hard. That’s the reality which Öppnadata.se faces. Only a minority of public organisations and municipalities in Sweden publish open data at the moment. So a decision hasn’t been made by a majority of the public entities for what software or solution will be used to publish open data. To tackle this problem, Öppnadata.se relies on an open standard from the World Wide Web Consortium called DCAT (Data Catalog Vocabulary). The open standard describes how to publish a list of datasets and it allows Swedish public bodies to pick whatever solution they like to publish datasets, as long as one of its outputs conforms with DCAT. Öppnadata.se actually uses a DCAT application profile which was specially created for Sweden by Metasolutions and defines in more detail what to expect, for example that Öppnadata.se expects to find dataset classifications according the Eurovoc classification system. Thanks to this effort significant improvements have been made to CKAN’s support for RDF and DCAT. They include application profiles (like the Swedish one) for harvesting and exposing DCAT metadata in different formats. So a CKAN instance can now automatically harvest datasets from a range of DCAT sources, which is exactly what Öppnadata.se does. For Öppnadata.se, the CKAN support also makes it easy for Swedish public bodies who use CKAN to automatically expose their datasets correctly so that they can be automatically harvested by Öppnadata.se. For more information have a look at the CKAN DCAT extension documentation.

    Dead or alive

    The Web is decentralised and always changing. A link to a webpage that worked yesterday might not work today because the page was moved. When automatically adding external links, for example, links to resources for a dataset, you run into the risk of adding links to resources that no longer exist. To counter that Öppnadata.se uses a CKAN extension called Dead or alive. It may not be the best name, but that’s what it does. It checks if a link is dead or alive. The checking itself is performed by an external service called deadoralive. The extension just serves a set of links that the external service decides to check to see if some links are alive. In this way dead links are automatically marked as broken and system administrators of Öppnadata.se can find problematic public bodies and notify them that they need to update their DCAT catalog (this is not automatic because nobody likes spam). These are only the automation highlights of the new Öppnadata.se. Other changes were made that have little to do with automation but are still not immediately visible, so a lot of Öppnadata.se’s beauty happens behind the scenes. That’s also the case for other open data portals. You might just visit your open data portal to get some open data, but you might not realise the amount of effort and coordination it takes to get that data to you. Image of Swedish flag by Allie_Caulfield on Flickr (cc-by) This post has been republished from the CKAN blog.

    Beauty behind the scenes

    - August 5, 2015 in automation, broken links, dcat, Deployments, Extensions, Featured, harvester, öppnadata.se, responsive design, Sweden

    Good things can often go unnoticed, especially if they’re not immediately visible. Last month the government of Sweden, through Vinnova, released a revamped version of their open data portal, Öppnadata.se. The portal still runs on CKAN, the open data management system. It even has the same visual feeling but the principles behind the portal are completely different. The main idea behind the new version of Öppnadata.se is automation. Open Knowledge teamed up with the Swedish company Metasolutions to build and deliver an automated open data portal.

    Responsive design

    In modern web development, one aspect of website automation called responsive design has become very popular. With this technique the website automatically adjusts the presentation depending on the screen size. That is, it knows how best to present the content given different screen sizes. Öppnadata.se got a slight facelift in terms of tweaks to its appearance, but the big news on that front is that it now has a responsive design. The portal looks different if you access it on mobile phones or if you visit it on desktops, but the content is still the same. These changes were contributed to CKAN. They are now a part of the CKAN core web application as of version 2.3. This means everyone can now have responsive data portals as long as they use a recent version of CKAN.
    New Öppnadata.se

    New Öppnadata.se

    Old Öppnadata.se

    Old Öppnadata.se

    Data catalogs

    Perhaps the biggest innovation of Öppnadata.se is how the automation process works for adding new datasets to the catalog. Normally with CKAN, data publishers log in and create or update their datasets on the CKAN site. CKAN has for a long time also supported something called harvesting, where an instance of CKAN goes out and fetches new datasets and makes them available. That’s a form of automation, but it’s dependent on specific software being used or special harvesters for each source. So harvesting from one CKAN instance to another is simple. Harvesting from a specific geospatial data source is simple. Automatically harvesting from something you don’t know and doesn’t exist yet is hard. That’s the reality which Öppnadata.se faces. Only a minority of public organisations and municipalities in Sweden publish open data at the moment. So a decision hasn’t been made by a majority of the public entities for what software or solution will be used to publish open data. To tackle this problem, Öppnadata.se relies on an open standard from the World Wide Web Consortium called DCAT (Data Catalog Vocabulary). The open standard describes how to publish a list of datasets and it allows Swedish public bodies to pick whatever solution they like to publish datasets, as long as one of its outputs conforms with DCAT. Öppnadata.se actually uses a DCAT application profile which was specially created for Sweden by Metasolutions and defines in more detail what to expect, for example that Öppnadata.se expects to find dataset classifications according the Eurovoc classification system. Thanks to this effort significant improvements have been made to CKAN’s support for RDF and DCAT. They include application profiles (like the Swedish one) for harvesting and exposing DCAT metadata in different formats. So a CKAN instance can now automatically harvest datasets from a range of DCAT sources, which is exactly what Öppnadata.se does. For Öppnadata.se, the CKAN support also makes it easy for Swedish public bodies who use CKAN to automatically expose their datasets correctly so that they can be automatically harvested by Öppnadata.se. For more information have a look at the CKAN DCAT extension documentation.

    Dead or alive

    The Web is decentralised and always changing. A link to a webpage that worked yesterday might not work today because the page was moved. When automatically adding external links, for example, links to resources for a dataset, you run into the risk of adding links to resources that no longer exist. To counter that Öppnadata.se uses a CKAN extension called Dead or alive. It may not be the best name, but that’s what it does. It checks if a link is dead or alive. The checking itself is performed by an external service called deadoralive. The extension just serves a set of links that the external service decides to check to see if some links are alive. In this way dead links are automatically marked as broken and system administrators of Öppnadata.se can find problematic public bodies and notify them that they need to update their DCAT catalog (this is not automatic because nobody likes spam). These are only the automation highlights of the new Öppnadata.se. Other changes were made that have little to do with automation but are still not immediately visible, so a lot of Öppnadata.se’s beauty happens behind the scenes. That’s also the case for other open data portals. You might just visit your open data portal to get some open data, but you might not realise the amount of effort and coordination it takes to get that data to you. Image of Swedish flag by Allie_Caulfield on Flickr (cc-by)

    Presenting public finance just got easier

    - March 20, 2015 in ckan, Open Spending

    This blog post is cross-posted from the CKAN blog. mexico_ckan_openspending CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new features of this release is the ability to create extensions that get called before and after a new file is uploaded, updated, or deleted on a CKAN instance. This may not sound like a major improvement but it creates a lot of new opportunities. Now it’s possible to analyse the files (which are called resources in CKAN) and take them to new uses based on that analysis. To showcase how this works, Open Knowledge in collaboration with the Mexican government, the World Bank (via Partnership for Open Data), and the OpenSpending project have created a new CKAN extension which uses this new feature. It’s actually two extensions. One, called ckanext-budgets listens for creation and updates of resources (i.e. files) in CKAN and when that happens the extension analyses the resource to see if it conforms to the data file part of the Budget Data Package specification. The budget data package specification is a relatively new specification for budget publications, designed for comparability, flexibility, and simplicity. It’s similar to data packages in that it provides metadata around simple tabular files, like a csv file. If the csv file (a resource in CKAN) conforms to the specification (i.e. the columns have the correct titles), then the extension automatically creates the Budget Data Package metadata based on the CKAN resource data and makes the complete Budget Data Package available. It might sound very technical, but it really is very simple. You add or update a csv file resource in CKAN and it automatically checks if it contains budget data in order to publish it on a standardised form. In other words, CKAN can now automatically produce standardised budget resources which make integration with other systems a lot easier. The second extension, called ckanext-openspending, shows how easy such an integration around standardised data is. The extension takes the published Budget Data Packages and automatically sends it to OpenSpending. From there OpenSpending does its own thing, analyses the data, aggregates it and makes it very easy to use for those who use OpenSpending’s visualisation library. So thanks to a perhaps seemingly insignificant extension feature in CKAN 2.3, getting beautiful and understandable visualisations of budget spreadsheets is now only an upload to a CKAN instance away (and can only get easier as the two extensions improve). To learn even more, see this report about the CKAN and OpenSpending integration efforts.

    Presenting public finance just got easier

    - March 20, 2015 in Technical, Updates

    This blog post is cross-posted from the CKAN blog. CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new […]

    Presenting public finance just got easier

    - March 20, 2015 in Technical, Updates

    This blog post is cross-posted from the CKAN blog. CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new […]

    Presenting public finance just got easier

    - March 20, 2015 in 特稿

    This blog post is cross-posted from the CKAN blog. CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new […]

    Presenting public finance just got easier

    - March 20, 2015 in Extensions, Feature, Featured, Releases, visualization

    mexico_ckan_openspending CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on. One of the new features of this release is the ability to create extensions that get called before and after a new file is uploaded, updated, or deleted on a CKAN instance. This may not sound like a major improvement  but it creates a lot of new opportunities. Now it’s possible to analyse the files (which are called resources in CKAN) and take them to new uses based on that analysis. To showcase how this works, Open Knowledge in collaboration with the Mexican government, the World Bank (via Partnership for Open Data), and the OpenSpending project have created a new CKAN extension which uses this new feature. It’s actually two extensions. One, called ckanext-budgets listens for creation and updates of resources (i.e. files) in CKAN and when that happens the extension analyses the resource to see if it conforms to the data file part of the Budget Data Package specification. The budget data package specification is a relatively new specification for budget publications, designed for comparability, flexibility, and simplicity. It’s similar to data packages in that it provides metadata around simple tabular files, like a csv file. If the csv file (a resource in CKAN) conforms to the specification (i.e. the columns have the correct titles), then the extension automatically creates the Budget Data Package metadata based on the CKAN resource data and makes the complete Budget Data Package available. It might sound very technical, but it really is very simple. You add or update a csv file resource in CKAN and it automatically checks if it contains budget data in order to publish it on a standardised form. In other words, CKAN can now automatically produce standardised budget resources which make integration with other systems a lot easier. The second extension, called ckanext-openspending, shows how easy such an integration around standardised data is. The extension takes the published Budget Data Packages and automatically sends it to OpenSpending. From there OpenSpending does its own thing, analyses the data, aggregates it and makes it very easy to use for those who use OpenSpending’s visualisation library. So thanks to a perhaps seemingly insignificant extension feature in CKAN 2.3, getting beautiful and understandable visualisations of budget spreadsheets is now only an upload to a CKAN instance away (and can only get easier as the two extensions improve). To learn even more, see this report about the CKAN and OpenSpending integration efforts.