You are browsing the archive for godi.

Avoin data osana datataloutta – sumeaa möhnää vai kirkasta lähdevettä?

Open Knowledge Finland - August 15, 2017 in avoin data, avoin talousdata, datatalous, Featured, godi, liiketoiminta, mydata, Open Government Data, ostolaskut, talous

HS pääkirjoitus 7.8.2017 tarttui tärkeään aiheeseen, datan ja tiedon avoimuuteen – johon myös lukijat vastasivat [1, 2]. Timo Paukku puolestaan toi kolumnissaan (HS 10.8.2017) esiin Viron toiminnan digitalisaation mallimaana, toistaen usein kuullun vertauksen datasta uutena öljynä. Data on keskeisen liiketoiminnan resurssi ja moottori niin kutsutulle datataloudelle, johon Suomikin enenevässä määrin pyrkii. Avoimessa yhteiskunnassa datan pitäisi kuitenkin muistuttaa enemmän vettä kuin öljyä: kaikkien saatavilla olevaa perushyödykettä, joka kastelee ja virkistää laajasti eri aloilla julkisella, yksityisellä ja kolmannella sektorilla – kansalaisia unohtamatta.

Odotukset avoimen datan liiketoimintaa kohtaan ovat olleet epärealistiset – merkittävät taloudelliset vaikutukset tulossa

Suurimmat avoimeen dataan liittyvät odotukset liittyvät sen liiketoimintaa piristävään vaikutukseen. EU:ssa arvioidaan luotavan 100 000 työpaikkaa avoimen datan ympärille vuoteen 2020 mennessä.  Avoimen datan potentiaaliseksi suoraksi hyödyksi on arvioitu 40 miljardia ja välillisiksi 100 miljardia euroa vuosittain pelkästään EU-alueella. Esim. Berends et al. listaavat joitain avoimen datan vaikutuksia. Lähde: https://www.europeandataportal.eu/sites/default/files/re-using_open _data.pdf Odotukset uutta liiketoimintaa kohtaan ovat niin suuret, että ne ovat väistämättä epärealistiset. Avoimen datan merkitys liiketoiminnalle sekä laajemmin taloudelle on merkittävä, mutta ei irrallisena saarekkeena, vaan osana laajempaa kehitystä, johon kuuluvat avoimen datan lisäksi myös ilmiöt ja teknologiat kuten My Data (ihmisten itseään koskeva ns. omadata), massadata, data-analytiikka ja tekoäly. Saadaksemme hyötyä irti avoimesta datasta, on meidän perinteisen innovaatiotoiminnan lisäksi panostettava avoimen datan löydettävyyteen ja laatuun sekä datatalouden osaamiseen kaikilla koulutustasoilla. On kuiteknin syytä muistaa, että suuri osa avoimen datan vaikuttavuudesta tulee muualta kuin suoraan liiketoiminnasta. Jos ajatellaan vaikkapa leikkausjonojen pituutta, hoitoonpääsytilastoja, velkaantumista, valtion hankintoja tai avoimia lobbaustietoja, on helppo kuvitella miten hyödyllistä olisi saada dataa ja analysoida sitä eri näkökulmista. Suomessa julkisen datan avaamista ja hyödyntämistä edistetään laajasti, hyvässä yhteistyössä julkishallinnon, yritysten ja järjestöjen kesken. Open Knowledge Finland ry ja kansainvälinen Open Knowledge -järjestöverkosto edistävät tiedon avaamista Suomessa ja ympäri maailmaa. Laajat julkiset hankkeet, kuten kuuden suurimman kaupungin 6Aika-hanke, edistävät avoimeen dataan perustuvan uuden liiketoiminnan syntymistä kuudessa suurimmassa kaupungissamme – www.databusiness.fi -sivustolta löytyy toista sataa avointa dataa hyödyntävää palvelukuvausta! Pääkaupunkiseudun kaupunkien erinomaiseen Helsinki Region Infoshare -palveluun, josta löytyy satoja tietoaineistoja, käydään puolestaan jatkuvasti tutustumassa ympäri maailmaa. Elinkeinoelämän tutkimuslaitos Etla ja Open Knowledge Finland selvittivät yhdessä avoimen datan vaikuttavuutta liiketoiminnassa. Selvityksen perusteella avointa dataa innovaatiotoiminnassaan hyödyntävät yritykset tuovat markkinoille uusia tavara- ja palveluinnovaatioita suhteellisesti useammin kuin muut yritykset. Dataa uusien innovaatioiden kehittämisessä käyttäneiden ICT-yritysten liikevaihto kasvoi vuosina 2012–14 keskimäärin yli 17 prosenttia muita enemmän. Erityisesti liikennetietojen hyödyntäminen kiihdytti liikevaihdon kasvua. (Tämän tutkimukset sisällöistä tulossa oman kirjoitus myöhemmin.)

Talousdataa on avattu ja avataan lisää lähiaikoina

Pääkirjoituksessa ettei dataa, jonka avulla voisi seurata esimerkiksi valtion ja kuntien viranomaisten rahankäyttöä, ole ollut saatavilla. Kirjoittaja on kyllä osin oikeassa siinä mielessä, että kansainvälisissä vertailuissa nimenomaan taloustietojen avoimuuden osalta Suomen sijoittuu surkeasti, mutta juuri nyt on hyvä mahdollisuus kohentaa tilannetta. Toisaalta, kaupunkien ostodataa on avattu useissa suomalaisissa kaupungeissa. Esimerkiksi Helsinki on avannut yksityiskohtaisen ostodatansa jo vuosia sitten (vrt. esim.  HS 28.11.2014) ja kaupunginjohtaja on arvioinut pelkästään datan avaamisen tuovan 1-2% säästöt vuosittain. Valmisteilla oleva ns. Hansel-laki avaisi koko julkishallinnon hankinnat ja ostot. Tämän seurauksena Suomi täyttäisi kansainvälisiä sitoumuksiaan, saisi verovarat tehokkaammin käyttöön, kilpailu julkisista hankinnoista olisi reilumpaa ja julkinen rahankäyttö olisi ylipäätään avoimempaa. Lähde: https://twitter.com/avoindatafi/status/879965329954398208 Pääkirjoituksessa mainittiin myös, että “toistaiseksi avattu data on verotietoja lukuunottamatta harmitonta”. Avoimuus arveluttaa monia. On selvää, että avattu data ei saa loukata esimerkiksi henkilön yksityisyyttä. Sen sijaan huoli avoimuuden ja toiminnan läpinäkyvyyden mukana mahdollisesti tulevasta kritiikistä ei ole kestävä peruste tiedon panttaamiselle. Avoimen datan avulla voidaan tunnistaa hallinnon tai yritysten tehottomuutta tai epäselvyyksiä, auttaa kansalaisia ymmärtämään talouden tai ympäristön tilaa paremmin tai kehittämään palveluita, jotka esimerkiksi helpottavat metsän hoitoa tai myyntiä. Julkisten tietovarantojemme monipuolinen ja älykäs hyödyntäminen on yhteiskuntamme etu ja kilpailuvaltti. Juuri siksi meidän on edelleen pistettävä painetta päättäjiä ja datanomistajia kohtaan. Tahdomme enemmän! The post Avoin data osana datataloutta – sumeaa möhnää vai kirkasta lähdevettä? appeared first on Open Knowledge Finland.

Valtion hankintatiedot avoimena datana – hieno edistysaskel tulossa?!

Open Knowledge Finland - August 9, 2017 in avoin data, avoin hallinto, Featured, godi, godi 2016, hansel, julkiset hankinnat, Open Government Data, Open Government Partnership, Open Spending

Tiedon avoimuutta on tarpeen lisätä, Helsingin Sanomien pääkirjoituksessa 7.8.2017 todetaan. Olemme samaa mieltä! Viime vuosina useat Suomen kunnat ovat julkaisseet tietoja omista hankinnoistaan, jopa kuittitasolla. Tämä käytäntö on laajenemassa uuden ns. Hansel-lain myötä, jonka myötä eri ministeriöiden, laitosten, virastojen ja mahdollisesti maakuntien hankinnat julkaistaisiin keskitetysti valtion hankintayhtiön Hansel Oy:n toimesta. Hallitus valmistelee uutta ns. Hansel-lakia (Hallituksen esitys HE 63/2017 vp Hallituksen esitys eduskunnalle laiksi Hansel Oy -nimisestä osakeyhtiöstä annetun lain muuttamisesta). Laki on käsittelyssä talousvaliokunnassa, jossa sen yksityiskohtia viimeistellään. Avoimuuden ja avoimen datan kannalta erityisen kiinnostavia ovat lakiehdotuksen kohdat, jossa ehdotetaan säädettäväksi uusi säännös hankintatiedon käsittelyyn liittyvästä tietojensaanti- ja käsittelyoikeudesta (2§ ja 5§). Nähdäksemme toteutuessaan Hansel-laki lisää hallinnon avoimuutta erinomaisella tavalla. Samalla kun Suomi täyttää kansainvälisiä sitoumuksiaan, saamme verovarat tehokkaammin käyttöön, kilpailu julkisista hankinnoista on reilumpaa ja julkinen rahankäyttö on ylipäätään avoimempaa.

Hansel-laki ja hankintoja koskeva avoin data

Hansel Oy on siis toiminut valtion yhteishankintayksikkönä ja kilpailuttanut asiakkailleen sellaisia tavara- ja palveluhankintoja, joita valtionhallinnossa käytetään laajasti. Hanselin tehtäviin on kuulunut myös asiakkaiden omien hankintojen kilpailuttaminen sekä erilaiset hankintatoimeen liittyvät asiantuntijatehtävät. Viime vuosina yhtiön tehtävät ovat kehittyneet muun muassa valtion hankintatoimen digitalisointiohjelman myötä, minkä vuoksi lakiin ehdotetaan tehtäväksi joitakin yhtiön tehtäviin liittyviä täsmennyksiä.  Laissa yhtiön tehtäviä siis ajantasaistetaan. Talousvaliokunnassa lakitekstiä on tiettävästi muotoiltu eteenpäin, mutta viimeisin julkinen versio (HE 63/2017) kuvaa Hanselin muuttuneita tehtäviä mm. seuraavasti (boldaus kirjoittajan): 2§ Yhtiön tehtävät 2 mom: Yhtiön tehtävänä on tuottaa asiakkailleen yhteishankintatoimintoja ja hankintojen tukitoimintoja. Yhtiö ylläpitää hankintasopimuksia ja tuottaa asiakkailleen hankintasopimuksiin liittyvää asiantuntijapalvelua. Lisäksi yhtiön tehtävänä on tuottaa asiakkailleen hankintatoimeen liittyviä asiantuntija- ja kehittämispalveluja sekä hankintatiedon käsittely- ja analysointipalveluja ja näihin liittyviä teknisiä ratkaisuja. 5 §  Tiedonsaantioikeus ja tietojen tuottaminen 4 mom: Yhtiö voi tuottaa, luovuttaa ja julkaista hankintatietoa käsittävää tietoaineistoa, jos tietoaineiston luovuttaminen ei sen muodostamisessa käytettyjen hakuperusteiden, tietojen määrän, laadun tai sisällön taikka tietoaineiston käyttötarkoituksen vuoksi ole vastoin sitä, mitä tietojen salassapidosta ja henkilötietojen suojasta säädetään. Alla muutama Open Knowledge Finland ry:n näkemys lakiin liittyen.

Hansel-laki lisää julkisten hankintojen tervettä kilpailua ja tehokkuutta

Hankintatietojen avoimuus edistää reilua kilpailua eri toimittajien kesken. Hankintojen vertailutietojen kautta saadaan kustannustehokkuutta hankintoihin ja sitä kautta verovarojen käyttöön. Kun hankinnat kuvataan vertailukelpoisesti, voidaan helposti seurata esimerkiksi, maksaako joku yksikkö huomattavan erilaista hintaa toiseen verrattuna tai onko hankinnoissa jotain muuta poikkeavaa tai erikoista ja kenties parannettavaa (kuten hankintojen kasautuminen vuoden loppuun). Pidemmällä tähtäimellä vertailukelpoiseen dataan voisi lisätä tai yhdistää vaikkapa alkuperätietoa, sertifiointeja, eettistä tietoa tai muuta vertailutietoa. Yksityiskohtainen hankintatieto voi auttaa ehkäisemään korruptiota ja harmaata taloutta.

Julkisuuslaki, läpinäkyvyys ja oikeus tietoon koskee myös hankintatietoja

Joka tapauksessa julkisuuslain mukaan kansalaisilla, järjestöillä ja medialla on jo nyt olemassa oikeus tietoon – myös hankintatietoon – silloin kun kyse ei ole erityisistä seikoista kuten esimerkiksi turvallisuusasioista tai tietyn tyyppisistä yrityssalaisuuksista. Riippumatta Hansel-laista, oikeus tähän tietoon on olemassa, eikä siltä osin ole tiedossa muutoksia.  Mutta kiinnostava muutos on, että Hansel-lain myötä saadaan selkeyttä ja yhdenmukaisuutta tiedon julkaisuun liittyviin käytänteihin ja laki toteuttaa ja tarkentaa siten julkisuuslain henkeä.

Yksi toimija hankintatiedon julkaisijana on tehokas tapa lisätä läpinäkyvyyttä ilman suurta hallinnollista taakkaa julkishallinnolle

Yksi datan avaamisen haasteista julkishallinnossa on ollut julkisuuslain erilaisten tulkintojen määrä – tämä tuli esille mm. Valtioneuvoston kanslian selvitys- ja tutkimustoiminnan “Avoimen datan hyödyntäminen ja vaikuttavuus”  -raportissat, jonka ETLA ja Open Knowledge Finland tekivät. Vastaavasti kaupungit avatessaan ostolaskujaan, ovat soveltaneet toisistaan poikkeavia käytäntöjä ja dataformaatteja. 6Aika-hanke ja kuntaliitto ovatkin pyrkineet ohjeistamalla yhtenäistämään käytäntöjä. Kaavailtu käytäntö yhtenäistää käytäntöjä eri ministeriöiden, virastojen ja tutkimuslaitosten kesken. Kaavailtu käytäntö keventää hallinnollista taakkaa kun asiat, kuten tiedon siivoaminen, formatointi, priorisointi, ongelmienratkaisu, dokumentointi, julkaisukäytännöt ym. tukitoiminnot hoidetaan yhdessä paikassa, eli Hansel toimii tässä siis eräänlaisena ns. clearinghousena, laadunvarmistajana ja tiedon hyödyntäjien rajapintana. Yhtenäiset käytännöt puolestaan paitsi lisäävät julkishallinnon tehokkuutta datan julkaisussa, myös helpottavat datan löydettävyyttä ja hyödynnettävyyttä.  Sivumennen sanoen, uuden lakiehdotuksen kaavailemat tietopalvelut täydentävät muita meneillään olevia hankkeita, kuten YTI-hanke ja Kuntatieto-ohjelma. Hansel on valtionvarainministeriön ohjaukessa ja hankintatietojen avoimuus yhtenä asiana Valtion hankintojen digitalisaatio -toteutusohjelmaa, joten tahtotilaa modernisointiin tuntuu olevan laajemmin.  

Kansainväliset johtajuus ja tehtyjen sitoumuksien lunastaminen

Yleisesti Suomi sijoittuu kansainvälisesti avoimeen dataan ja avoimeen tietoon liittyvissä vertailuissa melko hyvin. Esimerkiksi uusimmassa Open Knowledge Internationalin Government Open Data Index 2016 -vertailussa olemme sijalla 5. Toisaalta, nimenomaan taloustietojen avoimuudessa olemme varsin surkeita – hankintojen (“procurement” – hankintailmoitusten ja sopimusten) tiimoilta “45% avoin” ja ostojen (“Government spending” – todellinen kulutus) jopa hälyttävällä 0% tasolla! Hansel-lain myötä pysymme mukana kansainvälisessä kehityksessä kun vahvistamme todettuja heikkouksiamme.   Suomi on myös mukana USA:n ex-Presidentti Obaman aloittamassa avoimen hallinnon kumppanuusohjelmassa (Open Government Partnership), jossa eri maiden (yli 70 maata on mukana) hallinnot yhdessä kansalaisyhteiskunnan kanssa tekevät sitovia avoimuutta edistäviä konkreettisia toimenpiteitä ja sitoumuksia.  Hankintatietojen avoimuus on myös Suomen Avoimen hallinnon 3. Toimintaohjelmassa (2017-2019) yhtenä konkreettisena lupauksena. Toimintaohjelmassa sanotaan näin.
  1. sitoumus
Julkaistaan valtion hankintatiedot kansalaisille. Julkaistaan avoimesti verkossa tiedot siitä, mitä valtio ostaa, millä rahalla ja mistä. Valtion hankintatiedot julkaistaan keväällä 2017 avoimena datana. Samalla toteutetaan kaikille avoin palvelu, jossa kansalaiset ja yritykset voivat seurata lähes reaaliaikaisesti valtion hankintoihin liittyvän rahan käyttöä. Palvelujen tietosisältönä ovat hankintojen julkiset tiedot, joista käy ilmi, mitä valtion organisaatiot hankkivat ja mistä hankinnat tehdään. Sinänsä Hansel-laki ja sen kuvailemat hankintatietoon liittyvät tietopalvelut eivät ole välttämättä suunniteltuja “vain” kansalaisille, vaan palveluille on luonnollisesti useita eri käyttäjiä, kuten yritykset, media ja julkinen sektori itse. Ylipäätään olennaista on, että data avataan. Tällöin erilaiset toimijat voivat tehdä omista näkökulmistaan erilaisia kiinnostavia sovelluksia – joku tekee vertailuja tai visualisointeja, joku myynnin ja markkinoinnin työkaluja ja niin edelleen! Näin eri toimijat täydentävät Hanselin osaamista ja tarjontaa – tieto kun ei jakamalla kulu. Eräänlainen verrokki voisi olla valtion budjetti ja sen ympärillä olevat sovellukset: valtion budjettia kuvaava, Hahmota Oy:n tekemä www.valtionbudjetti.fi joka tavallaan täydentää VM:n omaa www.tutkibudjettia.fi -palvelua. Vastaavasti Hanselin mahdollisesti tuottaman verkkotyökalun (jolla voi tutkia ja analysoida hankintoja tietyin kriteerein) lisäksi on hyvin mahdollista, että syntyy muita palveluita tai analyysityökaluja hankintoihin. Toteuttaakseen kaavailtua lakia sekä em. avoimen hallinnon sitoumusta, Hansel onkin käsittääkseni hahmotellut tulevaa verkkopalvelua, jossa hankintoja voisi analysoida. Alla muutamia esimerkinomaisia ruutukaappauksia sovelluksesta, jotka antavat suuntaviivoja siitä miltä verkkopalvelu voisi näyttää. Nämä ruutukaappaukset ovat tietystikin suuntaa-antavia, mutta vaikuttavat lupaavalta. Tarkempia analyysejä varten itse kukin voisi sitten ladata tietoa sopivin kriteerein rajattuna. Ylipäätään talouden ja talouselämän avoimuutta ja avointa tietoa olisi järkevää lisätä jatkossa. Tavoitteena tulisi mielestämme olla, että “kolminaisuus”, eli  julkiset budjetit, sopimukset ja hankinnat olisivat saatavilla avoimesti standardimuotoisina. Hankintatiedot on erinomainen askel.   Odotamme Open Knowledge:ssa innolla uutta Hansel-lakia ja ylipäätään julkisten hankintojen lisääntyvää avoimuutta. Avoimuus on omiaan paitsi hälventämään mahdollista epäluottamusta, myös lisäämään tehokkuutta ja reilua kilpailua. Kyseessä on siis veronmaksajien etu ja oikeudenmukaisuus. The post Valtion hankintatiedot avoimena datana – hieno edistysaskel tulossa?! appeared first on Open Knowledge Finland.

Ποια δεδομένα χρειαζόμαστε; Η ιστορία της υποτροφίας του Cadasta GODI

Χριστίνα Καρυπίδου - June 26, 2017 in Featured, Featured @en, godi, News, ανοικτά δεδομένα, Νέα

Από τη Mor Rubinstein Αυτό το blogpost γράφτηκε από τις Lindsay Ferris και Mor Rubinstein Υπάρχουν πολλά δεδομένα εκεί έξω, αλλά ποιοι χρήστες δεδομένων χρειάζονται για να λύσουν τα προβλήματά τους; Πώς μπορούμε εμείς, ως εξωτερικό σώμα, να γνωρίζουμε ποια δεδομένα είναι ζωτικής σημασίας, ώστε να τα μετρήσουμε; Επιπλέον, τι πρέπει να κάνετε όταν δημοσιεύονται […]

What is the difference between budget, spending and procurement data?

Danny Lämmerhirt - May 18, 2017 in Global Open Data Index, godi, OpenSpending

Fiscal data is a complex topic. It comes in all different kind of formats and languages, its’ availability cannot be taken for granted and complexity around fiscal data needs special skills and knowledge to unlock and fully understand it. The Global Open Data Index (GODI) assesses three fiscal areas of national government: budgets, spending, and procurement. Repeatedly our team receives questions why some countries rank low in budgets, public procurement or spending, even though fiscal data is openly communicated. The quick answer: often we find information that is related to this data but does not exactly describe it in accordance with the described GODI data requirements. It appears to us that a clarification is needed between different fiscal data. This blogpost is dedicated to shed light on some of these questions. As part of our public dialogue phase, we also want to address our experts in the community. How should we continue to measure the status of these three key datasets in the future? Your input counts! Should we set the bar lower for GODI and avoid measuring transactional spending data at all? Is our assessment of transactional spending useful for you? You can leave us your feedback or join the discussion on this topic in our forum.

The different types of fiscal data

A government budget year produces different fiscal data types. Budgeting is the process where a government body sets its priorities as to how it intends to spend an amount of money over a specific time period (usually annually or semi-annually). Throughout the budgeting cycle  (the process of defining the budget), an initial budget can undergo revisions to result in a revised budget. Spending is the process of giving away money. This mean, the money might be given as a subsidy, a contract, refundable tax credit, pension or salary. Procurement is the process of selecting services from a supplier who fits best the need. That might involve selecting vendors, establishing payment terms, some strategic tender or other vetting mechanism meant to prevent corruption. Not only are the processes linked to each other, the data describing these processes can be linked too (e.g. in cases where identifiers exist linking spending to government budgets and public contracts). For laypersons, it might be difficult to tell the difference when they are confronted with a spending or procurement dataset: Is the money I see in a dataset spending, or part of contracting? The following paragraphs explain the differences.

Budget

As mentioned above, budgeting is called the process where a government body decides how to spend money over a certain time period. The amount is broken into smaller amounts (budget items) which can be classified as follows:
  • Administrative (which government sub-unit gets the money)
  • Functional (what the money is going to be used for)
  • Economic (how the money is going to be used, e.g., procurement, subsidies, salaries etc.)
  • Financing source (where the money should come from).
After the budget period ends, we know how much money was actually spent on each item – in theory. The Global Open Data Index assesses budget information at the highest administrative level (e.g. national government, federal government), which is broken down in one of these classifications. Here is an example of some fully open budget data of Argentina’s national government.

Example of Argentina’s national government budget 2017 (table shortened and cleaned)

The image shows the government entity, and expenditures split into economic classification (how the money is used). At the far right, we can see a column describing the total amount of money effectively spent on a planned budget expenditure. It basically compares allocated and paid money. This column must not be mixed with spending information on a transactional level (which displays each single transaction from a government unit to a recipient).

Spending

The Spending Data Handbook describes spending as “data relating to the specific expenditure of funds from the government”. Money might be given as a subsidy, as payment for a provided service, a salary (although salaries will seldom be published on a transactional level), a pension fund payment, a contract or a loan, to name just a few.
GODI focusses on transactions of service payments (often resulting from a prior procurement process). Monetary transactions are our baseline for spending data. GODI assesses the following information:
  • The amount that was transferred
  • The recipient (an entity external to the government unit)
  • When the transaction took place
  • Government office paying the transaction
  • Data split into individual transactions
GODI exclusively looks for single payment transfers. The reason why we are looking at this type of data is that spending patterns can be detected, and fraud or corruption uncovered. Some of the questions one might be able to address include: Who received what amount of money? Could government get its services from a cheaper service provider? Is government contracting to a cluster of related companies (supporting cartels)? GODI’s definition of spending data, even though ambitious in scope, does not consider the entire spectrum of transactional spending data. Being produced by many agencies, spending data is scattered  across different places online. We usually pick samples of specific spending data such as government payments to external suppliers (e.g. the single payments through a procurement process). Other types of payment, such as grants, loans or subsidies are then left aside. Our assessment is also ‘generous’ because we accept spending data that is only published above a certain threshold. The British Cabinet Office, a forerunner in disclosing spending data, only publishes data above £25,000. GODI accepts this as valid, even though we are aware that spending data below this amount remains opaque. There are also many more ways to expand GODI’s definition of spending data. For instance, we could ask if each transaction can be linked to a budget item or procurement contract so that we understand the spending context better.

Example image of British Spending data (Cabinet Office spending over £25,000)

Above is an example image of Great Britain’s Cabinet Office spending. You can see the date and the amount paid by government entity. Using the supplier name, we can track how much money was paid to the supplier. However, in this data no contract ID or contract name is provided that could allow to fully understand as part of what contracts these payments have been made.

Procurement

When purchasing goods and services from an external source, government units require a certain process for choosing the supplier who fits best the need. This process is called procurement and includes planning, tendering, awarding, contracting and implementation. Goals are to enable a fair competition among service providers and to prevent corruption. Many data traces enable to shed light on each procurement stage. For example one might want to understand from which budget a service is gonna be paid, or what amount of money has been awarded (with some negotiation possible) or finally contracted to a supplier. This blogpost by the Open Contracting Partnership illustrates how each of the procurement stages can be understood through different data. GODI focuses on two essential stages, that are considered to be a good proxy to understand procurement. These however do not display all information. Tender phase
  • Tenders per government office
  • Tender name
  • Tender description
  • Tender status
Award phase
  • Awards per government office
  • Award title
  • Award description
  • Value of the award
  • Supplier’s name
Any payment resulting out of government contracts with external suppliers (sometimes only one, sometimes more) has to  be captured in government spending. For example, there might a construction contractor that is being paid by milestone, or an office supplies dealer which is chosen as a supplier. Then each spending transaction is for a specific item purchased through a procurement process. Below you can see a procurement database of Thailand. It displays procurement phases, but does not display individual transactions following from these. This particular database does not represent actual spending data (monetary transactions), but preceding stages of the contracting process. Despite this the platform is misleadingly called “Thailand Government Spending”.

Procurement database in Thailand

Another example is a procurement database indicating how much money has been spent on a contract:

Example for the procurement website ‘Cuánto y a quién se contrató’ (Colombia)

The road ahead – how to measure spending data in the future

Overall, there is slow but steady progress around the openness of fiscal data. Increasingly, budget and procurement data is provided in machine-readable formats or openly licensed, sometimes presented on interactive government portals or as raw data (more detail see for example in the most recent blogpost of the Open Contracting Partnership around open procurement data). Yet, there is a long way to go for transactional spending data. Governments do first laudable steps by creating budget or procurement websites which demonstrate how much money will or has been spent in total. These may be confusingly named ‘spending’ portals because in fact they are linked to other government processes such as budgeting (e.g. how much money should be spent) or procurement (how much money has been decided to pay for an external service). The actual spending in form of single monetary transactions is missing. And to date there is no coherent standard or specification that would facilitate to document transactional spending. We want to address our experts in the community. How should we continue to measure the status of these three key datasets in the future? Your input counts!  You can leave us your feedback and discuss this topic in our forum.   This blog was jointly written by Danny Lämmerhirt and Diana Krebs (Project Manager for Fiscal Projects at Open Knowledge International)

How to Read the Global Open Data Index Results

Danny Lämmerhirt - May 2, 2017 in Global Open Data Index, godi, Open Government Data

The Global Open Data Index (GODI) is a tool to educate civil society and governments about open government data publication. We do so through presenting different information, including places scores, ranking, and scores for each data category per place, and comments by our submitters and reviewers. Even though we try to make this assessment coherent and transparent as possible, interpreting the results is not always straightforward. While open data has a very strict definition, scoring of any index is a discretional action. In real life, you can’t be partly open – either the data fit the criteria, or they do not.  This blog post will help GODI user to understand the following:  – What does the final score mean?  – How to interpret scores that vary between 0%, 40% or 70%?  – What does a score of 0% mean? For a more thorough explanation on how to read the results, go to index.okfn.org/interpretation/ 

What does the score mean?

Our scoring (ranging from 0% open to 100% open) does not necessarily show a gradual improvement towards open data. In fact, we assess very different degrees of data openness – which is why any score below 100 percent only indicates that a dataset is partially open. These levels of openness include public data, access-controlled data, as well as data gaps (See GODI methodology). To understand the differences we highly recommend reading each score together with our openness icon bar (see image below). For instance: a score of 70% can say that we found access-controlled, machine-readable data, that cannot be downloaded in bulk. Any score below 100% means “no access”, “closed access” or “public access”. Here we explain what each of them means, and how the data for each category look in practice.

Public Access Data

Data is publicly accessible if the public can see it online without any access controls. It does not imply that data can be downloaded, or that it is freely reusable. Often it means that data is presented in HTML on a website. The image above shows a search interface of a company register. It allows for targeted searches for individual companies but does not enable to retrieve all data at once. Individual search results (non-bulk) are displayed in HTML format and can then be downloaded as PDF (not machine-readable). Therefore the score is 70% and visualised as follow openness icon bar in our ranking:

Access-controlled data

Data is access-controlled if a provider regulates who, when, and how data can be accessed. Access control includes: * Registration/identification/authentication * Data request forms, data sharing agreement (stipulating use cases), * Ordering/purchasing data.   There are many reasons for establishing access controlled data including website traffic management, or to maintain control over how data is used. It is debatable whether some registration/authentication mechanisms reduce the openness of data (especially when registration is automated). Data request forms, on the other hand, are clearly not open data.   This image shows a data request form. The dataset is entirely hidden behind a “paywall”. This often prevents our research team from assessing the data at all. In this case, we could not verify in which format the data will be provided, and neither whether the data are actually weather forecast data (the particular category we look at). Therefore this access-controlled data gained 0 points and counts as 0% open. By contrast,  access-controlled data often score very high, up to 85% (because we subtract 15 out of 100 points for access-controls like registration requirements). 
 

How to read a score of 0%?

The are many reasons why datasets will score 0%. We tried to address the reasons in the reviewer or submitter’s comments as well. See here for the main reasons: Data gaps A data gap can mean that governments do not produce any data in a given category. Sometimes, if GODI shows a zero percent score, we see data gaps. For instance the case for Western African countries that lack air quality monitoring systems, or countries that have no established postcode system. Data gaps indicate that the government information systems are not ready to produce open data, sometimes because resources are missing, at times because it is not a priority of government.

Exist, but only to governmental use

Sometimes government has the data, but for many reasons choose not to open it to the public at all.  

Not granular

Since our criteria look for particular granularity, we considered all datasets that didn’t reach this granularity levels as not granular, and therefore they were regarded as not available. For example – Great Britain has published elections results, but not on poll station level, which is a crucial level to detect voter fraud. Therefore, while there is some data for UK elections, it is not at the right level and considered as non-existent.

 Do not fit our criteria

We are looking for particular datasets in GODI. When they don’t have all the characteristics we are looking for, we consider them as not available. For the full explanation on how to read the results see – index.okfn.org/interpretation/

Introducing the 4th Global Open Data Index – Advancing the State of Open Data Through Dialogue

Open Knowledge International - May 1, 2017 in Featured, Global Open Data Index, godi, Open Government Data

We are pleased to present the 4th edition of the Global Open Data Index (GODI), a global assessment of open government data publication. GODI compares national government in 94 places across the 15 key datasets that have been assessed by our community as the most useful for solving social challenges. For this edition, we received 1410 submitted of datasets, but only 10% of these are open according to the Open Definition. At Open Knowledge International (OKI), we believe it’s important to look further than just the numbers. GODI is not just a benchmark, it can be and should be used as a tool to improve open data publication and make data findable, useful and impactful. This is why we include a new phase, the dialogue phase for this edition. Measuring the publication of open government data is a constant challenge and while open data has a fixed definition, practices for data publishing that vary widely from one government to government, and even within each government department. The GODI community has faced a number of difficult questions that make assessment harder. ‘Which government agency publishes the data?’ or ‘Is the data published on a portal or a department web page?’ are just two examples. For this edition of the GODI, we gave attention to the development of the methodology to help us address some of these challenges in the assessment process. Not only did we incorporate feedback from the past GODIs, but we engaged in systematic and detailed in a consultation with the community. In addition, we had a more extensive review phase including an expanded quality assurance stage. We believe that these changes are necessary, and makes our assessment better than ever. In past years we published GODI as a snapshot in time and didn’t make any change to it after publication. Like any assessment tool, we are not perfect, and we did find ourselves publishing an errata section to accommodate errors. This year we want to be responsive to feedback from the community and government, and give both the opportunity to debate the results. We encourage every user to contest the results for a dataset by publishing a comment about it on our forum. We will be accepting feedback for the next 30 days, and on June 2nd we will re-evaluate the index, correct errors before closing it for submissions until the next edition. What are we looking to achieve from the dialogue stage?  
  1. Feedback from the full spectrum of stakeholders including government publishers to data users. This helps to improve future editions of the index.
  2. Help government to understand where the gaps are in their publication of data.
  3. Help citizens to find the data sets they are looking for.
  4. Help publishers of data to understand the difficulties of users accessing the data
Different challenges arise from open data publication today. Over the next couple of weeks, we will be publishing more of our insights and thoughts on these difficulties. Our main findings are:
  1. GODI helps to identify “data gaps.”
  2. Open data is not easy to find, and governments frequently don’t publish on their portals but across different government websites or split across many pages on one website.
  3. There is data online, but users find it difficult to access and work with
  4. Licenses remain an issue in open data publication
You can read the more about GODI findings on the insights page Learn how to read the results, and download the data for your own research Please take a look at the new edition of GODI and challenge us by letting us know what you think on the forum!  

Introducing the 4th Global Open Data Index – Advancing the State of Open Data Through Dialogue

Open Knowledge International - May 1, 2017 in Featured, Global Open Data Index, godi, Open Government Data

We are pleased to present the 4th edition of the Global Open Data Index (GODI), a global assessment of open government data publication. GODI compares national government in 94 places across the 15 key datasets that have been assessed by our community as the most useful for solving social challenges. For this edition, we received 1410 submitted of datasets, but only 10% of these are open according to the Open Definition. At Open Knowledge International (OKI), we believe it’s important to look further than just the numbers. GODI is not just a benchmark, it can be and should be used as a tool to improve open data publication and make data findable, useful and impactful. This is why we include a new phase, the dialogue phase for this edition. Measuring the publication of open government data is a constant challenge and while open data has a fixed definition, practices for data publishing that vary widely from one government to government, and even within each government department. The GODI community has faced a number of difficult questions that make assessment harder. ‘Which government agency publishes the data?’ or ‘Is the data published on a portal or a department web page?’ are just two examples. For this edition of the GODI, we gave attention to the development of the methodology to help us address some of these challenges in the assessment process. Not only did we incorporate feedback from the past GODIs, but we engaged in systematic and detailed in a consultation with the community. In addition, we had a more extensive review phase including an expanded quality assurance stage. We believe that these changes are necessary, and makes our assessment better than ever. In past years we published GODI as a snapshot in time and didn’t make any change to it after publication. Like any assessment tool, we are not perfect, and we did find ourselves publishing an errata section to accommodate errors. This year we want to be responsive to feedback from the community and government, and give both the opportunity to debate the results. We encourage every user to contest the results for a dataset by publishing a comment about it on our forum. We will be accepting feedback for the next 30 days, and on June 2nd we will re-evaluate the index, correct errors before closing it for submissions until the next edition. What are we looking to achieve from the dialogue stage?  
  1. Feedback from the full spectrum of stakeholders including government publishers to data users. This helps to improve future editions of the index.
  2. Help government to understand where the gaps are in their publication of data.
  3. Help citizens to find the data sets they are looking for.
  4. Help publishers of data to understand the difficulties of users accessing the data
Different challenges arise from open data publication today. Over the next couple of weeks, we will be publishing more of our insights and thoughts on these difficulties. Our main findings are:
  1. GODI helps to identify “data gaps.”
  2. Open data is not easy to find, and governments frequently don’t publish on their portals but across different government websites or split across many pages on one website.
  3. There is data online, but users find it difficult to access and work with
  4. Licenses remain an issue in open data publication
You can read the more about GODI findings on the insights page Learn how to read the results, and download the data for your own research Please take a look at the new edition of GODI and challenge us by letting us know what you think on the forum!  

Network update from OK Japan: Corporate transparency and taxpayers’ money ahead of the 2020 Tokyo Olympics

Open Knowledge Japan - January 19, 2017 in Chapters, Chapters updates, godi, japan, network, OK Japan, Open Spending

This blog post is part of our on-going Network series featuring updates from chapters across the Open Knowledge Network and was written by the Open Knowledge Japan team. The OK Japan chapter has been active in the open data space in activities such as the promotion of open data use and policy discussions. Since we formed the team in 2012, our members have been instrumental in promoting International Open Data Day in Japan and OpenSpending/ Where Does My Money Go. We published use cases and other notable developments in the space through our blog. Our members also took part in many different government boards, advised or worked with municipalities and departments on open data implementation. Below is some news about us and open data developments in Japan.

Transparency discussed

Late October, Open Knowledge Japan has co-organized, with OpenCorporates Japan an event discussing corporate ID and transparency issues, including the Panama Papers. The keynote talk was given by Chris Taggart, CEO and founder of OpenCorporates, who was visiting Tokyo that time.

okjapanChris Taggart and Japanese experts discussing transparency issues in Tokyo

Work meeting held for Global Open Data Census

We hosted an informal meeting inviting key government officials to work on the Global Open Data Census. The Census scores and Japan’s ranking have been discussed in the open data policy circle.

Relevance of open knowledge for Japan

Aside from what we did lately, there are recent news reports that make open knowledge issues very relevant in the country. Related to the 2020 Tokyo Olympics, we have been learning about many allegations of shady processes. For example, some large sums of tax money reported going from our government to an unnamed “consultant” so that Tokyo could become the host city for the 2020 Olympics. Tokyo has also been involved in other transparency issues – the governor resigned this year after criticisms related to his spending and lack of clear explanations on those, and was given a vote of no-confidence. The new governor uncovered additional problems with the ongoing project of relocating the Tsukiji market, the largest fish market in Tokyo, including potential underground water contamination.

tokyo_tower_special_lightup_invitation_for_2020_olympic_games_on_march_2013Image Credit: Tokyo Tower Special Lightup <Invitation for 2020 Olympic Games> (Shibakouen, Tokyo, Japan) (CC BY)

In the early part of 2017, we will be working towards International Open Data Day 2017. Japan has been one of the most active countries in terms of the number of localities participating in IODD in the past few years (with more than 60 cities participating in 2016!). Some of the issues we will be discussing through this and other occasions include the above-mentioned data plans that the national and prefectural governments will create, as well wider use of Open Spending Next that some of our members have started learning.

Network update from OK Japan: Corporate transparency and taxpayers’ money ahead of the 2020 Tokyo Olympics

Open Knowledge Japan - January 19, 2017 in Chapters, Chapters updates, godi, japan, network, OK Japan, Open Spending

This blog post is part of our on-going Network series featuring updates from chapters across the Open Knowledge Network and was written by the Open Knowledge Japan team. The OK Japan chapter has been active in the open data space in activities such as the promotion of open data use and policy discussions. Since we formed the team in 2012, our members have been instrumental in promoting International Open Data Day in Japan and OpenSpending/ Where Does My Money Go. We published use cases and other notable developments in the space through our blog. Our members also took part in many different government boards, advised or worked with municipalities and departments on open data implementation. Below is some news about us and open data developments in Japan.

Transparency discussed

Late October, Open Knowledge Japan has co-organized, with OpenCorporates Japan an event discussing corporate ID and transparency issues, including the Panama Papers. The keynote talk was given by Chris Taggart, CEO and founder of OpenCorporates, who was visiting Tokyo that time.

okjapanChris Taggart and Japanese experts discussing transparency issues in Tokyo

Work meeting held for Global Open Data Census

We hosted an informal meeting inviting key government officials to work on the Global Open Data Census. The Census scores and Japan’s ranking have been discussed in the open data policy circle.

Relevance of open knowledge for Japan

Aside from what we did lately, there are recent news reports that make open knowledge issues very relevant in the country. Related to the 2020 Tokyo Olympics, we have been learning about many allegations of shady processes. For example, some large sums of tax money reported going from our government to an unnamed “consultant” so that Tokyo could become the host city for the 2020 Olympics. Tokyo has also been involved in other transparency issues – the governor resigned this year after criticisms related to his spending and lack of clear explanations on those, and was given a vote of no-confidence. The new governor uncovered additional problems with the ongoing project of relocating the Tsukiji market, the largest fish market in Tokyo, including potential underground water contamination.

tokyo_tower_special_lightup_invitation_for_2020_olympic_games_on_march_2013Image Credit: by t-mizo Tokyo Tower Special Lightup <Invitation for 2020 Olympic Games> (Shibakouen, Tokyo, Japan) (CC BY 2.0)

In the early part of 2017, we will be working towards International Open Data Day 2017. Japan has been one of the most active countries in terms of the number of localities participating in IODD in the past few years (with more than 60 cities participating in 2016!). Some of the issues we will be discussing through this and other occasions include the above-mentioned data plans that the national and prefectural governments will create, as well wider use of Open Spending Next that some of our members have started learning.

Structuring a Global Online Survey – A Question Engine for Open Data Surveys!

Brook Elgie - January 17, 2017 in code, Global Open Data Index, godi, local index, Open Data Index, open data survey, tech

The Global Open Data Index (GODI) is one of our core projects at Open Knowledge International. The index measures and benchmarks the openness of government data around the world. Brook Elgie shares a behind-the-scenes look at the technical design of how we gather the data for the Index through our extensive Open Data Survey and how other organisations can use this survey codebase for their own purposes. The Global Open Data Index Survey is an annual survey of the state of government open data around the world. The survey asks a series of questions about the availability and quality of a set of key datasets. As well as providing a valuable snapshot of the state of open data around the world, it also promotes discussion and engagement between government and civil society organisations. This year Open Knowledge International made changes to the methodology and structure of the survey, and it was an ideal opportunity to revisit the way questions are handled technically within the survey codebase. As well as the survey for the Global Open Data Index, the same codebase hosts surveys for ‘local’ sites, for example, an individual country, or city administration. screen-shot-2017-01-14-at-01-25-05 Previously, the questions presented for each dataset were a hard-coded feature of the survey codebase. These questions were inflexible and couldn’t be tailored to the specific needs of an individual site. So, while each local site could customise the datasets they were interested in surveying, they had to use our pre-defined question set and scoring mechanisms. We also wanted to go beyond simple ‘yes/no’ question types. Our new methodology required a more nuanced approach and a greater variety of question types: multiple-choice, free text entry, Likert scales, etc. Also important is the entry form itself. The survey can be complex but we wanted the process of completing it to be clear and as simple as possible. We wanted to improve the design and experience to guide people through the form and provide in-context help for each question.

Question Sets

The previous survey hard-coded the layout order of questions and their behaviour as part of the entry form. We wanted to abstract out these details from the codebase into the CMS, to make the entry form more flexible. So we needed a data structure to describe not just the questions, but their order within the entry form and their relationships with other questions, such as dependencies. So we came up with a schema, written in JSON. Take this simple set of yes/no questions:
  1. Do you like apples?
  2. Do you like RED apples? (initially disabled, enable if 1 is ‘Yes’)
  3. Have you eaten a red apple today? (initially disabled, enable if 2 is ‘Yes’)
We want to initially display questions 1, 2, and 3, but questions 2 and 3 should be disabled by default. They are enabled once certain conditions are met. Here is what the form looks like: animated_apples And this is the Question Set Schema that describes the relationships between the questions, and their position in the form: Each question has a set of default properties, and optionally an ifProvider structure that defines conditional dependent features. Each time a change is made in the form, each question’s ifProvider should be checked to see if its properties need to be updated. For example, question 2, apple_colour, is initially visible, but disabled. It has a dependency on the like_apples question (the ‘provider’). If the value of like_apples is Yes, apple_colour‘s properties will be updated to make it enabled.

React to the rescue

The form is becoming a fairly complex little web application, and we needed a front-end framework to help manage the interactions on the page. Quite early on we decided to use React, a ‘Javascript library for building user interfaces’ from Facebook. React allows us to design simple components and compose them into a more complex UI. React encourages a one-way data flow; from a single source of truth, passed down into child components via properties. Following this principle helped identify the appropriate location in the component hierarchy for maintaining state; in the top level QuestionForm component. apples_composed Component’s hierarchy for the entry form:
  1. QuestionForm (red)
  2. QuestionField (orange)
  3. Sub-components: QuestionInstructions, QuestionHeader, and QuestionComments (green)
Changing values in the QuestionFields will update the state maintained in the QuestionForm, triggering a re-render of child components where necessary (all managed by React). This made it easy for one QuestionField to change its visible properties (visibility, enabled, etc) when the user changes the value of another field (as determined by our Question Set Schema). You can see the code for the entry form React UI on Github. Some other benefits of using React:
  • it was fairly easy to write automated tests for the entry form, using Enzyme
  • we can render the initial state of the form on the server and send it to the page template using our web application framework (Express)

Developing in the Open

As with all of Open Knowledge International’s projects, the Open Data Survey is developed in the Open and available as Open Source software: Open Data Survey on Github.