Best Practice: Holistic Metrics

Outline

The costs of public sector information in respect to added value has to be assessed taking into account large-scale detour effects and not merely at the level of the publishing organisation. This BP clarifies why this is a best practice, what can be the obstacles and approaches to actually implement the BP.

Links to the Revised PSI Directive

Policies and Legislation, Charging

Challenge

The publication of data and information according to regulations, principles, best practices or recommendations generally has a positive effect. However, in cases where the publisher creates a substantial revenue by monetizing the concerned data, the loss of income represents a hard to deny fact which may contribute to the decisions against publishing data or information. If the only metric applied is whether the department that generates and shares the data receives financial compensation for doing so, then the assessment is very likely to be unfavourable.

The PSI Directive mandates that data should be available for free or at marginal cost, that is, making the data available cannot itself be a revenue generator although charges may be levied to cover the difference between keeping the data within the organisation and making it available. Therefore more sophisticated assessments are required as inputs to a publication strategy, taking into account the whole data life cycle. While added efforts or lost income will likely be of an issue for the publisher, a variety of metrics should be used at different levels of the organisation up to the state or community as a whole to assess the value and impact of sharing PSI.

There is also need for a comprehensive ex-post impact assessment of opening up governmental data. The impacts of opening up governmental data can be divided to economic impacts and to other social impacts. The prerequisite for this assessment is a careful development of the monitoring and evaluation model for opening up governmental data as well as a systematic gathering of data for the impact assessment.

Solution

A range of metrics taken at a higher level will often show significant benefits to the organisation as a whole, such as greater efficiency, improved fulfillment of the public task and increased transparency.

Why is this a Best Practice?

Generally, assessing the benefits of actions exclusively at local scale will lead to micro-optimisations and missed opportunities at the larger context. This is not only disadvantageous in the case of deciding on opening up data and information, but is an administrative leftover from times where holistic measure was mostly impossible due to non-existent integrated Information Systems which can provide a quick and comprehensive overview on policy making.

How do I implement this Best Practice?

The actual implementation of public sector value assessment is very much influenced by the administrative organisational setup. If and open data strategy is implemented at the federal level, chances are high, that impact assessment will also take place at that level. However, even in cases the impact is assessed at higher level but the publishing organisation, possible loss of revenues is required either to be compensated or to be approved.

To facilitate the transition from assessing efforts and value from the local level to a higher level, techniques and methodologies of management by objectives or evidence based government could be used. The actual implementation will further depend on the level where the PSI directive got implemented. While all EU member states are required to implement EU directives into national law, some member states forward this obligation to provinces (mostly on NUTS-2 level), which adds additional difficulty when erecting holistic measures of costs and effects of data and information publication. Some individual departments, or sub-departments, are likely to see increased costs with no direct benefit to that department but at a higher level, the benefits should be evident and measurable.

Factors to bear in mind include, but are not limited to:

• the reduction in time spent dealing with individual requests for information;
• the reduction in the load on the Web site due to the elimination of traffic from screen scrapers;
• improved delivery of services, perhaps including new services;
• improved transparency, perhaps measured by the number of news items referring to published information;
• improved data quality due to more users able to detect errors;
• time and money saved by citizens;
• new products based on open data;
• more material available for learning, training and research.

These are just examples of factors to consider when establishing a more complete and holistic model to assess the added value of published information. Often the increased costs of data and information publication are simply measured on a local level, so are the benefits, which are often restricted to the number of downloads of a particular dataset and the number of external applications built on top of the data. As this BP shows, this is to restrictive and has to be assessed on a higher level to yield meaningful insights and actionable results.

Further reading

• The Economics of Open Data Chris Taggart, 2011
• The Value of Open Data, The ODI
• Infonomics Definition, Wikipedia
• The impact of open dataMininstry of Finance, Finland. Heli Koski, Elinkeinoelämän tutkimuslaitos

Where has this best practice been implemented?

References

• Samos Workshop Session: The Potential within the Government for Innovation and Efficiency from Open Data – Examples from the Norwegian public Sector
• Berlin Workshop Session: The Impacts of Open Data: Towards Ex Post Assessment
• Open Data Institute method report: Assessment tools for open data initiatives
• Open Data Institute technical report: Benchmarking open data automatically

Contact Info

Editor: Johann Höchtl, Danube University, Krems
Contribution: Anne Kauhanen-Simanainen, Ministry of Finance, Finland

Related Best Practices

• Establish an Open Data Ecosystem
• Develop an Open Data Publication Plan

Issue Tracker

Any matters arising from this BP, including implementation experience, lessons learnt, places where it has been implemented or guides that cite this BP can be recorded and discussed on the project's GitHub repository