Why ensuring the quality is so important?
Every time you open the journey planner to check the bus route home or the weather app to see how the weather tomorrow looks like, you utilize spatial data. Spatial data is a name for the data that is related to location – which is 80% of all the data in the world. Spatial data is increasingly used in all the sectors of the society, varying from smart transportation to environmental protection and citizen services. Spatial data used in applications is delivered more and more by using spatial web services, such as Web Map Service and Web Feature Service.
With human attention span of a few seconds, we expect to access information we need immediately. Even a small delay in the services we use, let alone a service disruption, gets us out of our minds. The modern word of millions of applications supports our impatience: we have the power to choose and there is always another, better and faster service we can switch to. This sets pressure to spatial web service providers: how to provide my users with a service level they are satisfied with? Continuously increasing competition and short resources bring an additional challenge. Organizations must make sure that they focus their resources on the areas that provide the biggest benefits to their target audience with the lowest cost.
To tackle the challenges caused by demanding users, tight competition and short resources, organizations need Key Performance Indicators (KPIs). KPIs are used to help managers to understand whether the business is successful and going towards the strategic and operative goals. In many areas of business, KPIs are a commonly recognized best practice, whereas on the field of spatial data delivery it’s a relatively new thing.
We at Spatineo have defined KPIs for spatial web services. The indicators have been selected based on the insights gained from dozens of spatial web service providers we have cooperated with for several years. Our KPIs aim to answer to the following questions:
- What’s the value the spatial web service provides to the users?
- What’s the value the spatial web service provides to the service provider and society?
- What are the factors hindering the success of the spatial web service?
Our recommended KPIs fall into two main categories: metrics related to quality of service and metrics related to service use (Table 1).
Table 1. KPIs for spatial web services
* indicator for user impact
** indicator for organizational and societal impact
*** indicator describing a success barrier
The indicators presented above are rather challenging to measure and thus we usually need an analytics software, such as Spatineo Monitor (Figure 1), to collect the data for the indicators reliably, efficiently and continuously.
Figure 1. Spatineo Monitor is a service-level monitoring and usage analytics tool for spatial web services
By using Finnish Environment Institute (SYKE) and National Land Survey of Finland (NLS) as case examples, we demonstrate real-life use cases of the key performance indicators for spatial web services. Both the cases describe the impact of spatial web services for the users, service provider and society, as well as the barriers preventing the services to succeed.
Case SYKE – Impact of Open Data & Service Performance
The Finnish Environment Institute (SYKE) published all its environmental data for open use in 2008. Since then, the use of SYKE’s open data has increased remarkably every year. For example, in 2016, when comparing August to July, the number of users of spatial web services increased by more than 50% while the number of requests increased by 32%, demonstrating the power of open data (Figure 2). On a yearly level, this means 62% increase in the number of users compared to the corresponding time phase a year before. Increased external use of environmental data does not just increase the environmental awareness and utilization rate of the data but also creates new services and business opportunities.
Figure 2. The use of SYKE’s environmental data services peaked after they opened the data for public
In summer 2016, the online map showing the water quality in Finnish lakes and rivers, provided by SYKE, rose to popularity after the Finnish Broadcasting Company YLE featured it on its website (Figure 3). Up to 85 000 visitors per day wanted to see whether the lake by their summerhouse is suitable for swimming. While this was a positive use case of open data, the unexpected popularity peak was something the application wasn’t prepared for.
Figure 3. The map showing the water quality in Finnish lakes and rivers rose to popularity after being featured in online media
The peak in user amounts caused major slow-downs and interruptions in the service. Due to this, 30 000 requests from users did not get any data from the service (Figure 4). Since multiple services were running on the same server, the overload of a single service had an impact on the SYKE’s other external services as well. For example, the response time of the Area Management and Limitations service increased by more than 900% while the number of users went down by almost 70% (Figure 5). Very soon after noticing the issue from a customer feedback in the day of the peaking, SYKE increased the server capacity and got everything running normally again.
Figure 4. 30 000 service requests were hindered by service slow-downs and interruptions
Figure 5. An overload of the water map service affected negatively also the quality of the other services running on the same server
The user impact of lacking service performance demonstrates well the importance of continuous monitoring. Keeping track of usage and service level indicators is imperative to identify changes in user amounts and service performance early enough. Thus the service capacity can be increased to meet the user requirements before any harm has occurred or customer reclamations received. Another useful practice is to test the performance of the services before the launch to ensure adequate capacity from the very beginning.
To summarize, in this case example, we introduced various KPIs to demonstrate the impact of spatial web services. The Requests hindered by service interruptions and user impact of service speed improvement indicators described the user impact. Whereas the number of users, number of requests and usage compared to previous years meters were used to show the impact for the service provider and society. Finally, requests hindered by service interruptions and services with the biggest speed changes demonstrated the impact of the factors hindering the success of the services.
National Land Survey of Finland – Impact of Service Popularity & Speed
In December 2016, topographic map, property division and orthophotos were the most used data layers of the spatial web services of the National Land Survey of Finland (NLS) (Figure 6). Identifying the popularity of the services can be used to focus resources, marketing efforts and investments on the services and datasets that provide the most value. This can mean updating and developing the most popular services more often than the less used ones or promoting new services that haven’t found their way to the large audience yet.
Figure 6. Identifying most popular data sets, trending data sets and top origins for service requests can be used for optimizing resources, marketing efforts and investments
Trending datasets (Figure 6) describe the layers that are gaining popularity. This indicator can be used to evaluate the effect of a promotion campaign or to plan the future server capacity needs. Knowing where the users are coming (top origins for service requests) makes it possible to pinpoint sites that can be used as marketing platforms to introduce new services to an existing user base (Figure 6).
A service of a fast response time is not just a matter of user friendliness but also efficiency. When the users can complete their tasks without delays, time and money will be saved. In February 2016, the users of NLS’s spatial web services saved four months of working time in total, thanks to the improvement in service speed (Figure 7). Furthermore, 72% less service requests were hindered by service interruptions.
Figure 7. In February 2016, the users of the NLS’s spatial web services saved 4 months of working time due to the service speed improvement
To summarize, this case example introduced various KPIs to describe the impact of spatial web services. The user impact of service speed improvement indicator was used to demonstrate the user impact. Whereas most popular data sets, trending data sets and top origins for service requests showed the impact for the service provider and society. Finally, the requests hindered by service interruptions indicator was used to describe the impact of the factors hindering the success of the services.
To conclude, measuring is a key to success. This is because it’s the only way to gain fact-based information on your organization’s performance, as well as realize if you’re doing something wrong and fix it before any major harm will occur. A set of Key Performance Indicators is a useful tool for finding the right things to measure – the ones that really matter and are critical to your spatial data delivery business. A good rule of thumb to keep in mind when setting KPIs is to ensure that they’re relevant and measured objectively and continuously.
