Water Map Popularity in Finland Proved the Importance of Public Maps’ Performance Testing

Finnish Environment Institute (SYKE) has recently launched a very handy map of water quality in Finland, which allows anyone to check the ecological status of Finnish lakes and rivers.  After a news article featuring the online water map was published at the very popular news website YLE, the Finnish Broadcasting Company on 1st of May 2016, the map gained unprecedented popularity up to 85 000 visitors per day wanted to see whether the lake by their summer cottage is ready for after-sauna swim. While being an inspiring use case of open data of SYKE, the unexpected success of the application also resulted in some technical problems for the group running the background services.

We had a chat with Mikko Hynninen, Development Engineer at the Finnish Environment Institute, to find out how they were able to react to the problems and what were the key takeaways from the situation for their team.

The map’s popularity peak happened on a national holiday (Vappu) and the capacity of the map was not initially sufficient for accommodating that many users”, Mr Hynninen told us. “One of the problems was that the exact timing of the media announcement of the map launch was not known, so the capacity of the services was not increased beforehand. Also the particular overloaded services were not technically monitored at the time, which prevented us noticing the traffic increase immediately. Customer feedback about the application problems made us notice pretty soon that there was something wrong.”

SYKE vesikartta

How was the capacity issue solved?

Finnish Environment Institute team was able to react to the service disruption in a timely manner. “As soon as the capacity problem was noticed, our GIS-group increased the service capacity as much as possible”, Mr. Hynninen adds. “The amount of users peaked on Sunday, and by Monday evening we managed to more than double the service capacity. On Monday night we further increased the capacity of our servers. The number of parallel processes in the services were adjusted based on statistics from ArcGIS Server. Our internal systems are isolated from the Internet services, so neither the national environmental administration nor our research activities were not affected by the traffic spike.”

“Successful communication and timing is crucial in making a popular service stand to the expectations”, Mikko summarizes the lessons learned. “After the news article about the water map was published, the number of requests multiplied by 250. Services intended for wide use of the citizens should only be provided using tiled and cacheable maps. We did know this beforehand, but had made an intentional decision not to pre-calculate these map tiles, since the application also required dynamically retrieved data.”

Ilkka Rinne, CTO of Spatineo, a leading European company providing availability monitoring, usage analytics and performance testing tools for Spatial Web Services, comments: “Open data and public web services like the ones driving the Water map of the Finnish Environment Institute are very important for the society and it is great to see how well this map was received by the Finnish citizens. This kind of cases where particular services suddenly become very popular for a short while are becoming commonplace as the online news sites take advantage of the tremendous possibilities in creating new and interesting dynamic maps that combine data from several open services. We are glad to be able to help our customers, such as the Finnish Environment Institute, in preparing for events like this and improving the already high quality of their Spatial Web Services.”

Monitoring of ArcGIS WMTS services

We have noticed a worrying trend among the WMTS services Spatineo is monitoring. Our monitoring is showing that the availability of WMTS services published with ArcGIS is really low. In fact, the availability level for the majority of these services is 0% or just barely above it. These services are provided by a variety of data providers and hosted on different servers, and furthermore other services served by the same ArcGIS servers are working very reliably, so we decided to investigate.

In case you’re not familiar with WMTS, here’s some background information. WMTS services are meant to deliver rendered map images as predefined tiles that may be rendered beforehand and at the very least can be cached after rendering. Predefined tile boundaries and caching makes WMTS services very quick compared to for example WMS. For a client to be able to access the data, these services define what kind of tiles are available in the service. This information is called an image pyramid. A pyramid defines multiple zoom levels (or resolutions) and the number of rows and columns of available tiles for each zoom level. In WMTS, an image pyramid is called a tile matrix set and one zoom level is a tile matrix. Each tile matrix is defined to be valid at rows 0..matrixHeight-1 and at columns 0..matrixWidth-1.

Image pyramid

Image pyramid, image credits

Server implementations often use the same tile matrix definition across different layers and even services as this makes it easier to access multiple layers from these sources. Especially these commonly used tile matrix sets tend to cover very big areas and thus, at high resolutions, a tile matrix may be made up of millions or even trillions of tiles. However, a single layer typically will not cover the entire area of the tile matrix set, so WMTS allows each layer to set up limits for each tile matrix. These limits set minimum and maximum bounds within each tile matrix. So while the entire matrix is valid at rows 0..matrixWidth-1, a layer may be limited to be valid only within minTileRow..maxTileRow, and similarly with columns. These limits play two important roles: to save resources on the server especially if the tiles are pre-generated, and to help clients focus on the relevant geographical areas where there is data available.

ArcGIS WMTS services however, never publish any such limits in their service description. However, in reality there are limits within these services: ArcGIS does serve tiles for all tiles defined in the capabilities document. The WMTS specification states that “The normal response to a valid GetTile operation request SHALL be a tile map that complies with the requested parameters…” (8.2.3, KVP requests) or “In response to a valid request for a Tile representation from a client, a WMTS server SHALL send either an image representation of the tile or a reference to an image…” (10.2.3, RESTful requests). This is to say, if you request a tile which is within the tile matrix set and any defined limits, the server should always respond with an image, or at least a link to an image. Services we’ve analysed either define tile matrices that in reality are not available at all, or return data only for a relatively small area within each tile matrix. All analysed services misrepresent what tiles are available.

Let’s take for example a service from ESRI Finland. You can see the capabilities document here.

This service defines a single layer and a single tile matrix set and no tile matrix limits. According to the service description, the tile matrix set has 13 tile matrices (indexed 0..12). The tile matrix 12 is defined 17411 tiles wide and 10150 tall. According to WMTS the service should return a tile at tile matrix 12, row 0 and column 0. However, the service responds with a HTTP 404 (not found). This is contrary to how WMTS services are supposed to work. You can try the request yourself.

This behaviour can cause unexpected failures in WMTS clients and it is also the reason why Spatineo’s monitoring shows such low numbers of availability for these services. We create varying monitoring requests based on the service description and analyze whether the service is responding according to the WMTS specification. Unfortunately, for most of the legal GetTile requests, these WMTS services published with ArcGIS do not respond correctly.

Fortunately in ArcGIS, you can specify a custom Capabilities document for OGC services. You can do this by following their documentation. With this procedure it is possible to write a Capabilities document that specifies proper limits to the tile matrices and hide the zoom levels that are not available or useful for the service in question. Unfortunately modifying this XML document by hand can be tedious work. But before ArcGIS is fixed, there’s little more you can do if you want your service to interoperate well with WMTS clients.

Want to stay ahead of the game, and get latest news from us? Subscribe now to our newsletter!

Update to SMS alerts

We have switched our SMS operator today in order to improve service to our customers. This change introduces a change that our customers might notice: SMS messages sent from Spatineo Monitor will now be sent from “Spatineo”. Previously alert messages had our switchboard number as the origin of the SMS message.

If you have any comments, we would love to hear any feedback about SMS alerts or our products in general!