The Curious Case of Insufficient Data – March 2018 Monitoring Disruption

There are times when you have to be honest to people that rely on you, and say you’re sorry to have let them down. This is how I’ve felt more than once during the last couple of weeks, as the alerting system of Spatineo Monitor has been sending a lot of uncalled-for alerts about “insufficient data” to many of our customers.

The core issue of writing the monitoring data to our database fast enough has now been fixed, and we have not sent any of these alerts since 4th April 2018. As these cases go, we could list a number of reasons for why this happened, but it does not change the fact that we should have done better. The entire Spatineo team, and particularly me be as the Head of Customer Experience, are truly sorry for the inconvenience caused by these inbox-filling alerts that had nothing to do with the how your services were doing, and we will contact the customers badly affected by this with an offer for compensation. Please contact our sales or support if you have any questions this blog post does not give an adequate answer for.

So you sent out thousands of superfluous alert messages – what happened?

Ironically, our alerting system was functioning exactly as it’s supposed to be all the time. In addition to the usual alert conditions in service availability, such as too slow average response time or too error results returned too often, we also create alerts when we no longer can reliably say if the service is down or not based on the particular way of measuring it: In technical terms, one possible state for an alert is insufficient data. This normally happens when we stop measuring the service in the way configured for the alert indicator, due to detecting that the particular combination of request parameters for measuring the service is no longer valid according to the updated service capabilities.

Let’s say a service health measurement is configured to use a particular WMS layer, and this layer disappears from the capabilities document of the measured service. We will automatically stop measuring the server using this now invalid meter, and if there are alert indicators bound to this meter, they will change into the “insufficient data” status after a while due to missing measurement data. We do try to keep measuring the service still, but since the layer is now different the same alerting thresholds may not be appropriate anymore, so the Monitor user should manually configure the alert limits for the new combination of layer, coordinate system, image size and format.

Monitoring tens of thousands of web services every 5 minutes is hard work – occasionally too hard for our servers to keep up for months in a row. To make sure our monitoring servers are producing a continuous flow of measurement data, and are not stuck, we continuously monitor all of them and reboot them if necessary. What happened several times between 23rd March and 4th April, was that we could not write the monitoring results into our monitoring database as fast as the data was collected. This caused our internal “watch dog” process to think that many of our monitoring agent servers were not functioning, and to react by rebooting those services automatically. Usually this does not cause any trouble, as the rebooting a single server occasionally has little effect on the whole process. When this happens to many servers repeatedly whoever, we sometimes end up with small gaps in the 5-minute-interval monitoring time series. These gaps were then correctly noticed by the alerting system as missing data for the particular alert indicator, and thus the “insufficient data” alert was sent out. As soon as the data stream was restored, the alert condition turned back to OK again.

While logical in hindsight, it took us quite a while to figure out that the slow data writing operations to the database were the core reason causing for all the trouble we, and unfortunately also our customers, were witnessing. When that became clear the fix was simple: we added more processing power to the database servers and the issue disappeared. Sigh.

Lessons learned the hard way

To prevent things like this from happening in the future we have now added a few more critical performance metrics for our monitoring system and we are using to track the system health, and to react quickly if things start looking bad. So now we are quite a bit better prepared for helping you keep your services running with high quality.

Once again, my sincerest apologies for the inconvenience caused on behalf of the whole technical team of Spatineo. I would like to thank all of our customers for continued support, and assure you that we are continuously working to improve the reliability and usefulness of our tools. We could not do this without your help and feedback.

Yours,

Ilkka Rinne
Head of Customer Experience and Interoperability
Spatineo Oy

Our story at the World Bank Land & Poverty Conference 2018

There were a number of very high quality papers and interesting projects being presented during the WBLP Conference. Most of them were about issues related to land administration and how to make it more effective. But one topic in particular dragged our attention.

WBLP LandConf2018 Spatineo

Impact of geospatial systems

We were invited to participate in the Experts Meeting titled “Improving Resilience and Resilience Impact of National Land and Geospatial Systems. The project is being coordinated by Dr. Abbas Rajabifard, Director of the the CSDILA: Centre for Spatial Data Infrastructures and Land Administration – University of Melbourne, Australia.

The project objective is to define the rationale, approach, and methodologies for improving the resilience and resilience impact of national land administration and geospatial systems by:

  1. Consolidating international cross-sectoral knowledge on land administration, geospatial systems and resilience.
  2. Developing an approach to produce country-level action plans for increasing resilience and resilience impact of national land and geospatial system.
  3. Producing five country-level action plans.
  4. Disseminating the results within the World Bank and globally through a Flagship Report and an International Forum.

During this meeting, the Inception Report was presented, which included a Disaster Risk Management, Land, and Geospatial literature review, international best practices and the analytical framework that support the development of the Flagship Study. As well as, there were discussions on the country-level case studies (Nicaragua, Nigeria, Norway, India, and Turkey) and respective action plans.

Our intake on Impact of geospatial systemsSpatineo Monitor

Where does Spatineo fit into this context? If you read just a little about us you certainly noticed that we work at the following areas in the geospatial business: Quality assurance of spatial web services and Impact Assessment of Spatial Data Infrastructures. Our solutions for quality assurance are used to guarantee that spatial web services are up and running 24/7 at the performance level they were assigned for. In emergency situations derived from disasters, access to reliable geospatial information is key to any effective response management plan. In general, we say that data is useful only if you have reliable access to it and during such undesired events this is even more crucial.

With our unique methodology for impact assessment, organisations responsible for spatial data infrastructures can identify key indicators from their strategic goals, then measure whether the society is receiving the expected value from the geospatial data. We look forward to find ways to cooperate with this project and add value to those communities affected by it.

Spatineo Impact process chart simple version

If you’re interested in assuring quality of your services or assessing impact of your SDI, start showcasing your success with Spatineo now – 14-day free trial of Spatineo Monitor

 

GDPR- What Does it Mean to Our Customers?

GDPR Spatineo Information Security kuva

EU is launching a new law related to information security which becomes effective on May 28th 2018. Spatineo is currently updating company’s information security evaluation reports to comply with the law. There are many things to take into consideration and this law, GDPR,  will give our customers many rights regarding their personal information.

This GDPR law concerns personal information. Personal information is considered to be for example

  • Log information
  • Location information
  • Emails
  • Or anything else that can be linked to a living person

The building and updating is quite a big and time consuming process. But we are focusing to this tasks because it will be beneficial not only to our customers, but for us too. With these reports we can follow and update the information security of customer records. For us, this law means also better personal information security from our partners and associates.

How Does Spatineo handle Information Security?

We are updating our information security reports to give to our customers detailed information on how we handle, collect, preserve and share all the data of our customers. The law requires us to define our information security chief and also, the responsibility on how we use the data. Plans for security breaches are good to have just in case, even though it’s a minor risk.

Spatineo Monitor produces usage analytics information for our customers. In short this means that we produce statistics on how our customers’ spatial web services are being accessed, which countries their end users are from etc. This analysis is done based on IP address information that is included within log files we process for our customers.

Some IP addresses can be used to identify a single person, and therefore we must consider IP addresses in log files as private information. However we have never identified or analyzed the activities of individual end users of spatial web services and therefore do not require full IP addresses to provide our services. This is why we are providing a tool that will anonymise the log files. The anonymisation is done before files are sent for processing to avoid both sending and storing private information. This helps protect the private data of all citizens using spatial web services.

What are the rights then?

Customers and interest groups are entitled to and has the rights to know all the information we collect, have and preserve. Customers are also entitled to receive detailed information on how personal information is handled in Spatineo.

Customers are entitled to:

  • Prohibit the usage of information
  • Correct the wrong information
  • Restrict used information
  • Resist the handling of information
  • Transfer personal information to another register
  • Resist any automated decisions

GDPR Spatial data - Spatineo

What happens next?

After updating our reports we will publish those on our website. At this moment everyone can check our Terms and Conditions of use and privacy statement for the web site of Spatineo Inc. We will inform our progress on our website.

Does Our Customers Need to Take Any Action?

The answer is no. We will do all the required updating regarding handling information and then publish it on our website for everyone to study. Our customers can just sit back and relax, and be confident that the information security is our top priority mission.