In the modern world, we take for granted many things that are only possible in the cloud. We have quick scaling that allows us to add or remove resources that are needed, high performance and data security. All these things are enabling us to make our data flow more efficient with the cloud. But what do we mean by data flow and how it benefits you?
What is data flow?
Data is the lifeblood of any organisation or enterprise. Without it, you cannot complete any tasks that are valuable to your business – product development, sales & marketing, management or other critical processes.
We’ll be using some geodata-related technologies as an example in this blog since we have quite a sophisticated understanding of them. Let’s say you are in the need of publishing your organisation’s spatial data via an open API. You would be using this data yourself in your daily activities and other stakeholders would greatly benefit from getting their hands on this dataset as well.
Data flow means utilizing data between all of these processes efficiently.
Setting up an efficient data flow
Imagine that you have an open API that is publishing spatial data from your internal systems. This data is then consumed by various applications within your organisation (or maybe even external organisations and enterprises). What can happen in this scenario is that one application will use up all the available resources in the cluster and publish too many requests as a result.
Let’s say that your service gets national coverage in the news, and people flock to see it for themselves. A normal onsite server would most likely crumble in a matter of minutes, but cloud-based solutions can just react to this by upscaling. This is a realistic scenario which happened a few years ago, and we actually wrote a blog about this!
Besides great flexibility, the cloud also offers a vast amount of storage space if needed. You might want to consider storing some large datasets in “cold storage” where you basically pay cents a year to have the data. Retrieving the data might cost a couple of euros, but it is still way cheaper than alternative storage methods. So maybe deposit your large dataset to the cold storage, and only retrieve what is needed with flexibility.
AWS vs. Azure vs. GCP
Cloud providers like Amazon Web Service (AWS), Microsoft Azure and Google Cloud Platform provide scalable computing resources on demand. AWS offers storage options like Amazon Simple Storage Service (S3) and Amazon Glacier. Google and Azure have similar storage options as well, so basically, all three have the same sort of capabilities.
The best choice will completely depend on your needs and budget of course. If you have a relatively small project with low traffic you won’t need a sophisticated solution as a Fortune 500 company, but you’ll still get just as efficient results as larger organizations! Smaller-scale projects might be better suited for a free tier service provided by AWS or GCP.
For data storage, AWS Glacier might be a good option in the long term. You can store data first to the S3 and move it to Glacier later if you access it less frequently and want to save on costs.
Azure and GCP have similar options, but the storage and pricing have a small variance among the competitors.
So ultimately the choice depends on the project you’re working on, and what service you are accustomed to using previously.
Containers work in several environments – Dock it and good to go!
Containers can be used in many environments from local to remote environments. As a lot of people use Containerisation technology, you might need to create a containerized application to work in remote environments. We wrote a post about this a while ago as well – have a look at it here: https://www.spatineo.com/how-aws-enables-automation-of-publishing-geospatial-data/
Geospatial data in the cloud
At some point, you might want to move some of your geospatial data into the cloud since it’s ultimately the superior option in price and efficiency-wise. Open-source technologies help with this.
You can use Geoserver to store your data in the cloud. It’s free and open-source software so it is very accessible and a great alternative to other options.
Your data is ingested by the backend of your choice. It can be a relational database management system such as MySQL or Postgres. It can also be a NoSQL database such as MongoDB Atlas.
A method of our choice has been for example the following: wrap a GeoServer spatial data API server application and the PostgreSQL/PostGIS database into their own Docker containers and set up an automation pipeline to deploy them into the Amazon cloud.
This is just one method we’ve accomplished with a well-polished spatial data pipeline or “data flow” to the cloud, and we are currently building many different variations of this for our customers. In case you are interested in getting your data to flow efficiently, feel free to send us a message!
Want to get your data flowing with ease? Get in touch with us now!
- Spatineo auttoi Puolustusvoimia paikkatietoaineistojen hankinnan automatisoinnissa
Spatineo auttoi Puolustusvoimia paikkatietoaineistojen hankinnan automatisoinnissa Puolustustusvoimien tavoitteena on tehostaa paikkatietoaineistojen hankinta-, käsittely- ja jakeluprosesseja mm. kehittämällä em. prosesseihin olemassa olevia automatisointimenetelmiä ja luomalla uusia. Toimeksiannossa tunnistettiin hankintaprosesseista automatisoitavia vaiheita ja asetettiin automatisoinnit potentiaaliseen toteutusjärjestykseen. Järjestyksen kriteereinä käytettiin aineistohankintaan nykyisin kuluvaa suhteellista aika-arviota ja automatisointiin kuluvaa suhteellista työmääräarviota. Työssä tunnistettiin […] - Who are we? – Suvi-Tuulia “Suffa” Haakana
Suvi-Tuulia, also known as Suffa, is our latest team member, and her expertice is focused on GIS and especially ESRI solutions. What does she do in her normal working day, and what are her favorite GIS tools? - Spatineo recognized as one of Global Top 100 Geospatial Companies
Spatineo has been selected as one of the Global Top 100 Geospatial Companies of 2023. Read more about us and how we did it from this blog! - What Is Geospatial Intelligence?
In this guide, we explore more about geospatial intelligence, what it is, use cases, and examples, so keep reading to learn more. - How to Make Data Flow in the Cloud
If you only knew what your geospatial data would be capable of achieving in the cloud. If you want to make your data flow more efficiently, you need the cloud. - Who are we? – Tiina Le
Tina Le is Spatineo’s HR Specialist, and her work involves making Spatineo a great place to work! Read more about Tiina and her work from out blog!
