Scaling a Meteorology Platform Analyzing Hundreds of Sources

Description

A global provider of a weather forecasting system turned to Altoros to rebuild a cloud-native platform for analyzing meteorological data and driving meaningful insights.

Brief results of the collaboration:

• The customer created a weather forecasting platform capable of analyzing meteorological data and delivering predictions with 97% precision.
• A microservices architecture and the AWS-based technology stack ensured ease of maintenance and scalability.
• Thanks to the delivered ETL pipeline, the system can transform unstructured meteorological data—gathered from 100+ weather stations, satellites, radiosondes, lightning detectors, etc.—for further analysis and visualization on synoptic charts.

The customer

Based in Greece, the customer is a provider of weather modification solutions, mainly to the agricultural industry. Founded in 1976, the company is among top 4 firms in its domain. The organization is well known for its hail suppression and precipitation increase (rain/snowfall) projects.

The need

The customer had a web-based app that analyzed meteorological data gathered from 6 types of sources. The latter included hundreds of the company-owned weather stations, radars, radiosondes, and lightning detectors, as well as satellites and other equipment deployed worldwide by international agencies. However, the technology stack behind the app was outdated, impairing performance and maintainability.

Representing mostly meteorology professionals, the firm relied on cloud-native development expertise at Altoros to develop a new robust system with a focus on high-precision data analysis and visualization.

The challenges

Under the project, the team at Altoros had to address the following issues:

• Since the customer planned to expand to the European market and add more sources of meteorological data, it was important to ensure scalability.
• The meteorological data from multiple sources was unstructured, requiring a common format for further analysis and visualization on synoptic charts.
• To visualize up-to-date information, it was crucial to extract the meteorological data from the customer's aggregation server on a regular basis.

The solution

Stage 1. After analyzing the customer requirements, cloud-native engineers at Altoros conducted research around the optimal technology stack and architectural design. The team opted for Amazon Web Services, as it has all the necessary tooling to ensure scalability. In addition, Amazon Elastic Beanstalk helped to automate platform deployment.

A microservices-based architecture contributed to the system’s maintainability, allowing to build new modules without refactoring the whole system.

Stage 2. The data engineers built an ETL pipeline for working with unstructured information, such as meteorological maps (thermodynamic, pressure, streamline/winds, precipitation, etc.). For unification, the data was transformed to the JSON format and then loaded to the analytical module.

Stage 3. After that, the developers at Altoros delivered a scheduler to enable data updates between the customer’s aggregation server and the new system at preconfigured intervals.

Stage 4. In addition, the team created an algorithm that estimated temperature at a specified location—based on the data collected from the diverse sources—with a 97% precision.

Stage 5. The developers at Altoros made it possible to visualize data in a range of synoptic charts that illustrated different weather conditions (rainfall, wind, etc.).

Stage 6. Finally, QA engineers conducted unit, functional, performance, and acceptance tests.

The outcome

Partnering with Altoros, the customer developed a weather forecasting solution that aggregates, analyzes, and visualizes meteorological data on a range of synoptic charts. Gathering data from 6 source types (the WRF Model system, 100+ weather stations, radiosondes, satellites, and lightning detectors), the platform now features 97% prediction accuracy. Thanks to a microservices-based architecture and Amazon technology stack, the system is easy to maintain and scales to aggregate meteorological data gathered across the whole Europe.