Building a Query Microservice to Process Millions of Events per Minute

Description

A cybersecurity provider turned to Altoros to build a scalable data query microservice, ensure stable performance, and enable data visualization for its platform.

Brief results of the collaboration:

• The company developed a data query microservice—supplying data for app behavior analysis, risk prioritization, microsegmentation, and anomaly detection—in just two months.
• Thanks to the chosen architectural approach and tooling, the microservice is capable of processing 6.8 millions of events per minute without impact on performance. It also lays the foundation for future scaling.
• A comprehensive UI enables users to visualise multiple analysis findings in a single chart.
• The startup was able to deliver new features every 2–3 weeks. This allowed to get early feedback and test feature feasibility on the market before showcasing the product to investors. It is expected that series A funding will gather up to $100 million for further development.

The customer

Based in the USA, the customer operates at the intersection of data science and cybersecurity. The company has a flagship platform designed to prevent data leaks and security breaches by monitoring existing infrastructure, detecting anomalies, and accessing risks.

The need

A transition from monoliths to microservices, cloud, and software as a service created numerous connections between app components. The proliferation of API development only increases the number of connections prone to vulnerabilities. In an enterprise system, monitoring these connections, which start at dozens, is anything but a trivial task. Using its expertise in data science and cybersecurity, the customer built a prototype of a platform that monitors API behavior, data flows, and traffic patterns to identify / prioritize risks, as well as detect anomalies. Relying on Altoros, the startup wanted to build a data query microservice and develop a user interface for data visualization. With an upcoming funding round, there were tight deadlines for delivery.

The challenges

Under the project, the team at Altoros had to address the following issues:

• As a data query microservice would aggregate massive massive arrays of data, it was important to ensure its uninterrupted, stable performance.
• The changing risks—identified in infrastructure connections in real time—had to be instantly mapped on the dashboard and prioritized by severity.
• The system monitored infrastructure connections across 300 metrics. For user convenience, multiple metrics visualized in a single chart must not overlap.

The solution

Developers at Altoros were engaged in front- and back-end development, as well as took on project management responsibilities. Our team was actively involved in creating a product roadmap, prioritizing backlog, and outlining recommendations for implementation. Following best practices of Agile development, engineers at Altoros were able to deliver new functionality every 2–3 weeks.

In two months, our developers created a data query microservice, which aggregates information for app behavior analysis, microsegmentation, risk prioritization, and anomaly detection. To find an optimal variant for processing massive arrays of data, the team at Altoros researched and tested a number of big data and analytics tools. As a result, the engineers implemented a pipeline based on Apache Kafka and Apache Druid. To ensure stable performance and fault tolerance, developers at Altoros built an event-driven architecture. Thanks to all this, the microservice is capable of ingesting over 6.8 million events per minute (at peak times, it can process up to 300,000 events per second).

Our developers delivered algorithms that identify risks and classify them as low, medium, and critical. This data is mapped on a dashboard with a 1–100 score assigned to each risk. The engineers also provided a possibility to monitor how the severity of risks changes over different time periods (30 minutes, 2 hours, and 24 hours).

To properly visualize hundreds of metrics, our team worked out a comprehensive UI architecture. In addition, engineers at Altoros made it possible to sort the analysis findings by 86 filters. This way, a user can visualize multiple insights in a single chart.

The outcome

Partnering with Altoros, the customer delivered a data query microservice serving business-critical processes in its product in just two months. Thanks to the chosen architectural approach and tooling, it became possible to process 6.8 million events per minute (at peak times, it can process up to 3x more) without impact on performance. This lays the foundation for future scaling. The startup was able to deliver new features every 2–3 weeks, which provided an opportunity to get early feedback and test their feasibility on the market. This way, the startup was able to build up the functionality to showcase to investors. It is expected that series A funding will gather up to $100 million for further development.