Architecting data streaming for ultra-high throughput, dispersed IoT workloads, real-time complex event processing and response
Challenge
Subeca provides intelligent and innovative solutions in the realm of water utility management. Their development in the areas of Amazon Sidewalk and LoRaWAN communications embedded devices provides unmatched customer monitoring and control over their water utilities.
Subeca was looking to develop the core infrastructure necessary to ingest, process and handle data from in excess of 1 million devices. The data in question encompasses a near-real-time use case, wherein data is streamed from IoT devices using a combination of LoRaWAN and Amazon Sidewalk technologies.
Subeca sought out the guidance and assistance of Protagona to build this core infrastructure stack in the image of best practices and scalability for future state.
Solution
Leveraging AWS data streaming technology in Kinesis Data Streams, Protagona have built a near-infinitely scalable mechanism to ingest data from millions of devices simultaneously.
The solution also employs in-flight data processing, which allows adjoining services to react to delta events, which in turn allows for complex event handling and workflow.
Paired with streaming data from AWS IoT core and Amazon Sidewalk, this allows Subeca to operate a processing + alerting framework for their infrastructure in real-time.
Tech Stack
- Amazon Kinesis Data Streams
- Amazon Kinesis Data Firehose
- Amazon Sidewalk
- AWS DynamoDB
- AWS Lambda
- AWS S3
- AWS IoT Core
- AWS IoT Rules
- AWS SQS
- AWS SNS
Outcomes
Real-Time Responses
Subeca achieves real-time event processing and response by efficiently decoding and immediately acting upon received messages. This eliminates lag time, allowing prompt reactions to events, informed decision-making, streamlined operations, and improved customer experiences.
Future-Proofing
The infrastructure stack created by Protagona for Subeca features ingest, processing, storage and database layers that have theoretically limitless capacity for scaling. Meaning that Subeca can guarantee capability to handle all future business expansion without the need to re-architect.
Incredible Cost Efficiency & Performance
By leveraging a combination of in-flight data processing and utilizing the best processing methods available, the solution achieved remarkable results. It surpassed earlier estimates from all parties by delivering faster processing times and cost savings.