Designing data ingestion and data processing pipelines to enable the collection, integration, and transformation of data To scale new customers on HungryFriend’s platform
“The experience and thought-leadership Protagona brought to the table was exceptional. The value provided by the entire team has been instrumental in the advancement of our product architecture and our ability to create new frameworks within the AWS ecosystem to allow us the ability to scale in new ways.” Michael Smallwood- CEO of HungryFriend
Challenge
HungryFriend is a revolutionary platform that unites passionate individuals and socially-conscious restaurants, all in support of causes close to their hearts. By simply linking their payment card, users can effortlessly contribute to their chosen causes while dining at participating restaurants. Central to HungryFriend’s success is their robust data ecosystem, a cornerstone for generating actionable insights and personalized recommendations.
In a world driven by data, companies encounter the complex task of efficiently harnessing and processing extensive data from diverse origins. HungryFriend recognized the need to seamlessly integrate data from standard providers, ensuring scalability and leveraging AWS resources to fuel their mission.
Solution
HungryFriend enlisted Protagona’s expertise to establish a streamlined data integration process powered with AWS services. The focus was on efficiently consuming data from a standard provider while harnessing AWS capabilities for scalability.
- Continuous Data Transfer: An automated job through AWS EventBridge facilitated ongoing data transfer into the raw zone of HungryFriend’s DataLake.
- Data Catalog Generation: AWS Glue was employed to crawl the raw data, generating a comprehensive data catalog. This catalog empowered HungryFriend to execute robust data queries and delve into advanced analytics using Athena.
- Data Processing Pipeline: Protagona crafted a dynamic data processing pipeline utilizing AWS Lambda and EventBridge. This pipeline ingested, parsed, and dispatched data deltas via AWS SQS, effectively distributing key data across the HungryFriend platform and third-party integrations.
Tech Stack
- AWS Lambda
- AWS S3
- AWS Eventbridge
- AWS SQS
- AWS Glue
- Amazon Athena
- AWS Transfer SFTP
Outcomes
Data Integration – Protagona’s integration solution streamlined data acquisition, reducing merchant onboarding and processing times
Scalability – Protagona’s AWS-powered architecture ensures scalability with data pipelines that can be extended across multiple regions as their customer and merchant base scales nationwide.
Embracing Cloud Native Services – By introducing a cloud native solution using AWS best practices with a completely serverless architecture, HungryFriend now has a path to embrace and adopt Cloud Native services in their existing platform
Looking Ahead – Emboldened by the success of this collaboration, HungryFriend and Protagona plan to further amplify data capabilities. Future steps include incorporating advanced analytics and machine learning to enhance user experiences and maximize the impact on supported causes.