AWS Real-Time Streaming Pipeline
Cloud data pipeline project using AWS services and Snowflake to support scalable ingestion and delivery workflows.
Problem Solved
I wanted hands-on experience building a more scalable cloud-based pipeline pattern for data ingestion and delivery than my earlier local and SQLite-focused projects provided.
Outcome
Built and deployed a data pipeline using AWS API Gateway, Lambda, and S3, with supporting Snowflake and dbt components for downstream data workflow development.
Technologies
This project expanded my portfolio experience in data engineering by utilizing the cloud environment.
In particular, I aimed at processing data through a more realistic ingestion and delivery pipeline that utilized AWS infrastructure, not just local ones. The core components of this approach were API Gateway, Lambda, and S3, although Snowflake and dbt were also incorporated into the flow.
By doing this, I acquired useful experience in designing service-oriented pipelines, demonstrating an evolution from analysis-related projects to cloud data engineering ones.