Business Implications
Delivers timely market insight with minimal ops: streaming ingestion, low-latency lookups, cheap historical analytics, and actionable alerts. The pattern is reusable for additional tickers and metrics, supports cost-controlled scaling, and forms a strong foundation for production-grade, event-driven data products.
Steps Performed
Provisioned Kinesis, streamed live quotes, processed with Lambda to S3/DynamoDB, cataloged data with Glue, queried via Athena, and emitted SMA-based trend alerts through SNS
1.
Set Up Kinesis Stream
Created stock-market-stream in On-demand mode for elastic, free-tier-friendly throughput. This serves as the ingestion backbone receiving JSON records (OHLC, prev close, volume, timestamp) every ~30 seconds.
2.
Local Python Producer
Configured local Python (boto3, yfinance) and AWS CLI credentials. Implemented stream_stock_data.py to fetch AAPL OHLC/volume, compute Δ and Δ%, and put_record into Kinesis at a fixed cadence.
3.
Process & Persist (Lambda)
Deployed Lambda (Kinesis trigger, small batch) to: land raw JSON to S3 (raw-data/<symbol>/timestamp.json), and write processed records (metrics, moving average, anomaly flag) into DynamoDB (stock-market-data, PK symbol, SK timestamp).
4.
Query Historical Data (Athena)
Registered S3 raw zone in Glue Catalog (stock_data_db.stock_data_table, JSON schema). Pointed Athena to an S3 results bucket and executed SQL (top movers, avg volume, anomaly filters) over the raw lake data.
5.
Real-Time Trend Alerts (SNS)
Enabled DynamoDB Streams (New image) and built TrendAnalysis Lambda. Computed SMA-5 vs SMA-20 crossovers from recent records and published SNS alerts (Email/SMS) for BUY/SELL signals.
AWS Services Used
Amazon Kinesis Data Streams
AWS Lambda
Amazon DynamoDB
Amazon S3
AWS Glue Data Catalog
Amazon Athena
Amazon SNS
AWS IAM
Python (boto3, yfinance)
Docker/Local dev shell (optional)
AWS CLI
SQL (Athena)
Technical Tools Used
Event-driven streaming & serverless data engineering
Low-latency NoSQL modeling on DynamoDB
S3 data lake & Athena querying
Alerting with SNS and Lambda
IAM least-privilege design
Skills Demonstrated
Near-Real-Time Stock Analytics Pipeline (AWS Kinesis)
Stream, process, store, query & alert—serverlessly
Built a near real-time stock analytics pipeline using Amazon Kinesis, AWS Lambda, DynamoDB, S3, Athena, and SNS. The system ingests tick data, computes metrics/anomalies, stores raw + processed datasets, enables SQL analytics on S3, and sends trend alerts—low-ops and cost-efficient.
