HPWREN Environment Sensor & Ingestion Pipelines

Background & Business Challenge

The High Performance Wireless Research & Education Network (HPWREN) provides statewide environmental sensors, weather stations, and cameras monitoring wildfires across California. The system is heavily relied upon by universities, researchers, and first responders during emergency operations.

The problem: Over 20 years of continuous camera recordings and sensor telemetry resulted in terabytes of raw files stored across custom, on-premises network storage. The existing file manipulation framework relied on monolithic Perl scripts that were slow, difficult to modify, and expensive to scale. The university needed to transition to a modern, cost-efficient, and secure cloud storage pipeline.

Solutions Architecture & Pipelines

We designed and deployed a serverless, event-driven architecture utilizing AWS components to replace the legacy system entirely. Raw files are now ingested via secure channels, processed asynchronously, and indexed directly in DynamoDB for real-time querying.

• AWS DataSync & S3 Transfer Acceleration: Automated ingestion pipelines to safely push historical on-premises archives and ongoing inputs into secure S3 buckets.
• S3 Intelligent-Tiering & Lifecycle Policies: Configured S3 rules that transition historical, rarely-accessed image batches into low-cost glacier classes, saving thousands in storage overhead.
• Event-driven Lambda & Step Functions: Replaced the monolithic Perl cron jobs with modular Python handlers. These run automatically on S3 object creation triggers to perform image resizing, frame aggregation to time-lapse videos, and telemetry extraction.

Outcomes & Deliverables

The migrated environment represents a significant advancement in data durability and access control for California's emergency response networks:

• 40% Overhead Reduction: Replacing virtual machines and storage arrays with serverless functions eliminated active infrastructure management tasks and lowered compute costs.
• Enhanced Telemetry Access: CloudFront content distribution allows global researchers to query high-resolution sensor feeds and wildfire time-lapses in milliseconds.
• Centralized Security Trails: Configured AWS CloudTrail, KMS key rotation, and CloudWatch tracking to guarantee full data traceability and compliance.

Technologies Used