Back to Live SCADA Operations

GridPulse Architectural Blueprint & Pipeline EnginePhases 1–11

End-to-end data pipeline, statistical formulations, and engineering defense guide.

Interactive Pipeline Block Architecture

Click any pipeline block below to inspect its technical specification & SLAs:

11 Operational Stages Mapped
Phase 11000 ms (1 Hz)

IoT Edge Smart Sub-Meters

Python Simulator / Modbus RS-485 / MQTT

ingestionInspect
Phase 2< 15 ms

Apache Kafka Message Broker

Kafka 3.7 / Docker Compose

ingestionInspect
Phase 4< 2500 ms

Apache Spark Structured Streaming

PySpark 3.5 / 1-Min Tumbling Windows

processingInspect
Phase 3< 50 ms query latency

Neon Cloud PostgreSQL (Hot Path)

PostgreSQL 16 / Neon Serverless

storageInspect
Phase 5 & 6Sub-second analytical queries

Medallion Data Lakehouse & dbt

Parquet / DuckDB / dbt-duckdb

storageInspect
Phase 5Scheduled (every 6 hours)

In-Place Parquet Compactor

Python / DuckDB Parquet Engine

storageInspect
Phase 7 & 10< 100 ms commit time

Delta Lake ACID Log & MinIO S3

Delta Lake (deltalake) / MinIO S3 / boto3

storageInspect
Phase 9Every 15 mins (freshness) / Daily (ML)

Workflow Orchestration (Airflow)

Apache Airflow 2.8 / Python DAGs

processingInspect
Phase 8< 200 ms inference

AI Demand Forecaster & Peak Shaving

Scikit-Learn (Ridge / XGBoost) / Python

intelligenceInspect
Phase 11< 50 ms per reading

Hostel Ghost Load & NILM Engine

Median & MAD Robust Statistics / NILM State Machine

intelligenceInspect
Phase 4 & 1160 FPS UI rendering

Next.js SCADA Portal & Webhooks

Next.js 16 / React 19 / Recharts / Webhooks

presentationInspect
Phase 11

Hostel Ghost Load & NILM Engine

[Median & MAD Robust Statistics / NILM State Machine]

Detects unattended split ACs and lighting left ON in unoccupied hostel rooms during lecture hours (Mon-Fri 10:00–16:00). Uses robust Median/MAD baselines, 45-min hysteresis state machines, and NILM disaggregation.

Inputs & Data Sources:
  • • Wing sub-meter telemetry (kW, PF, timestamp)
  • • Wi-Fi AP client occupancy density
Outputs & Deliverables:
  • • Tier 1 Ghost Load Incident Alert
  • • Wasted kWh, INR cost, and CO2 emissions
  • • NILM cause classification
Repository Code Location:
analysis/ghost_load_detector.py
Latency Budget SLA:

< 50 ms per reading

Engineering Rationale:
  • • Uses Median & MAD instead of Mean & Std Dev: resists contamination up to 50% from regular historical waste.
  • • 45-minute hysteresis window (Z > 2.5 to accumulate, Z < 1.5 to reset) eliminates false alarms from electric kettles or irons.
  • • On held-out simulated weeks: recall 60-68% and precision 96-100% across 40 randomized episodes; reliable from about 6 kW of extra load per wing.

Hostel Ghost Load Detection: The Non-Intrusive Intelligence Strategy

Addressing unattended power wastage when students leave hostel rooms for lectures without installing invasive or expensive hardware sensors in 500+ rooms.

View Live in Dashboard →
1Robust Median & MAD Baseline

Computes typical power per (day_of_week, 15_min_slot) tuple. Because waste already happens regularly, a Gaussian mean-based baseline learns waste as normal. The median has a breakdown point of 50%, resisting contamination.

2Hysteresis State Machine

Operates with dual thresholds: Z > 2.5 accumulates elevated minutes, while Z < 1.5 resets the buffer. An incident only opens when elevated_minutes ≥ 45, preventing alert flapping from short kettle boils or irons.

3NILM Load Disaggregation

Separates inductive compressor motors (PF ~0.78 with 20–40 min periodic duty cycles) from resistive lighting and study lamps (PF ~0.99 flat load).

Wi-Fi Occupancy Density Cross-Validation (Zero Hardware Addition)

Campus Cisco/Aruba Wi-Fi access points log connected MAC addresses per wing. When wing electrical demand surges to 28 kW during lecture hours while the Wi-Fi controller reports only 1–3 connected devices, this divergence delivers strong empirical evidence of unoccupied waste without invading student privacy.