docs/DATA_PROVENANCE.md

# AETERNA AI — Data Provenance Documentation
**Version**: 1.0.0  
**Branch**: refactor/jsc-data-integrity-v1  
**Date**: 2026-08-28

> This document classifies every data field used in AETERNA AI by its provenance type.

## Classification Types

| Type | Meaning |
|------|----------|
| `OBSERVED` | Directly measured by an authoritative body (BPS, DLH, BMKG) |
| `DERIVED` | Mathematically computed from observed sources |
| `SYNTHETIC` | Procedurally generated by simulation scripts |
| `EXTERNAL_REALTIME` | Fetched from a live public API |
| `MODEL_OUTPUT` | Produced by an ML or simulation model |
| `UNVERIFIED` | Origin unclear or not yet validated against authoritative source |

---

## Training Dataset: `data/synthetic_spatial_training_data_2024_2025.csv`

> **IMPORTANT**: This dataset is SYNTHETIC SIMULATION DATA, not observed DLH/SIPSN ground truth.
> It was generated by `scripts/generate_real_kecamatan_dataset.py` using procedural simulation.
> All model evaluation metrics derived from this dataset are **Synthetic Benchmarks only**.

| Field | Type | Source | Granularity | Limitations |
|-------|------|--------|-------------|-------------|
| `Tanggal` | DERIVED | Calendar | Daily | Simulated date range 2024-01-01 to 2025-12-31 |
| `Location` | OBSERVED | BPS administrative boundaries | Kecamatan | 44 kecamatan DKI Jakarta |
| `City` | OBSERVED | BPS administrative boundaries | Kota/Kabupaten | Jakarta Pusat, Utara, Barat, Selatan, Timur, Kepulauan Seribu |
| `Population_Jiwa` | UNVERIFIED | Manually entered (claimed to be BPS 2023/2024) | Kecamatan | **Needs validation against official BPS API.** Values have not been cross-checked against BPS publication. See BPS Adapter section. |
| `Normal_Avg_Ton` | SYNTHETIC | Manually defined `base_ton` constants in generator script | Kecamatan | Not derived from DLH/SIPSN measurement. Direct source of target leakage in synthetic evaluation context. |
| `Zone_Type` | UNVERIFIED | Manually categorized zone types | Kecamatan | Not sourced from official Jakarta spatial zoning authority (BAPPEDA/BPRD). Needs validation. |
| `Rainfall_mm` | SYNTHETIC | `np.random.exponential` simulation | Daily/Jakarta-wide | Not real meteorological data for training set. In inference, replaced by EXTERNAL_REALTIME (Open-Meteo). |
| `Rain_Lag_1` | DERIVED | Shifted `Rainfall_mm` by 1 day | Daily | Inherits synthetic limitation from `Rainfall_mm`. |
| `Is_Weekend` | DERIVED | Calendar computation from `Tanggal` | Daily | Deterministic from date. |
| `Hari_Dalam_Minggu` | DERIVED | Calendar computation from `Tanggal` (0=Mon) | Daily | Deterministic from date. |
| `Bulan` | DERIVED | Calendar computation from `Tanggal` | Monthly | Deterministic from date. |
| `Is_Mudik` | DERIVED | Hardcoded Lebaran window (Apr 2024, Mar–Apr 2025) | Daily | Islamic calendar dates approximated, not from BMKG/government calendar source. |
| `Ada_Event` | SYNTHETIC | Manually curated event calendar in generator script | Kecamatan/Daily | Not from official DLH/Dispora/Pemprov DKI event records. |
| `Event_Crowd_Headcount` | SYNTHETIC | Manually defined crowd estimates in generator script | Event/Day | Not from official attendance records (Kepolisian, Penyelenggara). |
| `Volume_Sampah_Ton` | SYNTHETIC | Procedurally generated from `base_ton` + multipliers + noise | Kecamatan/Daily | **Not observed DLH/SIPSN data.** This is the model target in training but is entirely synthetic. |

---

## Inference / Runtime Data

| Field | Type | Source | URL | Limitations |
|-------|------|--------|-----|-------------|
| `Rainfall_mm` (inference) | EXTERNAL_REALTIME | Open-Meteo API | https://api.open-meteo.com/v1/forecast | Free tier, no SLA. Coverage: global at coordinate level. Temporal: daily forecast. |
| `Population_Jiwa` (inference) | UNVERIFIED | Hardcoded in `KECAMATAN_DATABASE` (app.py) | — | Same values as training dataset. Not yet validated against official BPS API. |
| `Normal_Avg_Ton` (inference) | UNVERIFIED | Hardcoded in `KECAMATAN_DATABASE` (app.py) | — | Internal baseline, not from DLH measurement. |
| `Event_Crowd_Headcount` (inference) | DERIVED | `data/event_jakarta_2026.txt` | — | Manually curated. Not from official organizer records. |
| `Volume_Sampah_Ton` (inference) | MODEL_OUTPUT | AETERNA Stacking Regressor / Chronos-T5 | — | Forecast, not observation. Labeled as FORECAST in API response. |

---

## Official Data Sources — Status & Access

### BPS DKI Jakarta (Badan Pusat Statistik)
- **URL**: https://jakarta.bps.go.id/
- **API**: https://webapi.bps.go.id/ (requires registration)
- **Available**: Population data by kecamatan, annual publication
- **Status**: ⚠️ ADAPTER STUB BUILT — credentials not yet configured. See `data_sources/bps.py`.
- **Action required**: Register at https://webapi.bps.go.id/ to obtain API key.

### Open-Meteo (Weather)
- **URL**: https://open-meteo.com/
- **API**: https://api.open-meteo.com/v1/forecast
- **Available**: Free, no authentication required
- **Status**: ✅ ACTIVE — used in inference via `fetch_rainfall_forecast()` in `app.py`
- **Granularity**: Daily precipitation, coordinate-based
- **Limitation**: Forecast accuracy degrades beyond 7 days.

### DLH DKI Jakarta (Dinas Lingkungan Hidup)
- **URL**: https://lingkunganhidup.jakarta.go.id/
- **API**: No documented public REST API found.
- **Available daily waste data**: Not publicly accessible at kecamatan level.
- **Status**: ⚠️ ADAPTER STUB BUILT — no live data available. See `data_sources/dlh.py`.
- **Note**: Aggregate city-level data may be available through SIPSN.

### SIPSN (Sistem Informasi Pengelolaan Sampah Nasional)
- **URL**: https://sipsn.menlhk.go.id/
- **API**: No documented public REST API found. Data available via web interface only.
- **Available**: Annual/periodic city-level aggregate waste tonnage for Jakarta.
- **Status**: ⚠️ ADAPTER STUB BUILT — no live API available. See `data_sources/sipsn.py`.
- **Note**: City-level data from SIPSN MUST NOT be disaggregated to kecamatan level and labeled OBSERVED.

### Jakarta Open Data
- **URL**: https://data.jakarta.go.id/
- **Available**: Various datasets, not specifically daily kecamatan waste tonnage.
- **Status**: ⚠️ Investigated — no daily kecamatan waste volume dataset found.

### UPST / TPST Bantargebang
- **URL**: https://upst.jakarta.go.id/ (informational)
- **API**: No documented public API.
- **Status**: ⚠️ No programmatic access available.

---

## Mode B Validation Status

| Level | Observed Data Available | Action |
|-------|------------------------|--------|
| City (Jakarta) | Partial (SIPSN annual/periodic) | Can be ingested as OBSERVED with city-level granularity only |
| Kecamatan (daily) | **NOT AVAILABLE** | Do NOT disaggregate city-level to kecamatan and label OBSERVED |
| Kecamatan (estimated) | Can be derived | Must be labeled DERIVED or ESTIMATED, not OBSERVED |

**Current evaluation mode**: MODE A — SYNTHETIC BENCHMARK ONLY  
**Mode B status**: NOT YET AVAILABLE — infrastructure built, awaiting authoritative data

---

## Composition Ratios

| Waste Type | Ratio | Source | Classification |
|-----------|-------|--------|----------------|
| Sisa Makanan (Organic) | 50.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED — no specific publication cited |
| Plastik | 22.8% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Kertas | 11.5% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Logam | 2.1% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Kaca | 3.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Tekstil | 4.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Lainnya | 6.0% | Residual | DERIVED |

**Recommended**: Verify composition ratios against official SIPSN or DLH Jakarta annual waste characterization reports.