AETERNA AI — Data Provenance Documentation
Version: 1.0.0
Branch: refactor/jsc-data-integrity-v1
Date: 2026-08-28
This document classifies every data field used in AETERNA AI by its provenance type.
Classification Types
| Type | Meaning |
|---|---|
OBSERVED |
Directly measured by an authoritative body (BPS, DLH, BMKG) |
DERIVED |
Mathematically computed from observed sources |
SYNTHETIC |
Procedurally generated by simulation scripts |
EXTERNAL_REALTIME |
Fetched from a live public API |
MODEL_OUTPUT |
Produced by an ML or simulation model |
UNVERIFIED |
Origin unclear or not yet validated against authoritative source |
Training Dataset: data/synthetic_spatial_training_data_2024_2025.csv
IMPORTANT: This dataset is SYNTHETIC SIMULATION DATA, not observed DLH/SIPSN ground truth. It was generated by
scripts/generate_real_kecamatan_dataset.pyusing procedural simulation. All model evaluation metrics derived from this dataset are Synthetic Benchmarks only.
| Field | Type | Source | Granularity | Limitations |
|---|---|---|---|---|
Tanggal |
DERIVED | Calendar | Daily | Simulated date range 2024-01-01 to 2025-12-31 |
Location |
OBSERVED | BPS administrative boundaries | Kecamatan | 44 kecamatan DKI Jakarta |
City |
OBSERVED | BPS administrative boundaries | Kota/Kabupaten | Jakarta Pusat, Utara, Barat, Selatan, Timur, Kepulauan Seribu |
Population_Jiwa |
UNVERIFIED | Manually entered (claimed to be BPS 2023/2024) | Kecamatan | Needs validation against official BPS API. Values have not been cross-checked against BPS publication. See BPS Adapter section. |
Normal_Avg_Ton |
SYNTHETIC | Manually defined base_ton constants in generator script |
Kecamatan | Not derived from DLH/SIPSN measurement. Direct source of target leakage in synthetic evaluation context. |
Zone_Type |
UNVERIFIED | Manually categorized zone types | Kecamatan | Not sourced from official Jakarta spatial zoning authority (BAPPEDA/BPRD). Needs validation. |
Rainfall_mm |
SYNTHETIC | np.random.exponential simulation |
Daily/Jakarta-wide | Not real meteorological data for training set. In inference, replaced by EXTERNAL_REALTIME (Open-Meteo). |
Rain_Lag_1 |
DERIVED | Shifted Rainfall_mm by 1 day |
Daily | Inherits synthetic limitation from Rainfall_mm. |
Is_Weekend |
DERIVED | Calendar computation from Tanggal |
Daily | Deterministic from date. |
Hari_Dalam_Minggu |
DERIVED | Calendar computation from Tanggal (0=Mon) |
Daily | Deterministic from date. |
Bulan |
DERIVED | Calendar computation from Tanggal |
Monthly | Deterministic from date. |
Is_Mudik |
DERIVED | Hardcoded Lebaran window (Apr 2024, Mar–Apr 2025) | Daily | Islamic calendar dates approximated, not from BMKG/government calendar source. |
Ada_Event |
SYNTHETIC | Manually curated event calendar in generator script | Kecamatan/Daily | Not from official DLH/Dispora/Pemprov DKI event records. |
Event_Crowd_Headcount |
SYNTHETIC | Manually defined crowd estimates in generator script | Event/Day | Not from official attendance records (Kepolisian, Penyelenggara). |
Volume_Sampah_Ton |
SYNTHETIC | Procedurally generated from base_ton + multipliers + noise |
Kecamatan/Daily | Not observed DLH/SIPSN data. This is the model target in training but is entirely synthetic. |
Inference / Runtime Data
| Field | Type | Source | URL | Limitations |
|---|---|---|---|---|
Rainfall_mm (inference) |
EXTERNAL_REALTIME | Open-Meteo API | https://api.open-meteo.com/v1/forecast | Free tier, no SLA. Coverage: global at coordinate level. Temporal: daily forecast. |
Population_Jiwa (inference) |
UNVERIFIED | Hardcoded in KECAMATAN_DATABASE (app.py) |
— | Same values as training dataset. Not yet validated against official BPS API. |
Normal_Avg_Ton (inference) |
UNVERIFIED | Hardcoded in KECAMATAN_DATABASE (app.py) |
— | Internal baseline, not from DLH measurement. |
Event_Crowd_Headcount (inference) |
DERIVED | data/event_jakarta_2026.txt |
— | Manually curated. Not from official organizer records. |
Volume_Sampah_Ton (inference) |
MODEL_OUTPUT | AETERNA Stacking Regressor / Chronos-T5 | — | Forecast, not observation. Labeled as FORECAST in API response. |
Official Data Sources — Status & Access
BPS DKI Jakarta (Badan Pusat Statistik)
- URL: https://jakarta.bps.go.id/
- API: https://webapi.bps.go.id/ (requires registration)
- Available: Population data by kecamatan, annual publication
- Status: ⚠️ ADAPTER STUB BUILT — credentials not yet configured. See
data_sources/bps.py. - Action required: Register at https://webapi.bps.go.id/ to obtain API key.
Open-Meteo (Weather)
- URL: https://open-meteo.com/
- API: https://api.open-meteo.com/v1/forecast
- Available: Free, no authentication required
- Status: ✅ ACTIVE — used in inference via
fetch_rainfall_forecast()inapp.py - Granularity: Daily precipitation, coordinate-based
- Limitation: Forecast accuracy degrades beyond 7 days.
DLH DKI Jakarta (Dinas Lingkungan Hidup)
- URL: https://lingkunganhidup.jakarta.go.id/
- API: No documented public REST API found.
- Available daily waste data: Not publicly accessible at kecamatan level.
- Status: ⚠️ ADAPTER STUB BUILT — no live data available. See
data_sources/dlh.py. - Note: Aggregate city-level data may be available through SIPSN.
SIPSN (Sistem Informasi Pengelolaan Sampah Nasional)
- URL: https://sipsn.menlhk.go.id/
- API: No documented public REST API found. Data available via web interface only.
- Available: Annual/periodic city-level aggregate waste tonnage for Jakarta.
- Status: ⚠️ ADAPTER STUB BUILT — no live API available. See
data_sources/sipsn.py. - Note: City-level data from SIPSN MUST NOT be disaggregated to kecamatan level and labeled OBSERVED.
Jakarta Open Data
- URL: https://data.jakarta.go.id/
- Available: Various datasets, not specifically daily kecamatan waste tonnage.
- Status: ⚠️ Investigated — no daily kecamatan waste volume dataset found.
UPST / TPST Bantargebang
- URL: https://upst.jakarta.go.id/ (informational)
- API: No documented public API.
- Status: ⚠️ No programmatic access available.
Mode B Validation Status
| Level | Observed Data Available | Action |
|---|---|---|
| City (Jakarta) | Partial (SIPSN annual/periodic) | Can be ingested as OBSERVED with city-level granularity only |
| Kecamatan (daily) | NOT AVAILABLE | Do NOT disaggregate city-level to kecamatan and label OBSERVED |
| Kecamatan (estimated) | Can be derived | Must be labeled DERIVED or ESTIMATED, not OBSERVED |
Current evaluation mode: MODE A — SYNTHETIC BENCHMARK ONLY
Mode B status: NOT YET AVAILABLE — infrastructure built, awaiting authoritative data
Composition Ratios
| Waste Type | Ratio | Source | Classification |
|---|---|---|---|
| Sisa Makanan (Organic) | 50.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED — no specific publication cited |
| Plastik | 22.8% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Kertas | 11.5% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Logam | 2.1% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Kaca | 3.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Tekstil | 4.2% | Cited as DLH DKI Jakarta statistics | UNVERIFIED |
| Lainnya | 6.0% | Residual | DERIVED |
Recommended: Verify composition ratios against official SIPSN or DLH Jakarta annual waste characterization reports.