README.md

# ACE2S-SHiELD+ reference data

This bucket contains reference data used in training and evaluating models
described in the [ACE2S-SHiELD+ manuscript](https://arxiv.org/abs/2606.07928). 
The reference data is derived from data generated from simulations with GFDL's 
SHiELD model. All simulations were configured identically, with the exception of
the initial conditions, ocean boundary conditions, and the concentration of 
CO<sub>2</sub>. We refer readers to the 
[ACE2](https://www.nature.com/articles/s41612-025-01090-0), 
[ACE2-SOM](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2024JH000575), 
and [ACE2S-SHiELD+](https://arxiv.org/abs/2606.07928) papers for more specific 
details regarding these runs, but a high-level summary of this bucket's contents
is provided in the table below.

| Directory                      | Description                                                                                                                                                                                                   | Time range                                                                                                                          |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `2pctCO2-ensemble`             | 2-member ensemble of 70-year ~100 km slab-ocean-coupled SHiELD simulations with CO<sub>2</sub> increasing at a rate of 2% per year from 1x to 4x the present-day concentration.                               | Spin-up<sup>1</sup>:  2030-01-01T06:00:00 through 2031-01-01T00:00:00<br><br>Main: 2031-01-01T06:00:00 through 2101-01-01T00:00:00  |
| `AMIP-constant-CO2`            | ~42-year ~100 km SHiELD AMIP simulation with CO<sub>2</sub> fixed at the 1979 level.                                                                                                                          | Spin-up: 1978-10-01T06:00:00 through 1980-01-01T00:00:00 <br><br> Main: 1980-01-01T06:00:00 through 2021-12-16T12:00:00             |
| `AMIP-ensemble`                | 2-member ensemble of ~82-year ~100 km SHiELD AMIP simulations.                                                                                                                                                | Spin-up: 1939-10-01T06:00:00 through 1940-01-01T00:00:00 <br><br> Main: 1940-01-01T06:00:00 through 2021-12-16T12:00:00             |
| `AMIP-plus-4K`                 | ~42-year ~100 km SHiELD AMIP simulation with the SST uniformly increased by +4 K.                                                                                                                             | Spin up: 1979-01-01T06:00:00 through 1980-01-01T00:00:00 <br><br> Main: 1980-01-01T06:00:00 through 2021-12-16T12:00:00             |
| `abrupt-4xCO2`                 | 10-year ~100 km slab-ocean-coupled SHiELD simulation with CO<sub>2</sub> abruptly quadrupled at the start of the run.                                                                                         | Main: 2020-01-01T06:00:00 through 2030-01-01T00:00:00                                                                               |
| `abrupt-4xCO2-ensemble`        | 36-member ensemble of 90-day ~100 km slab-ocean-coupled SHiELD abrupt 4xCO<sub>2</sub> simulations started from initial conditions separated by one month over the course of 3 years.                         | Main: full 90-day periods of each store, which start one month apart for each ensemble member starting with 2031-01-01T06:00:00.    |
| `equilibrium-climate-ensemble` | 5-member ensembles of spun-up 10-year ~100 km slab-ocean-coupled SHiELD simulations with 1x, 2x, 3x, and 4x the present-day CO<sub>2</sub> concentration.                                                     | Spin-up<sup>2</sup>: 2030-01-01T06:00:00 through 2031-01-01T00:00:00 <br><br> Main: 2031-01-01T06:00:00 through 2041-01-01T00:00:00 |
| `random-CO2-ensemble`          | 3-member ensembles of 5-year ~100 km SHiELD ramped-SST-random-CO<sub>2</sub> simulations with random CO<sub>2</sub> trajectories logarithmically centered about 1x, 2x, and 4x the present-day concentration. | Spin-up: 2019-10-01T06:00:00 through 2020-01-01T00:00:00 <br><br> Main: 2020-01-01T06:00:00 through 2025-01-01T00:00:00             |

For how these datasets were used in training and/or evaluation in the
ACE2S-SHiELD+ manuscript, see the configurations in the `ACE-experiments`
directory of the
[`ace2s-shield-plus-paper` repository](https://github.com/ai2cm/ace2s-shield-plus-paper)
on GitHub.

## Opening data directly with xarray

With the 
[`huggingface_hub` Python package](https://github.com/huggingface/huggingface_hub#use-the-python-library) 
installed in your environment, you can open and interact with these datasets 
directly with `xarray`. There are no egress fees and therefore no credentials 
are required. Opening a dataset is as simple as:

```
>>> import xarray as xr
>>> STORE = "hf://buckets/allenai/ACE2S-SHiELD-plus-reference-data/random-CO2-ensemble/ramped-sst-1xCO2-random-perturbation-ic_0001.zarr"
>>> ds = xr.open_zarr(STORE, chunks={"time": 360})
```

Here `chunks={"time": 360}` is specified to override the native inner chunk size
along the `"time"` dimension, which is `1` for optimal performance during 
training. For analyses that depend on aggregations across the `"time"` dimension
it is significantly more performant to use the shard size, in this case `360`,
as the dask chunk size. If your use-case more resembles the training regime,
then you should likely omit it.

## License

This dataset is licensed under [CC BY
4.0](https://creativecommons.org/licenses/by/4.0/legalcode.en). It is intended
for research and educational use in accordance with Ai2's [Responsible Use
Guidelines](https://allenai.org/responsible-use).

---

<sup>1</sup> Note `ic_0001.zarr` omits its spin-up period.<br><sup>2</sup> Spin-up data is only provided for select runs and is split off from the `main` segments in the separate `spin-up` subdirectory.