README.md

---
viewer: false
pretty_name: warc2zip examples
source_datasets:
  - original
tags:
  - warc
  - archives
  - warc2zip
  - commoncrawl
---

# warc2zip examples

Small, self-contained WARC files for trying out and testing
[warc2zip](https://github.com/commoncrawl/warc2zip).


## Files

| File | Size | Source crawl | Records | Contents |
|------|-----:|--------------|--------:|----------|
| [`500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz?download=true) | 13 MB | CC-MAIN-2026-25 (June 2026 wide crawl) | 500 captures | Slice of a regular Common Crawl WARC: `response` **+** `request` **+** `metadata` records for each capture |
| [`HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz?download=true) | 1.07 GB | CC-MAIN-2026-21 | 35,700 captures | Homepages only (`response` records only) |
| [`IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz?download=true) | 427 MB | CC-MAIN-2025-13 | 35,700 captures | US federal government hosts, prepared for the [End of Term Archive](https://eotarchive.org/) (`response` records only) |

The file names follow Common Crawl's repackage convention, `<FLAVOUR>-REPACKAGE-<CRAWL-ID>.warc.gz`,
the same scheme used for e.g. `EOT-2024-REPACKAGE-CC-MAIN-2024-42-GOV-000000-001.warc.gz` in the End of Term Archive.

### warc2zip outputs

For each WARC above, the ZIP archives produced by `warc2zip` are also provided, so the expected output can be
inspected without running the tool. Three variants exist per WARC, matching the tool's output modes:

- **`.zip`** — the default *flat* format (`--format flat`): one payload file per `response` record, named
  `1000000.html`, `1000001.html`, … directly under the top-level directory, plus the CSV/JSONL header tables.
- **`.metadata-only.zip`** — flat format run with `--metadata-only`: the header tables and `warcinfo` only, no payloads.
- **`.sidecar.zip`** — the *sidecar* format (`--format sidecar`): payloads grouped into one directory per host, with
  the WARC and HTTP headers of every record written as small text files next to each payload (see
  [Sidecar layout](#sidecar-layout)). The header tables are still generated.

| File | Size | Entries | Contents |
|------|-----:|--------:|----------|
| [`500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip?download=true) | 13 MB | 516 | 500 payloads (490 `.html`, 5 `.pdf`, 5 other) + `warcinfo` + 13 CSV header tables |
| [`500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip?download=true) | 806 KB | 16 | `warcinfo`, `manifest.jsonl`/`manifest.csv` and header tables only; no payloads |
| [`500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip?download=true) | 15 MB | 3,516 | 500 payloads in 479 host directories, each with 6 sidecars (`.response.*`, `.request.*`, `.metadata.*`) + `warcinfo` + 13 CSV header tables |
| [`HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip?download=true) | 1.08 GB | 35,716 | 35,700 payloads (35,637 `.html`, 63 other) + `warcinfo` + 13 CSV header tables |
| [`HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip?download=true) | 35 MB | 16 | `warcinfo`, `manifest.jsonl`/`manifest.csv` and header tables only; no payloads |
| [`HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip?download=true) | 1.13 GB | 107,116 | 35,700 payloads in 35,684 host directories, each with 2 sidecars (`.response.warc`, `.response.http`) + `warcinfo` + 13 CSV header tables |
| [`IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip?download=true) | 434 MB | 35,716 | 35,700 payloads (all `.html`) + `warcinfo` + 13 CSV header tables |
| [`IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metdata-only.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metdata-only.zip?download=true) | 36 MB | 16 | `warcinfo`, `manifest.jsonl`/`manifest.csv` and header tables only; no payloads |
| [`IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip`](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip?download=true) | 494 MB | 107,116 | 35,700 payloads in 6 host directories (35,420 under `www.voanews.com`, the rest under `*.pancanal.com`), each with 2 sidecars + `warcinfo` + 13 CSV header tables |

Inside each ZIP, everything sits under a single top-level directory derived from the `WARC-Filename` of the source
WARC plus a conversion timestamp and short hash, e.g. `homepages_CC-MAIN-2026-21_20260907T143504_b3c3/`
(see [Things to know](#things-to-know)).

#### Sidecar layout

In the `.sidecar.zip` files each capture is placed in a directory named after the host of its `WARC-Target-URI`,
and every WARC record belonging to the capture is unpacked into text files named by appending a suffix to the payload
file name. Headers are kept in raw `Name: value` form. For a capture from the 500-record file, which retains
`request` and `metadata` records, this gives seven entries:

```
CC-MAIN-20260618163205-20260618193205-00999_20260908T123325_2c87/
├── 010mes.com/
│   ├── 1000000.html                        payload
│   ├── 1000000.html.response.warc          WARC headers of the response record
│   ├── 1000000.html.response.http          HTTP response headers
│   ├── 1000000.html.request.warc           WARC headers of the request record
│   ├── 1000000.html.request.http           HTTP request line and headers
│   ├── 1000000.html.metadata.warc          WARC headers of the metadata record
│   └── 1000000.html.metadata.warc-fields   body of the metadata record (fetchTimeMs, charset, languages, …)
├── acc-ern.tul.cz/
│   ├── 1000041.pdf
│   └── …
├── warcinfo.warc, warcinfo.warc-fields
├── manifest.jsonl, manifest.csv
└── *.csv header tables (same set as the flat format)
```

The two cdx_toolkit repackages contain `response` records only, so their sidecar ZIPs carry just
`.response.warc` and `.response.http` per payload; the `request_*.csv` and `metadata*.csv` tables are still written
but hold only a header row.

## Using them with warc2zip

Installation, usage and ready-made commands for each of these files are documented in the
[warc2zip repository](https://github.com/commoncrawl/warc2zip) — see in particular
[WARC examples for testing](https://github.com/commoncrawl/warc2zip#warc-examples-for-testing).
`warc2zip` can read the files straight from the bucket URLs above; no download is required.

The WARC files are ordinary `.warc.gz` files and also work with any other WARC tooling.

## File details

### `500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz`

The first 500 captures of a regular Common Crawl segment WARC
(`CC-MAIN-20260618163205-20260618193205-00999.warc.gz`), kept exactly as the crawler wrote them:
each capture consists of a `request`, a `response` and a `metadata` record, preceded by the original
`warcinfo` record. This is the one to use for exercising request and metadata record handling.

Record counts: 1 `warcinfo`, 501 `request`, 500 `response`, 500 `metadata`.

`warcinfo`:

```
isPartOf: CC-MAIN-2026-25
publisher: Common Crawl
description: Wide crawl of the web for June 2026
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-233
software: Apache Nutch 1.21 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.7-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/
```

Note that the retained `warcinfo` record reports `isPartOf: CC-MAIN-2026-25`, the crawl the source
segment belongs to.

### `HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz`

Homepage captures (site roots such as `https://example.com/`, occasionally `/index.html` and the like) extracted from CC-MAIN-2026-21
with [cdx_toolkit](https://github.com/cocrawler/cdx_toolkit). Only `response` records are included;
the matching `request` and `metadata` records are not.

Record counts: 1 `warcinfo`, 35,700 `response`.

`warcinfo`:

```
software: pypi_cdx_toolkit/0.9.40.dev89+g53a7ef76c
isPartOf: CC-MAIN-2026-21
description: Repackage of CC-MAIN-2026-21 containing only response records of homepages
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>
```

### `IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz`

Captures from hosts classified as US federal government, extracted from CC-MAIN-2025-13 with
cdx_toolkit as part of the [End of Term Archive](https://eotarchive.org/) effort.
Only `response` records are included.

Record counts: 1 `warcinfo`, 35,700 `response`.

`warcinfo`:

```
software: pypi_cdx_toolkit/0.9.40.dev91+ga04800ea0
isPartOf: CC-MAIN-2025-13
description: Repackage of CC-MAIN-2025-13 containing only response records of US federal government hosts
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>
```

## Things to know

- **`WARC-Filename` still carries the original file names.** The two cdx_toolkit repackages were renamed
  after they were written, so their `warcinfo` records say `WARC-Filename: homepages_CC-MAIN-2026-21.warc.gz`
  and `WARC-Filename: is_us_federal_CC-MAIN-2025-13.warc.gz`; the 500-record file names its source segment,
  `CC-MAIN-20260618163205-20260618193205-00999.warc.gz`. Tools that derive names from `WARC-Filename` will
  use those values.
- **`manifest.*` records bare file names in every format.** The `filename` column is `1000000.html` also in the
  `.sidecar.zip` files, where the entry actually lives at `<host>/1000000.html`; the host directory has to be
  derived from `warc_target_uri` when joining the manifest to ZIP entries.
- **Each ZIP was produced by a separate run**, so the conversion timestamp and hash in the top-level directory
  differ between the `.zip`, `.metadata-only.zip` and `.sidecar.zip` of the same WARC. The header tables and
  manifests inside are otherwise identical across the three variants.
- **The US federal repackage is dominated by a single host.** Its 35,700 captures come from only six hosts,
  which is why its sidecar ZIP has almost everything under `www.voanews.com/`.
- The content is a redistribution of Common Crawl data and is subject to the
  [Common Crawl Terms of Use](https://commoncrawl.org/terms-of-use).