RichardScottOZ/comic-strip-encoder-v1

Comic Strip Encoder v1 (Stage 4)

This model is a Transformer sequence encoder designed to generate narrative-aware, contextualized embeddings of comic book page strips. It serves as "Stage 4" of the Comic Analysis Framework v2.0.

Where comic-panel-vlm-v1 (Stage 3) generates a 512-dimensional embedding per panel in isolation, this model takes a full page's worth of panel embeddings (a strip) and runs them through a Transformer encoder. Every output embedding is then conditioned on the panels surrounding it — the model has learned what a panel means in the context of the story around it. The outputs are:

• contextualized_panels (N, 512) — per-panel embeddings enriched with sequential narrative context
• strip_embeddings (512,) — a single vector summarising an entire page/strip

These are intended as the primary inputs for downstream retrieval, reranking, and narrative analysis tasks (Stage 5).

Model Architecture

The comic-strip-encoder-v1 is a BERT-style Transformer Encoder (Stage4SequenceModel):

1. Input Projection: Linear layer mapping 512-d panel embeddings into the model's d_model space.
2. Positional Encoding: Learned positional encodings for panel sequence order.
3. Panel Sequence Transformer:
   • 6 Transformer encoder layers
   • 8 attention heads
   • Pre-norm (LayerNorm before attention) for training stability
   • Attention masking for variable-length strips (max 16 panels)
4. Strip Aggregation: A learned [CLS]-style query attends over all panel outputs to produce a single strip-level vector.
5. Task-Specific Heads (7 total, used during training):

Head	Task	Paper
ReadingOrderHead	Pairwise panel ordering (adjacency matrix)	ComicsPAP
PanelPickingHead	Select missing panel from candidates	ComicsPAP
CharacterCoherenceHead	Visual identity consistency across panels	ComicsPAP
VisualClosureHead	Action continuation plausibility	ComicsPAP
TextClosureHead	Dialogue continuation plausibility	ComicsPAP
CaptionRelevanceHead	Text-visual alignment scoring	ComicsPAP
TextClozeHead	Select correct dialogue given visual context	Text-Cloze

At inference time only the Transformer backbone + strip aggregator are required for embedding generation. The task heads can be used directly for scoring tasks.

Training Data & Methodology

The model was trained on sequences of panel embeddings generated by comic-panel-vlm-v1 across approximately 1 million comic pages, filtered for narrative/story content by Stage 2 (CoSMo PSS).

Research Foundation

• ComicsPAP (arXiv:2503.08561): Five discriminative tasks for sequential comic understanding. State-of-the-art LMMs perform near chance on these tasks; domain-trained sequence models are necessary.
• Text-Cloze (arXiv:2403.03719): Multimodal transformers outperform RNNs by ~10% on dialogue cloze tasks; domain-adapted encoders are critical.

Training Objectives

L_total = Σ(weighted task losses) + 0.5 * L_contrastive + 0.3 * L_reading_order

Task weights during multi-task training:

task_weights = {
    'panel_picking':        1.0,   # Primary ComicsPAP task
    'text_cloze':           1.0,   # Primary Text-Cloze task
    'reading_order':        0.7,
    'visual_closure':       0.8,
    'text_closure':         0.8,
    'character_coherence':  0.5,
    'caption_relevance':    0.5,
}

Key design choice — discriminative not generative: candidates are selected from a pool rather than generated, following the ComicsPAP framework. This makes training tractable and evaluation unambiguous.

Usage

The codebase is available at the Comic Analysis GitHub Repository under src/version2/stage4_sequence_modeling_framework.py.

Example: Generating Strip & Panel Embeddings

import torch
from stage4_sequence_modeling_framework import Stage4SequenceModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 1. Initialize model
model = Stage4SequenceModel(d_model=512, num_layers=6, nhead=8).to(device)

# Load weights from Hugging Face
state_dict = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/RichardScottOZ/comic-strip-encoder-v1/resolve/main/best_model.pt",
    map_location=device
)
model.load_state_dict(state_dict['model_state_dict'])
model.eval()

# 2. Inputs: panel embeddings from comic-panel-vlm-v1
# panel_embeddings: (B, N, 512)  — N panels on one page, up to 16
# panel_mask:       (B, N)       — True where panel exists

panel_embeddings = torch.randn(1, 6, 512).to(device)  # 1 page, 6 panels
panel_mask = torch.ones(1, 6, dtype=torch.bool).to(device)

# 3. Generate embeddings
with torch.no_grad():
    outputs = model(panel_embeddings, panel_mask)

contextualized_panels = outputs['contextualized_panels']  # (1, 6, 512)
strip_embedding       = outputs['strip_embedding']        # (1, 512)

print(f"Contextualized panels: {contextualized_panels.shape}")
print(f"Strip embedding:       {strip_embedding.shape}")

Example: Reading Order Scoring

with torch.no_grad():
    # order_matrix[0, i, j] = score indicating if panel i comes before panel j
    order_matrix = model.reading_order_head(panel_embeddings)  # (1, N, N)

# Compute sorting order based on average row scores
predicted_order = order_matrix[0].sum(dim=1).argsort(descending=True)
print(f"Predicted reading order: {predicted_order.tolist()}")

Example: Panel Picking (ComicsPAP-style)

# context: panels from the strip with one masked out
# candidates: 5 panel embeddings (1 correct, 4 distractors)
context_emb    = contextualized_panels[:, :5, :]   # (1, 5, 512)
candidate_embs = torch.randn(1, 5, 512).to(device) # (1, 5 candidates, 512)

with torch.no_grad():
    scores = model.panel_picking_head(context_emb.mean(dim=1), candidate_embs)
    predicted_idx = scores.argmax(dim=-1)
print(f"Predicted panel index: {predicted_idx.item()}")

Pipeline Position

Stage 1: Raw Comics → Panel crops + OCR text
Stage 2: CoSMo (PSS) → Narrative page classification
Stage 3: comic-panel-vlm-v1 → Multimodal panel embeddings (V + T + Composition) → (N, 512)
Stage 4: comic-strip-encoder-v1 → Contextualized panel + strip embeddings ← THIS MODEL
Stage 5: Storage & Query → Zarr store + semantic search

Intended Use

• Narrative reranking: Stage 3 retrieves top-N candidates; Stage 4 strip embeddings rerank by sequence coherence.
• Story-level similarity: Encode a query as a single panel → Stage 4 → compare strip embeddings across a corpus (story-level search, not panel-level).
• Reading order auditing: Use the ReadingOrderHead pairwise matrix to verify or correct panel sequencing in digitised comics.
• Narrative flow verification: Score a proposed page sequence for coherence using the closure heads.
• Localisation/dialogue drift auditing: Use the TextClozeHead to flag pages where dialogue is likely misattributed or out of order.

Limitations

• Fixed max sequence length: 16 panels per page (memory constraint at training time).
• Discriminative only: Task heads require candidate sets; not a generative model.
• Page-level only: Does not model multi-page narrative arcs.
• Upstream dependency: Requires Stage 3 (comic-panel-vlm-v1) embeddings as input; raw images are not accepted directly.
• No explicit character re-identification: The CharacterCoherenceHead scores visual consistency but does not track named characters across pages.

Performance Expectations

Task	Expected Accuracy	Random Baseline
Panel Picking	60–70%	20%
Visual Closure	55–65%	20%
Text Closure	50–60%	20%
Reading Order	75–85%	50%
Text-Cloze	50–60%	25%

Citation

If you use this model or the associated framework, please link back to the Comic Analysis GitHub Repository.

Related work this model is based on:

@article{comicspap2025,
  title={ComicsPAP: A Panel-Aware Pipeline for Comic Understanding},
  year={2025},
  url={https://arxiv.org/abs/2503.08561}
}

@article{textcloze2024,
  title={Text-Cloze: Multimodal Dialogue Prediction in Comics},
  year={2024},
  url={https://arxiv.org/abs/2403.03719}
}