Upload README.md with huggingface_hub

README.md

@@ -1,88 +1,245 @@
-# Model Card · RTX 5090 Vision Encoder-Decoder Image Captioning
+---
+language:
+  - en
+library_name: transformers
+pipeline_tag: image-to-text
+base_model: nlpconnect/vit-gpt2-image-captioning
+tags:
+  - vision-encoder-decoder
+  - image-captioning
+  - vision-language
+  - onnx
+  - transformers.js
+  - onnxruntime
+  - pytorch
+  - vit
+  - gpt2
+---
+
+# Fine-Tuned ViT-GPT-2 Vision Encoder-Decoder Image Captioning
+
+## Model repository
+
+`anmol-unitmole/image-caption-generation-vision-encoder-decoder-model`
 
 ## System
 
-`02-image-caption-generation-vision-encoder-decoder`
+`image-caption-generation-vision-encoder-decoder-model`
 
 ## Status
 
-**Execution-ready, not yet result-complete.** This repository contains the complete training, evaluation, optimization, and deployment workflow. Fine-tuned weights, new metrics, ONNX parity results, and the final public URL must be generated on the target RTX 5090 system.
+**Training, held-out evaluation, browser-compatible ONNX export, dynamic quantization, and PyTorch-versus-ONNX parity validation are complete.**
+
+This repository contains the browser-compatible model configuration, processor, tokenizer, generation configuration, and quantized ONNX artifacts for a fine-tuned ViT-GPT-2 Vision Encoder-Decoder image-captioning model.
+
+The model was fine-tuned locally using an NVIDIA GeForce RTX 5090. The browser artifacts are intended for Transformers.js and ONNX Runtime Web inference inside a static Hugging Face Space.
+
+The final public Static Space remains subject to successful browser loading and end-to-end caption-generation validation.
 
 ## Architecture
 
-- Python checkpoint: `nlpconnect/vit-gpt2-image-captioning`
-- Vision encoder: ViT
+- Base checkpoint: `nlpconnect/vit-gpt2-image-captioning`
+- Vision encoder: Vision Transformer — ViT
 - Text decoder: GPT-2 with encoder cross-attention
-- Framework: PyTorch and Hugging Face `VisionEncoderDecoderModel`
-- Browser baseline: `Xenova/vit-gpt2-image-captioning`
+- Python framework: PyTorch
+- Hugging Face architecture: `VisionEncoderDecoderModel`
+- Pretrained browser baseline: `Xenova/vit-gpt2-image-captioning`
+- Fine-tuned browser repository: `anmol-unitmole/image-caption-generation-vision-encoder-decoder-model`
 - Browser runtime: Transformers.js and ONNX Runtime Web
+- Preferred browser provider: WebGPU
+- Fallback browser provider: WebAssembly
+
+## Task
+
+The model generates English-language captions from input images.
+
+Supported project features include:
+
+- greedy caption generation;
+- beam-search caption generation;
+- caption-length controls;
+- alternate-text draft generation;
+- pretrained-versus-fine-tuned comparison;
+- caption-structure diagnostics;
+- genuine precomputed token-generation traces for bundled sample images;
+- model latency and evaluation reporting.
 
 ## Training method
 
-1. Freeze the vision encoder and adapt the decoder/cross-modal path.
-2. Unfreeze the top six vision blocks with differential learning rates.
-3. Conservatively fine-tune the full architecture.
-4. Select checkpoints using generated-caption validation metrics, with validation-loss fallback.
+The model was trained using a staged adaptation strategy.
+
+### Stage 1 — Decoder warm-up
+
+- Freeze the complete vision encoder.
+- Train the decoder and cross-modal generation pathway.
+- Preserve pretrained visual representations during initial adaptation.
+
+### Stage 2 — Upper encoder adaptation
+
+- Unfreeze the upper eight ViT encoder blocks.
+- Use a lower learning rate for the vision encoder.
+- Continue adapting the decoder using a higher learning rate.
+- Evaluate generated captions during training.
+
+### Stage 3 — Conservative full fine-tuning
+
+- Unfreeze the full encoder-decoder architecture.
+- Apply reduced learning rates.
+- Select the best checkpoint using validation CIDEr with configured fallback behavior.
+
+Training controls included:
+
+- BF16 mixed precision;
+- TF32 support;
+- fused AdamW fallback;
+- gradient accumulation;
+- label smoothing;
+- warm-up scheduling;
+- cosine learning-rate decay;
+- minimum learning-rate floor;
+- gradient clipping;
+- early-stopping support;
+- resumable checkpoints;
+- best-checkpoint selection;
+- environment and GPU-memory reporting.
+
+## Hardware
+
+The final experiment was executed locally using:
 
-Implemented controls include BF16/FP16 handling, TF32, fused AdamW fallback, gradient accumulation, label smoothing, warm-up, cosine decay, minimum LR floor, gradient clipping, early stopping, resumable checkpoints, environment capture, and peak GPU-memory reporting.
+- GPU: NVIDIA GeForce RTX 5090
+- GPU memory: approximately 31.84 GB
+- CUDA build: CUDA 13.0
+- PyTorch: 2.12.0 with CUDA 13.0
+- Precision: BF16 where supported
+- Operating system: Windows 11
 
-## Data
+## Dataset
 
-Primary experiment: Flickr30k. Multiple references remain grouped for evaluation. The repository does not redistribute the complete dataset.
+The primary experiment used Flickr30k.
+
+The processed experiment contained:
+
+| Split | Unique images | Caption records |
+|---|---:|---:|
+| Training | 29,000 | 144,767 |
+| Validation | 1,014 | 5,062 |
+| Test | 1,000 | 4,985 |
+
+Multiple reference captions remained grouped for evaluation.
+
+The complete Flickr30k dataset is not redistributed through this model repository. Users must obtain the dataset according to its original access and licensing requirements.
 
 ## Inputs
 
-RGB JPG, PNG, WEBP, or BMP images after safe decoding, EXIF orientation correction, and checkpoint-native image processing.
+Supported browser image formats:
+
+- JPG/JPEG
+- PNG
+- WEBP
+- BMP
+
+Input processing includes:
+
+- safe image decoding;
+- EXIF orientation correction;
+- RGB conversion;
+- model-native image resizing and normalization;
+- processor-compatible pixel-value generation;
+- validation for unsupported or corrupt images.
 
 ## Outputs
 
-- greedy caption;
-- ranked beam candidates;
-- concise model-generated alt-text draft;
-- token transition scores where available;
+The model may provide:
+
+- one greedy caption;
+- multiple ranked beam-search candidates;
+- a concise alternate-text draft;
+- generation latency;
+- generation configuration;
 - sequence score where available;
-- latency and generation settings;
-- optional genuine cross-attention output.
+- genuine token-transition traces for bundled sample images;
+- caption-structure diagnostics.
+
+Generated captions are model predictions and are not guaranteed to be factually correct.
+
+## Evaluation protocol
+
+The pretrained baseline and fine-tuned model were evaluated on the same held-out 1,000-image Flickr30k test subset.
+
+### Main results
+
+| Metric | Pretrained baseline | Fine-tuned model |
+|---|---:|---:|
+| BLEU-1 | 0.5779 | 0.4350 |
+| BLEU-4 | 0.1366 | 0.1078 |
+| METEOR | 0.3337 | 0.4050 |
+| ROUGE-L | 0.3759 | 0.3041 |
+| CIDEr | 0.2336 | 0.0595 |
+| CLIPScore | 0.6601 | 0.7211 |
+| Distinct-1 | 0.0652 | 0.0708 |
+| Distinct-2 | 0.2331 | 0.2478 |
+| Unique-caption ratio | 0.9500 | 1.0000 |
+| Average repetition rate | 0.1126 | 0.1463 |
+| Mean GPU latency | 0.1163 seconds | 0.2762 seconds |
+
+### Result interpretation
+
+The fine-tuned model did not outperform the pretrained baseline on every metric.
+
+Observed improvements included:
+
+- higher METEOR;
+- higher CLIPScore;
+- higher Distinct-1;
+- higher Distinct-2;
+- a 1.0 unique-caption ratio.
 
-## Evaluation
+Observed regressions included:
 
-Required final evidence:
+- lower BLEU-1;
+- lower BLEU-4;
+- lower ROUGE-L;
+- lower CIDEr;
+- increased caption length;
+- increased repetition;
+- slower generation latency.
 
-- BLEU-1 and BLEU-4;
-- METEOR;
-- ROUGE-L;
-- CIDEr;
-- CLIPScore;
-- distinct-1 and distinct-2;
-- unique-caption ratio and repetition rate;
-- greedy-versus-beam analysis;
-- model size and latency;
-- quality challenge performance;
-- manual failure analysis;
-- PyTorch-versus-ONNX parity.
+These mixed results are reported intentionally. The experiment demonstrates that fine-tuning can improve semantic similarity and lexical diversity while reducing reference-overlap metrics or generation efficiency.
 
-No new score is claimed until the corresponding artifact exists under `outputs/`.
+The fine-tuned model should therefore not be described as universally superior to the pretrained baseline.
 
-## Intended use
+## Model size
 
-Education, portfolio demonstration, image-captioning experimentation, draft metadata, human-reviewed visual documentation, and quality-reporting research prototypes.
+The fine-tuned Python checkpoint contains approximately:
 
-## Prohibited or unsuitable use
+- Parameters: 239,195,904
+- Estimated FP32 parameter size: 912.46 MB
+- Estimated FP16 parameter size: 456.23 MB
+- Serialized checkpoint directory size: 917.11 MB
 
-Identity recognition, surveillance, sensitive-attribute inference, accessibility-critical publishing without review, product release, official quality decisions, medical, legal, financial, hiring, insurance, security, or safety-critical decisions.
+Browser deployment uses quantized ONNX artifacts rather than the original PyTorch checkpoint.
 
-## Limitations
+## ONNX export
 
-The model can hallucinate, omit important objects, confuse colors and counts, produce generic descriptions, repeat phrases, and fail on poor-quality or out-of-distribution images. Flickr30k is not an industrial inspection dataset. Quality challenge results measure descriptive behavior only and do not establish defect-detection competence.
+The model was exported using the `image-to-text-with-past` task and ONNX opset 18.
 
-## Confidence and interpretability
+The browser export produced:
 
-Generated-token probability is not calibrated correctness. The browser's structural heuristic is not presented as model confidence. Token and attention views are interpretability aids and do not prove reasoning.
+- `encoder_model.onnx`
+- `decoder_model.onnx`
+- `decoder_with_past_model.onnx`
+- `decoder_model_merged.onnx`
 
-## Privacy
+The files were then processed through dynamic ONNX Runtime quantization.
 
-Do not upload personal, confidential, medical, proprietary, security-sensitive, or unlicensed images to a public Space. Public samples must be synthetic, public-domain, or properly licensed.
+## Browser model files
 
-## Deployment
+This repository contains the following quantized ONNX files:
 
-The fine-tuned Python checkpoint must be converted to a Transformers.js-compatible ONNX repository and pass parity/browser checks before `fine_tuned_model_id` is enabled in `web/metadata.json`.
+```text
+onnx/
+├── encoder_model_quantized.onnx
+├── decoder_model_quantized.onnx
+├── decoder_with_past_model_quantized.onnx
+└── decoder_model_merged_quantized.onnx