---
language: en
license: apache-2.0
base_model: microsoft/prophetnet-large-uncased
tags:
  - summarization
  - research-paper
  - seq2seq
  - prophetnet
  - lora
  - peft
datasets:
  - custom
metrics:
  - rouge
  - bertscore
---

# ProphetNet-Large-Summarization

A fine-tuned version of [microsoft/prophetnet-large-uncased](https://huggingface.co/microsoft/prophetnet-large-uncased) for summarizing research papers into concise summaries. This is the first stage of a two-step **Research Paper Simplifier** pipeline.

## Model Description

This model takes a section of a research paper as input and generates a plain-language summary. Fine-tuned using LoRA (PEFT) with 4-bit quantization for efficient training.

## Pipeline

```
Research Paper ──► [ProphetNet-Large-Summarization] ──► Summary ──► [ProphetNet-Large-Story-Generation] ──► Story
```

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | microsoft/prophetnet-large-uncased |
| Task | Summarization |
| Max input length | 2048 tokens |
| Max target length | 256 tokens |
| Learning rate | 3e-5 |
| Batch size | 2 |
| Gradient accumulation steps | 4 |
| Warmup steps | 1500 |
| Weight decay | 0.01 |
| Fine-tuning method | LoRA (r=16, alpha=64, targets: query_proj, value_proj) |
| Quantization | 4-bit NF4 (bitsandbytes) |

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("harsharajkumar273/ProphetNet-Large-Summarization")
model = AutoModelForSeq2SeqLM.from_pretrained("harsharajkumar273/ProphetNet-Large-Summarization")

text = "Your research paper section here..."
word_count = len(text.split())
prompt = f"Summarize this part of the research paper to less than {word_count // 10} words:\n{text}"

inputs = tokenizer(prompt, return_tensors="pt", max_length=2048, truncation=True)
outputs = model.generate(**inputs, max_length=256, num_beams=4)
summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(summary)
```

## Evaluation Metrics

Evaluated using ROUGE and BERTScore on a held-out 10% test split.

## Related Models

- [harsharajkumar273/Bart-Base-Summarization](https://huggingface.co/harsharajkumar273/Bart-Base-Summarization)
- [harsharajkumar273/T5-Base-Summarization](https://huggingface.co/harsharajkumar273/T5-Base-Summarization)
- [harsharajkumar273/ProphetNet-Large-Story-Generation](https://huggingface.co/harsharajkumar273/ProphetNet-Large-Story-Generation) — next stage