Added :Added the support for the lastest model

.gitignore

@@ -17,7 +17,8 @@ __pycache__/
 # ---- Jupyter / IPython ----
 .ipynb_checkpoints/
 *.ipynb
-
+notebook/
+*.csv
 # ---- Model & Data Artifacts ----
 *.pth
 *.pt

config.py

@@ -28,7 +28,7 @@ class Config:
     REAL_FORGED_MODEL_LOCAL_PATH = os.getenv("REAL_FORGED_MODEL_LOCAL_PATH", "Model/real_forged/fft_cnn_model_78.pth")
     DOCUMENT_FORGERY_MODEL_PATH = os.getenv(
         "DOCUMENT_FORGERY_MODEL_PATH",
-        "features/Model/document_forgery/ela_cnn_model.pth",
+        "features/Model/document_forgery/pixel_forgery_v3_best.pth",
     )
     # Decision thresholds for document forgery detector (probabilities in 0..1)
     DOCUMENT_FORGERY_POSSIBLE_LOW = float(os.getenv("DOCUMENT_FORGERY_POSSIBLE_LOW", "0.40"))

notebook/ai_vs_human/final_archi.md

@@ -1,426 +0,0 @@
-# AI vs Human Text Detector V3 - Final Architecture Summary
-dataset = "Pujan-Dev/english_aivshuman"
-**Model Version**: V3  
-**Type**: Hybrid Feature Engineering + TF-IDF Classifier  
-**Output Directory**: `./v3_model/`  
-**Date**: March 2026
-
----
-
-## 📊 Overview
-
-The V3 model is a **non-transformer, feature-based ML classifier** that distinguishes between AI-generated and human-written text using a hybrid approach combining engineered linguistic features with TF-IDF text representations.
-
-```
-┌─────────────┐
-│  Input Text │
-└──────┬──────┘
-       │
-       ├──────────────────────────────────┐
-       │                                  │
-       ▼                                  ▼
-┌──────────────────┐            ┌─────────────────┐
-│  Text Features   │            │   Engineered    │
-│    (TF-IDF)      │            │    Features     │
-│                  │            │  (16 features)  │
-│ • Word (1-2gram) │            │                 │
-│ • Char (3-5gram) │            │ • Perplexity    │
-│                  │            │ • Burstiness    │
-│ Max 200k features│            │ • Stylometry    │
-└────────┬─────────┘            └─────────┬───────┘
-         │                                │
-         │        ┌───────────────┐       │
-         └───────►│ StandardScaler│◄──────┘
-                  └───────┬───────┘
-                          │
-                  ┌───────▼──────────┐
-                  │ Sparse Matrix    │
-                  │   Concat (hstack)│
-                  └───────┬──────────┘
-                          │
-                  ┌───────▼────────┐
-                  │   Logistic     │
-                  │  Regression    │
-                  │  (GridSearchCV)│
-                  └───────┬────────┘
-                          │
-                  ┌───────▼────────┐
-                  │  Prediction    │
-                  │ (Human vs AI)  │
-                  └────────────────┘
-```
-
----
-
-## 🏗️ Architecture Components
-
-### 1. **Data Loading**
-
-**Function**: `load_dataset_recursive(max_samples=20000)`
-
-- **Source**: Recursively scans `./DATASET/` folder
-- **Formats Supported**: `.jsonl`, `.json`, `.csv`
-- **Schema Support**:
-  - Schema 1: `human_text` + `ai_text` columns
-  - Schema 2: `text` + `label`/`ai_gen` columns
-- **Labels**: 
-  - `0` = Human text
-  - `1` = AI-generated text
-- **Preprocessing**: Text normalization (whitespace cleanup)
-- **Max Samples**: 20,000 (configurable)
-- **Random State**: 42
-
----
-
-### 2. **Feature Extraction Pipeline**
-
-The model extracts **3 types of features** in parallel:
-
-#### 2.1 **Perplexity Features** (1 feature)
-
-**Model**: `distilgpt2` (Hugging Face Transformers)
-
-```python
-class PerplexityCalculator:
-    - Model: distilgpt2
-    - Max Length: 512 tokens
-    - Metric: exp(cross_entropy_loss)
-    - Cap: 10,000 (outlier protection)
-    - Fallback: 100.0 on error
-```
-
-**What it measures**: Language model surprise/naturalness
-- Lower perplexity → More predictable (often AI)
-- Higher perplexity → Less predictable (often human)
-
----
-
-#### 2.2 **Burstiness Features** (5 features)
-
-Measures sentence length variation patterns.
-
-**Features**:
-1. `burst_mean` - Average sentence length (words)
-2. `burst_std` - Standard deviation of sentence lengths
-3. `burst_max` - Maximum sentence length
-4. `burst_min` - Minimum sentence length
-5. `burst_range` - Range (max - min)
-
-**Theory**: Human writing has more variation in sentence length (high burstiness), while AI text tends to be more uniform.
-
----
-
-#### 2.3 **Stylometry Features** (10 features)
-
-Writing style and readability metrics.
-
-**Features**:
-1. `num_words` - Total word count
-2. `num_chars` - Total character count
-3. `num_sentences` - Total sentence count
-4. `avg_word_len` - Average word length
-5. `avg_sent_len` - Average sentence length
-6. `lexical_diversity` - Unique words / total words
-7. `punct_ratio` - Punctuation density
-8. `caps_ratio` - Capitalization ratio
-9. `flesch_reading` - Flesch Reading Ease score
-10. `flesch_grade` - Flesch-Kincaid Grade Level
-
-**Library**: `textstat` + `nltk`
-
----
-
-### 3. **TF-IDF Vectorization**
-
-#### 3.1 **Word-Level TF-IDF**
-
-```python
-TfidfVectorizer(
-    analyzer="word",
-    ngram_range=(1, 2),        # Unigrams + bigrams
-    min_df=3,                  # Minimum document frequency
-    max_df=0.98,               # Maximum document frequency
-    max_features=120000,       # Cap at 120k features
-    sublinear_tf=True          # log(tf) scaling
-)
-```
-
-**Output**: Sparse matrix of word/phrase importance scores
-
----
-
-#### 3.2 **Character-Level TF-IDF**
-
-```python
-TfidfVectorizer(
-    analyzer="char_wb",        # Character n-grams (word boundaries)
-    ngram_range=(3, 5),        # 3-char to 5-char sequences
-    min_df=3,
-    max_df=0.98,
-    max_features=80000,        # Cap at 80k features
-    sublinear_tf=True
-)
-```
-
-**Purpose**: Captures sub-word patterns and stylistic signatures
-
----
-
-### 4. **Feature Preprocessing**
-
-**Engineered Features**: 
-- Scaled using `StandardScaler` (z-score normalization)
-- Converted to sparse CSR matrix for memory efficiency
-
-**Hybrid Feature Vector**:
-```python
-hybrid_vec = hstack([word_tfidf, char_tfidf, engineered_features_scaled])
-```
-
-**Final Feature Dimensionality**:
-- Word TF-IDF: Up to 120,000 features
-- Char TF-IDF: Up to 80,000 features  
-- Engineered: 16 features
-- **Total**: Up to ~200,016 features (sparse)
-
----
-
-### 5. **Model Training**
-
-#### 5.1 **Train-Test Split**
-```python
-train_size: 80% (16,000 samples)
-test_size: 20% (4,000 samples)
-stratified: Yes (balanced across classes)
-random_state: 42
-```
-
-#### 5.2 **Classifier**
-
-**Algorithm**: Logistic Regression
-
-**Hyperparameter Tuning**: GridSearchCV with 3-fold stratified cross-validation
-
-**Search Space**:
-```python
-{
-    "C": [0.5, 1.0, 2.0, 4.0],           # Regularization strength
-    "class_weight": [None, "balanced"],   # Class balancing
-    "solver": "saga",                     # Stochastic Average Gradient
-    "penalty": "l2",                      # L2 regularization
-    "max_iter": 2500,
-    "n_jobs": -1                          # Parallel processing
-}
-```
-
-**Scoring Metric**: F1 Score (balanced for precision/recall)
-
----
-
-### 6. **Model Evaluation**
-
-**Metrics Tracked**:
-- **Accuracy**: Overall correct predictions
-- **F1 Score**: Harmonic mean of precision/recall
-- **ROC-AUC**: Area under ROC curve
-- **Confusion Matrix**: True/false positives/negatives
-- **Classification Report**: Per-class precision/recall/F1
-
-**Visualizations**:
-1. Confusion Matrix
-2. ROC Curve
-3. Feature Importance (top engineered features)
-4. Perplexity Distribution (Human vs AI)
-5. Lexical Diversity Distribution
-6. Burstiness STD Distribution
-
----
-
-### 7. **Model Persistence**
-
-**Output Directory**: `./v3_model/`
-
-**Saved Artifacts**:
-
-| File | Description |
-|------|-------------|
-| `classifier.pkl` | Trained Logistic Regression model |
-| `scaler.pkl` | StandardScaler for engineered features |
-| `word_vectorizer.pkl` | Word-level TF-IDF vectorizer |
-| `char_vectorizer.pkl` | Character-level TF-IDF vectorizer |
-| `feature_names.json` | List of engineered feature names (16 features) |
-| `metadata.json` | Model performance metrics & configuration |
-
-**Metadata Contents**:
-```json
-{
-  "selected_model": "hybrid_tfidf_logistic",
-  "cv_best_f1": 0.xxxx,
-  "num_engineered_features": 16,
-  "num_word_tfidf_features": 120000,
-  "num_char_tfidf_features": 80000,
-  "train_samples": 16000,
-  "test_samples": 4000,
-  "train_accuracy": 0.xxxx,
-  "train_f1": 0.xxxx,
-  "test_accuracy": 0.xxxx,
-  "test_f1": 0.xxxx
-}
-```
-
----
-
-### 8. **Inference Pipeline**
-
-**Function**: `predict_v3(text: str) -> dict`
-
-**Process**:
-```python
-1. Normalize text (whitespace cleanup)
-2. Extract engineered features (16 features)
-3. Scale engineered features (StandardScaler)
-4. Generate word TF-IDF vector
-5. Generate char TF-IDF vector
-6. Concatenate all features (sparse matrix)
-7. Predict with Logistic Regression
-8. Return prediction + probabilities + features
-```
-
-**Output Schema**:
-```python
-{
-    "text": str,                    # Truncated input (100 chars)
-    "word_count": int,              # Number of words
-    "predicted_label": int,         # 0=Human, 1=AI
-    "predicted_name": str,          # "human" or "ai"
-    "probability_human": float,     # P(Human) [0-1]
-    "probability_ai": float,        # P(AI) [0-1]
-    "features": dict                # All 16 engineered features
-}
-```
-
-**Batch Function**: `predict_v3_batch(texts: list[str]) -> list[dict]`
-
----
-
-## 🔧 Configuration
-
-```python
-@dataclass
-class V3Config:
-    max_samples: int = 20000       # Max training samples
-    test_size: float = 0.2         # Test split ratio
-    output_dir: str = "./v3_model" # Model save directory
-    random_state: int = 42         # Reproducibility seed
-    cv_folds: int = 3              # Cross-validation folds
-```
-
----
-
-## 📦 Dependencies
-
-**Core Libraries**:
-- `scikit-learn` - ML algorithms, TF-IDF, metrics
-- `pandas` - Data manipulation
-- `numpy` - Numerical operations
-- `scipy` - Sparse matrix operations
-
-**Feature Extraction**:
-- `transformers` - DistilGPT2 for perplexity
-- `torch` - PyTorch backend for transformers
-- `nltk` - Sentence tokenization (`punkt_tab`)
-- `textstat` - Readability metrics
-
-**Visualization**:
-- `matplotlib` - Plotting
-- `seaborn` - Statistical visualizations
-
----
-
-## 🎯 Key Design Decisions
-
-### Why Not Transformers?
-1. **Speed**: No GPU required, fast inference
-2. **Interpretability**: Explainable features
-3. **Efficiency**: Smaller model size (~500MB vs 5GB+)
-4. **Robustness**: Works on any text length
-
-### Why Hybrid Features?
-1. **TF-IDF**: Captures content and vocabulary patterns
-2. **Perplexity**: Measures language model naturalness
-3. **Burstiness**: Detects sentence variation patterns
-4. **Stylometry**: Analyzes writing style signatures
-
-### Why Logistic Regression?
-1. **Scalability**: Handles 200k+ sparse features efficiently
-2. **Speed**: Fast training and inference
-3. **Interpretability**: Clear feature importance via coefficients
-4. **Robustness**: Well-suited for high-dimensional sparse data
-
----
-
-## 📈 Expected Performance
-
-**Typical Results** (20k samples):
-- **Test Accuracy**: 85-95%
-- **Test F1 Score**: 0.85-0.95
-- **Inference Speed**: ~50-100 texts/second (CPU)
-- **Model Size**: ~500 MB total
-
-**Best For**:
-- ✅ General English text classification
-- ✅ Articles, essays, reviews
-- ✅ Medium to long texts (50+ words)
-
-**Limitations**:
-- ⚠️ Very short texts (<10 words) may be unreliable
-- ⚠️ Perplexity calculation is the bottleneck (uses GPU if available)
-- ⚠️ Domain-specific jargon may affect performance
-- ⚠️ Non-English text requires retraining
-
----
-
-## 🔄 Model Loading Example
-
-```python
-from pathlib import Path
-import pickle
-import json
-
-model_dir = Path("./v3_model")
-
-# Load all artifacts
-classifier = pickle.load(open(model_dir / "classifier.pkl", "rb"))
-scaler = pickle.load(open(model_dir / "scaler.pkl", "rb"))
-word_vectorizer = pickle.load(open(model_dir / "word_vectorizer.pkl", "rb"))
-char_vectorizer = pickle.load(open(model_dir / "char_vectorizer.pkl", "rb"))
-feature_names = json.load(open(model_dir / "feature_names.json", "r"))
-metadata = json.load(open(model_dir / "metadata.json", "r"))
-
-# Use predict_v3() function for inference
-result = predict_v3("Your text here...")
-```
-
----
-
-## 💡 Future Improvements
-
-1. **Model Versioning**: Add versioning system for model updates
-2. **Confidence Thresholds**: Flag uncertain predictions
-3. **Batch Optimization**: Vectorized batch inference
-4. **Model Wrapper Class**: Encapsulate all logic in `AIPredictorV3` class
-5. **Perplexity Caching**: Cache calculations for faster inference
-6. **Ensemble Methods**: Combine multiple models for better accuracy
-7. **Active Learning**: Iterative retraining with user feedback
-8. **Multi-language Support**: Train separate models per language
-
----
-
-## 📝 Citation & Credits
-
-**Framework**: scikit-learn + HuggingFace Transformers  
-**Perplexity Model**: DistilGPT2 (OpenAI/Hugging Face)  
-**Readability Metrics**: textstat library  
-
-
-**Architecture Type**: Hybrid Feature Engineering + Logistic Regression

notebook/ai_vs_human/main.ipynb

@@ -1,1110 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "e522047b",
-   "metadata": {},
-   "source": [
-    "# AI vs Human Text Detector using BERT\n",
-    "Using google-bert/bert-base-cased with HC3 dataset or local data (~20k samples)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 35,
-   "id": "16eddd36",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from functools import partial\n",
-    "\n",
-    "import datasets\n",
-    "from datasets import Dataset, DatasetDict, concatenate_datasets\n",
-    "import evaluate\n",
-    "import numpy as np\n",
-    "import torch\n",
-    "from transformers import (\n",
-    "    AutoModelForSequenceClassification,\n",
-    "    AutoTokenizer,\n",
-    "    PreTrainedTokenizer,\n",
-    "    BatchEncoding,\n",
-    "    DataCollatorWithPadding,\n",
-    "    Trainer,\n",
-    "    TrainingArguments,\n",
-    ")\n",
-    "from peft import LoraConfig, get_peft_model"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "99bca750",
-   "metadata": {},
-   "source": [
-    "## Load AI Detection Dataset (~20k samples)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 36,
-   "id": "2945f87a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def get_raid_dataset(max_samples: int = 20000, use_local: bool = True) -> DatasetDict:\n",
-    "    \"\"\"Load AI detection dataset and limit to ~20k samples\"\"\"\n",
-    "    \n",
-    "    print(\"Loading AI vs Human text dataset...\")\n",
-    "    \n",
-    "    all_texts = []\n",
-    "    all_labels = []\n",
-    "    \n",
-    "    # Try loading HC3 dataset (Human ChatGPT Comparison Corpus)\n",
-    "    try:\n",
-    "        print(\"Attempting to load HC3 dataset...\")\n",
-    "        dataset = datasets.load_dataset(\"Hello-SimpleAI/HC3\", \"all\", split=\"train\")\n",
-    "        \n",
-    "        # HC3 format: has 'question', 'human_answers', 'chatgpt_answers'\n",
-    "        for item in dataset:\n",
-    "            # Add human answers\n",
-    "            if 'human_answers' in item and item['human_answers']:\n",
-    "                for answer in item['human_answers'][:1]:  # Take first answer\n",
-    "                    if answer and len(answer.strip()) > 0:\n",
-    "                        all_texts.append(answer)\n",
-    "                        all_labels.append(0)  # 0 for human\n",
-    "            \n",
-    "            # Add AI answers\n",
-    "            if 'chatgpt_answers' in item and item['chatgpt_answers']:\n",
-    "                for answer in item['chatgpt_answers'][:1]:  # Take first answer\n",
-    "                    if answer and len(answer.strip()) > 0:\n",
-    "                        all_texts.append(answer)\n",
-    "                        all_labels.append(1)  # 1 for AI\n",
-    "        \n",
-    "        print(f\"✓ Loaded {len(all_texts)} samples from HC3 dataset\")\n",
-    "    except Exception as e:\n",
-    "        print(f\"⚠ Could not load HC3 dataset: {e}\")\n",
-    "    \n",
-    "    # Load local data and combine\n",
-    "    if use_local:\n",
-    "        try:\n",
-    "            print(\"Loading local dataset...\")\n",
-    "            import pandas as pd\n",
-    "            df = pd.read_json(\"./DATASET/basic_Data.jsonl\", lines=True)\n",
-    "            \n",
-    "            # Build a proper binary classification dataset: human_text -> 0, ai_text -> 1\n",
-    "            if {\"human_text\", \"ai_text\"}.issubset(df.columns):\n",
-    "                local_texts = list(df[\"human_text\"].dropna()) + list(df[\"ai_text\"].dropna())\n",
-    "                local_labels = [0] * len(df[\"human_text\"].dropna()) + [1] * len(df[\"ai_text\"].dropna())\n",
-    "                \n",
-    "                all_texts.extend(local_texts)\n",
-    "                all_labels.extend(local_labels)\n",
-    "                \n",
-    "                print(f\"✓ Loaded {len(local_texts)} samples from local data\")\n",
-    "            else:\n",
-    "                print(\"⚠ Local dataset doesn't have required columns\")\n",
-    "        except Exception as e:\n",
-    "            print(f\"⚠ Could not load local dataset: {e}\")\n",
-    "    \n",
-    "    # Check if we have any data\n",
-    "    if len(all_texts) == 0:\n",
-    "        raise ValueError(\"No data loaded! Check HC3 dataset or local data availability\")\n",
-    "    \n",
-    "    # Create combined dataset\n",
-    "    combined_dataset = Dataset.from_dict({\n",
-    "        \"text\": all_texts,\n",
-    "        \"label\": all_labels\n",
-    "    })\n",
-    "    \n",
-    "    print(f\"Total combined samples: {len(combined_dataset)}\")\n",
-    "    \n",
-    "    # Shuffle and limit to max_samples\n",
-    "    combined_dataset = combined_dataset.shuffle(seed=42)\n",
-    "    if len(combined_dataset) > max_samples:\n",
-    "        combined_dataset = combined_dataset.select(range(max_samples))\n",
-    "        print(f\"Limited to {max_samples} samples\")\n",
-    "    \n",
-    "    # Filter out empty texts\n",
-    "    combined_dataset = combined_dataset.filter(lambda x: x['text'] is not None and len(x['text'].strip()) > 0)\n",
-    "    \n",
-    "    # Split into train/test (95/5 split)\n",
-    "    dataset_split = combined_dataset.train_test_split(test_size=0.05, seed=42)\n",
-    "    \n",
-    "    print(f\"\\n✓ Dataset ready!\")\n",
-    "    print(f\"  Train samples: {len(dataset_split['train'])}\")\n",
-    "    print(f\"  Test samples: {len(dataset_split['test'])}\")\n",
-    "    \n",
-    "    # Check label distribution\n",
-    "    import numpy as np\n",
-    "    train_labels = np.array(dataset_split['train']['label'])\n",
-    "    print(f\"  Label distribution (train):\")\n",
-    "    print(f\"    Human (0): {(train_labels == 0).sum()}\")\n",
-    "    print(f\"    AI (1): {(train_labels == 1).sum()}\")\n",
-    "    \n",
-    "    return dataset_split"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 37,
-   "id": "38d8478c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Loading AI vs Human text dataset...\n",
-      "Attempting to load HC3 dataset...\n",
-      "⚠ Could not load HC3 dataset: Dataset scripts are no longer supported, but found HC3.py\n",
-      "Loading local dataset...\n",
-      "✓ Loaded 19940 samples from local data\n",
-      "Total combined samples: 19940\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Filter: 100%|██████████| 19940/19940 [00:00<00:00, 95584.60 examples/s] \n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "✓ Dataset ready!\n",
-      "  Train samples: 18943\n",
-      "  Test samples: 997\n",
-      "  Label distribution (train):\n",
-      "    Human (0): 9477\n",
-      "    AI (1): 9466\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Load dataset\n",
-    "raw_datasets = get_raid_dataset(max_samples=20000)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f60191f6",
-   "metadata": {},
-   "source": [
-    "## Initialize Model and Tokenizer"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "315bb737",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Loading weights: 100%|██████████| 199/199 [00:00<00:00, 1208.24it/s, Materializing param=bert.pooler.dense.weight]                               \n",
-      "BertForSequenceClassification LOAD REPORT from: google-bert/bert-base-cased\n",
-      "Key                                        | Status     | \n",
-      "-------------------------------------------+------------+-\n",
-      "cls.predictions.transform.LayerNorm.bias   | UNEXPECTED | \n",
-      "cls.seq_relationship.weight                | UNEXPECTED | \n",
-      "cls.predictions.transform.dense.weight     | UNEXPECTED | \n",
-      "cls.seq_relationship.bias                  | UNEXPECTED | \n",
-      "cls.predictions.bias                       | UNEXPECTED | \n",
-      "cls.predictions.transform.dense.bias       | UNEXPECTED | \n",
-      "cls.predictions.transform.LayerNorm.weight | UNEXPECTED | \n",
-      "classifier.weight                          | MISSING    | \n",
-      "classifier.bias                            | MISSING    | \n",
-      "\n",
-      "Notes:\n",
-      "- UNEXPECTED\t:can be ignored when loading from different task/architecture; not ok if you expect identical arch.\n",
-      "- MISSING\t:those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Model loaded: google-bert/bert-base-cased\n",
-      "Device: cuda\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Use google-bert/bert-base-cased\n",
-    "base_model_name = \"google-bert/bert-base-cased\"\n",
-    "\n",
-    "tokenizer = AutoTokenizer.from_pretrained(base_model_name)\n",
-    "model = AutoModelForSequenceClassification.from_pretrained(\n",
-    "    base_model_name,\n",
-    "    num_labels=2,\n",
-    ").to(device='cuda' if torch.cuda.is_available() else 'cpu')\n",
-    "\n",
-    "print(f\"Model loaded: {base_model_name}\")\n",
-    "print(f\"Device: {'cuda' if torch.cuda.is_available() else 'cpu'}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0a772192",
-   "metadata": {},
-   "source": [
-    "## Apply LoRA for Parameter-Efficient Fine-tuning"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 39,
-   "id": "ba294e50",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "trainable params: 2,680,322 || all params: 110,992,132 || trainable%: 2.4149\n"
-     ]
-    }
-   ],
-   "source": [
-    "peft_config = LoraConfig(\n",
-    "    r=16,\n",
-    "    target_modules=\"all-linear\",\n",
-    "    lora_alpha=16,\n",
-    "    bias=\"none\",\n",
-    "    lora_dropout=0.05,\n",
-    "    use_rslora=True,\n",
-    "    modules_to_save=[\"classifier\"],\n",
-    ")\n",
-    "\n",
-    "model = get_peft_model(model, peft_config)\n",
-    "model.print_trainable_parameters()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3cf58dd8",
-   "metadata": {},
-   "source": [
-    "## Preprocessing and Tokenization"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 40,
-   "id": "c7992ba4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Map: 100%|██████████| 18943/18943 [00:01<00:00, 10132.04 examples/s]\n",
-      "Map: 100%|██████████| 997/997 [00:00<00:00, 11498.07 examples/s]"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Tokenization complete!\n",
-      "Tensor columns: ['input_ids', 'attention_mask', 'token_type_ids', 'labels']\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "def _preprocess_function(\n",
-    "    batch: dict,\n",
-    "    tokenizer: PreTrainedTokenizer,\n",
-    "    max_length: int = 512,\n",
-    ") -> BatchEncoding:\n",
-    "    model_inputs = tokenizer(\n",
-    "        batch[\"text\"],\n",
-    "        max_length=max_length,\n",
-    "        truncation=True,\n",
-    "    )\n",
-    "    model_inputs[\"labels\"] = batch[\"label\"]\n",
-    "    return model_inputs\n",
-    "\n",
-    "\n",
-    "preprocess_function = partial(_preprocess_function, tokenizer=tokenizer)\n",
-    "tokenized_datasets = raw_datasets.map(\n",
-    "    preprocess_function,\n",
-    "    batched=True,\n",
-    "    remove_columns=[\"text\", \"label\"],\n",
-    ")\n",
-    "\n",
-    "# Ensure PyTorch tensors and expected columns\n",
-    "available_columns = tokenized_datasets[\"train\"].column_names\n",
-    "tensor_columns = [\n",
-    "    column_name\n",
-    "    for column_name in [\"input_ids\", \"attention_mask\", \"token_type_ids\", \"labels\"]\n",
-    "    if column_name in available_columns\n",
-    "]\n",
-    "tokenized_datasets.set_format(type=\"torch\", columns=tensor_columns)\n",
-    "\n",
-    "print(\"Tokenization complete!\")\n",
-    "print(\"Tensor columns:\", tensor_columns)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "31db700b",
-   "metadata": {},
-   "source": [
-    "## Define Metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 41,
-   "id": "899e4408",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "metric_accuracy = evaluate.load(\"accuracy\")\n",
-    "metric_f1 = evaluate.load(\"f1\")\n",
-    "\n",
-    "\n",
-    "def _compute_metrics(\n",
-    "    eval_pred: tuple[np.ndarray, np.ndarray],\n",
-    "    metric_accuracy: evaluate.EvaluationModule,\n",
-    "    metric_f1: evaluate.EvaluationModule,\n",
-    ") -> dict[str, float]:\n",
-    "    predictions, labels = eval_pred\n",
-    "\n",
-    "    if isinstance(predictions, tuple):\n",
-    "        predictions = predictions[0]\n",
-    "\n",
-    "    predictions = np.argmax(predictions, axis=1)\n",
-    "\n",
-    "    accuracy = metric_accuracy.compute(predictions=predictions, references=labels)\n",
-    "    f1 = metric_f1.compute(predictions=predictions, references=labels)\n",
-    "\n",
-    "    assert accuracy is not None and f1 is not None\n",
-    "\n",
-    "    result = {\n",
-    "        \"accuracy\": accuracy[\"accuracy\"],\n",
-    "        \"f1\": f1[\"f1\"],\n",
-    "    }\n",
-    "\n",
-    "    return result\n",
-    "\n",
-    "\n",
-    "compute_metrics = partial(\n",
-    "    _compute_metrics, metric_accuracy=metric_accuracy, metric_f1=metric_f1\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "34890c4d",
-   "metadata": {},
-   "source": [
-    "## Training Configuration"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "9717d666",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "train_batch_size = 4\n",
-    "gradient_accumulation_steps = 8\n",
-    "eval_batch_size = 4\n",
-    "\n",
-    "training_args = TrainingArguments(\n",
-    "    \"./models/bert-base-raid-classifier\",\n",
-    "    num_train_epochs=5,\n",
-    "    learning_rate=5e-5,\n",
-    "    weight_decay=0.1,\n",
-    "    per_device_train_batch_size=train_batch_size,\n",
-    "    per_device_eval_batch_size=eval_batch_size,\n",
-    "    gradient_accumulation_steps=gradient_accumulation_steps,\n",
-    "    fp16=torch.cuda.is_available(),\n",
-    "    save_strategy=\"steps\",\n",
-    "    save_total_limit=2,\n",
-    "    save_steps=64,\n",
-    "    metric_for_best_model=\"eval_accuracy\",\n",
-    "    load_best_model_at_end=True,\n",
-    "    eval_strategy=\"steps\",\n",
-    "    eval_steps=64,\n",
-    "    logging_strategy=\"steps\",\n",
-    "    logging_steps=16,\n",
-    "    remove_unused_columns=False,\n",
-    ")\n",
-    "\n",
-    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e840a954",
-   "metadata": {},
-   "source": [
-    "## Initialize Trainer and Train"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "0fa3ed58",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Starting training...\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "\n",
-       "    <div>\n",
-       "      \n",
-       "      <progress value='2960' max='2960' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
-       "      [2960/2960 1:03:52, Epoch 5/5]\n",
-       "    </div>\n",
-       "    <table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       " <tr style=\"text-align: left;\">\n",
-       "      <th>Step</th>\n",
-       "      <th>Training Loss</th>\n",
-       "      <th>Validation Loss</th>\n",
-       "      <th>Accuracy</th>\n",
-       "      <th>F1</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <td>64</td>\n",
-       "      <td>5.212345</td>\n",
-       "      <td>0.625602</td>\n",
-       "      <td>0.661986</td>\n",
-       "      <td>0.634093</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>128</td>\n",
-       "      <td>3.753965</td>\n",
-       "      <td>0.458432</td>\n",
-       "      <td>0.771314</td>\n",
-       "      <td>0.809045</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>192</td>\n",
-       "      <td>3.100017</td>\n",
-       "      <td>0.287685</td>\n",
-       "      <td>0.889669</td>\n",
-       "      <td>0.891089</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>256</td>\n",
-       "      <td>2.328572</td>\n",
-       "      <td>0.390553</td>\n",
-       "      <td>0.830491</td>\n",
-       "      <td>0.855432</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>320</td>\n",
-       "      <td>2.129814</td>\n",
-       "      <td>0.238838</td>\n",
-       "      <td>0.911735</td>\n",
-       "      <td>0.917757</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>384</td>\n",
-       "      <td>1.657923</td>\n",
-       "      <td>0.388610</td>\n",
-       "      <td>0.856570</td>\n",
-       "      <td>0.874671</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>448</td>\n",
-       "      <td>1.758504</td>\n",
-       "      <td>0.179176</td>\n",
-       "      <td>0.933801</td>\n",
-       "      <td>0.937262</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>512</td>\n",
-       "      <td>1.352967</td>\n",
-       "      <td>0.344061</td>\n",
-       "      <td>0.867603</td>\n",
-       "      <td>0.882979</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>576</td>\n",
-       "      <td>1.528169</td>\n",
-       "      <td>0.143238</td>\n",
-       "      <td>0.945838</td>\n",
-       "      <td>0.947368</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>640</td>\n",
-       "      <td>1.692302</td>\n",
-       "      <td>0.185934</td>\n",
-       "      <td>0.925777</td>\n",
-       "      <td>0.930582</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>704</td>\n",
-       "      <td>1.194244</td>\n",
-       "      <td>0.189194</td>\n",
-       "      <td>0.927783</td>\n",
-       "      <td>0.932203</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>768</td>\n",
-       "      <td>1.089103</td>\n",
-       "      <td>0.191697</td>\n",
-       "      <td>0.926780</td>\n",
-       "      <td>0.931455</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>832</td>\n",
-       "      <td>1.313780</td>\n",
-       "      <td>0.133464</td>\n",
-       "      <td>0.949850</td>\n",
-       "      <td>0.950593</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>896</td>\n",
-       "      <td>1.144064</td>\n",
-       "      <td>0.161593</td>\n",
-       "      <td>0.943831</td>\n",
-       "      <td>0.946463</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>960</td>\n",
-       "      <td>1.503407</td>\n",
-       "      <td>0.211920</td>\n",
-       "      <td>0.921765</td>\n",
-       "      <td>0.927374</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1024</td>\n",
-       "      <td>1.106765</td>\n",
-       "      <td>0.182482</td>\n",
-       "      <td>0.931795</td>\n",
-       "      <td>0.935606</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1088</td>\n",
-       "      <td>1.450451</td>\n",
-       "      <td>0.127360</td>\n",
-       "      <td>0.956871</td>\n",
-       "      <td>0.958212</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1152</td>\n",
-       "      <td>1.380015</td>\n",
-       "      <td>0.131538</td>\n",
-       "      <td>0.957874</td>\n",
-       "      <td>0.959064</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1216</td>\n",
-       "      <td>0.755666</td>\n",
-       "      <td>0.158870</td>\n",
-       "      <td>0.940822</td>\n",
-       "      <td>0.943432</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1280</td>\n",
-       "      <td>0.863713</td>\n",
-       "      <td>0.157785</td>\n",
-       "      <td>0.943831</td>\n",
-       "      <td>0.946565</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1344</td>\n",
-       "      <td>0.821364</td>\n",
-       "      <td>0.172321</td>\n",
-       "      <td>0.944835</td>\n",
-       "      <td>0.947469</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1408</td>\n",
-       "      <td>0.957095</td>\n",
-       "      <td>0.226298</td>\n",
-       "      <td>0.922768</td>\n",
-       "      <td>0.927835</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1472</td>\n",
-       "      <td>0.868089</td>\n",
-       "      <td>0.197520</td>\n",
-       "      <td>0.934804</td>\n",
-       "      <td>0.938505</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1536</td>\n",
-       "      <td>1.310811</td>\n",
-       "      <td>0.140865</td>\n",
-       "      <td>0.953862</td>\n",
-       "      <td>0.955426</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1600</td>\n",
-       "      <td>0.708888</td>\n",
-       "      <td>0.152195</td>\n",
-       "      <td>0.943831</td>\n",
-       "      <td>0.946565</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1664</td>\n",
-       "      <td>0.717255</td>\n",
-       "      <td>0.176768</td>\n",
-       "      <td>0.942828</td>\n",
-       "      <td>0.945663</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1728</td>\n",
-       "      <td>1.143681</td>\n",
-       "      <td>0.156816</td>\n",
-       "      <td>0.951856</td>\n",
-       "      <td>0.953757</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1792</td>\n",
-       "      <td>0.638254</td>\n",
-       "      <td>0.176596</td>\n",
-       "      <td>0.944835</td>\n",
-       "      <td>0.947469</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1856</td>\n",
-       "      <td>1.133300</td>\n",
-       "      <td>0.119119</td>\n",
-       "      <td>0.967904</td>\n",
-       "      <td>0.968317</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1920</td>\n",
-       "      <td>1.061837</td>\n",
-       "      <td>0.140624</td>\n",
-       "      <td>0.957874</td>\n",
-       "      <td>0.959381</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1984</td>\n",
-       "      <td>0.708067</td>\n",
-       "      <td>0.189490</td>\n",
-       "      <td>0.940822</td>\n",
-       "      <td>0.943863</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2048</td>\n",
-       "      <td>0.761451</td>\n",
-       "      <td>0.150488</td>\n",
-       "      <td>0.951856</td>\n",
-       "      <td>0.953846</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2112</td>\n",
-       "      <td>0.609547</td>\n",
-       "      <td>0.189622</td>\n",
-       "      <td>0.940822</td>\n",
-       "      <td>0.943863</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2176</td>\n",
-       "      <td>0.803254</td>\n",
-       "      <td>0.173354</td>\n",
-       "      <td>0.946841</td>\n",
-       "      <td>0.949282</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2240</td>\n",
-       "      <td>0.664540</td>\n",
-       "      <td>0.154308</td>\n",
-       "      <td>0.952859</td>\n",
-       "      <td>0.954764</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2304</td>\n",
-       "      <td>0.691763</td>\n",
-       "      <td>0.144127</td>\n",
-       "      <td>0.963892</td>\n",
-       "      <td>0.964706</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2368</td>\n",
-       "      <td>1.092195</td>\n",
-       "      <td>0.157182</td>\n",
-       "      <td>0.957874</td>\n",
-       "      <td>0.959381</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2432</td>\n",
-       "      <td>0.752286</td>\n",
-       "      <td>0.231035</td>\n",
-       "      <td>0.933801</td>\n",
-       "      <td>0.937736</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2496</td>\n",
-       "      <td>0.757014</td>\n",
-       "      <td>0.185019</td>\n",
-       "      <td>0.948847</td>\n",
-       "      <td>0.951103</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2560</td>\n",
-       "      <td>0.766771</td>\n",
-       "      <td>0.153019</td>\n",
-       "      <td>0.958877</td>\n",
-       "      <td>0.960078</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2624</td>\n",
-       "      <td>0.434590</td>\n",
-       "      <td>0.201383</td>\n",
-       "      <td>0.946841</td>\n",
-       "      <td>0.949282</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2688</td>\n",
-       "      <td>0.565482</td>\n",
-       "      <td>0.181478</td>\n",
-       "      <td>0.952859</td>\n",
-       "      <td>0.954764</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2752</td>\n",
-       "      <td>0.568177</td>\n",
-       "      <td>0.201250</td>\n",
-       "      <td>0.946841</td>\n",
-       "      <td>0.949282</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2816</td>\n",
-       "      <td>0.611295</td>\n",
-       "      <td>0.173839</td>\n",
-       "      <td>0.954865</td>\n",
-       "      <td>0.956606</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2880</td>\n",
-       "      <td>0.716351</td>\n",
-       "      <td>0.187448</td>\n",
-       "      <td>0.948847</td>\n",
-       "      <td>0.951103</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2944</td>\n",
-       "      <td>0.603852</td>\n",
-       "      <td>0.184578</td>\n",
-       "      <td>0.948847</td>\n",
-       "      <td>0.951103</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table><p>"
-      ],
-      "text/plain": [
-       "<IPython.core.display.HTML object>"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "TrainOutput(global_step=2960, training_loss=1.3125710455146995, metrics={'train_runtime': 3832.8474, 'train_samples_per_second': 24.711, 'train_steps_per_second': 0.772, 'total_flos': 8360830141838376.0, 'train_loss': 1.3125710455146995, 'epoch': 5.0})"
-      ]
-     },
-     "execution_count": 43,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "trainer = Trainer(\n",
-    "    model,\n",
-    "    training_args,\n",
-    "    train_dataset=tokenized_datasets[\"train\"],\n",
-    "    eval_dataset=tokenized_datasets[\"test\"],\n",
-    "    data_collator=data_collator,\n",
-    "    compute_metrics=compute_metrics,\n",
-    ")\n",
-    "\n",
-    "print(\"Starting training...\")\n",
-    "trainer.train()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cde9bbb1",
-   "metadata": {},
-   "source": [
-    "## Final Evaluation"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 44,
-   "id": "bb81afb9",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Evaluating model...\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "\n",
-       "    <div>\n",
-       "      \n",
-       "      <progress value='250' max='250' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
-       "      [250/250 00:14]\n",
-       "    </div>\n",
-       "    "
-      ],
-      "text/plain": [
-       "<IPython.core.display.HTML object>"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "Final Results:\n",
-      "Accuracy: 0.9679\n",
-      "F1 Score: 0.9683\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"Evaluating model...\")\n",
-    "eval_results = trainer.evaluate()\n",
-    "print(\"\\nFinal Results:\")\n",
-    "print(f\"Accuracy: {eval_results['eval_accuracy']:.4f}\")\n",
-    "print(f\"F1 Score: {eval_results['eval_f1']:.4f}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8bf17a40",
-   "metadata": {},
-   "source": [
-    "## Save Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 45,
-   "id": "e580bfd6",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Model saved successfully!\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Save the final model\n",
-    "trainer.save_model(\"./trained_model/bert-base-raid-final\")\n",
-    "print(\"Model saved successfully!\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "99c0a2f0",
-   "metadata": {},
-   "source": [
-    "## test the model\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 46,
-   "id": "016cc53e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Prediction for human-written text:\n",
-      "{'predicted_label': 0, 'probability_human': 0.9988395571708679, 'probability_ai': 0.0011604195460677147}\n",
-      "\n",
-      "Prediction for AI-generated text:\n",
-      "{'predicted_label': 0, 'probability_human': 0.9988927245140076, 'probability_ai': 0.0011073390487581491}\n"
-     ]
-    }
-   ],
-   "source": [
-    "def predict(text: str) -> dict[str, float]:\n",
-    "    inputs = tokenizer(\n",
-    "        text,\n",
-    "        max_length=512,\n",
-    "        truncation=True,\n",
-    "        return_tensors=\"pt\",\n",
-    "    ).to(model.device)\n",
-    "\n",
-    "    with torch.no_grad():\n",
-    "        outputs = model(**inputs)\n",
-    "        logits = outputs.logits\n",
-    "        probabilities = torch.softmax(logits, dim=-1).cpu().numpy()[0]\n",
-    "        predicted_label = np.argmax(probabilities)\n",
-    "\n",
-    "    return {\n",
-    "        \"predicted_label\": int(predicted_label),\n",
-    "        \"probability_human\": float(probabilities[0]),\n",
-    "        \"probability_ai\": float(probabilities[1]),\n",
-    "    }\n",
-    "    \n",
-    "text = \"Ai will replace this world. today in the nepal election someone might win by using ai.\"\n",
-    "text_by_ai = \"This is a sample text generated by AI.Also This is an long text by AI.\"\n",
-    "print(\"Prediction for human-written text:\")\n",
-    "print(predict(text))\n",
-    "print(\"\\nPrediction for AI-generated text:\")\n",
-    "print(predict(text_by_ai))\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7c6c2a5d",
-   "metadata": {},
-   "source": [
-    "def predict"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 47,
-   "id": "1b287605",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Using 512 samples for RAID quick test\n"
-     ]
-    },
-    {
-     "ename": "OutOfMemoryError",
-     "evalue": "CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 3.68 GiB of which 719.12 MiB is free. Process 2034 has 46.03 MiB memory in use. Process 1961 has 6.78 MiB memory in use. Including non-PyTorch memory, this process has 2.90 GiB memory in use. Of the allocated memory 2.71 GiB is allocated by PyTorch, and 85.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
-      "\u001b[31mOutOfMemoryError\u001b[39m                          Traceback (most recent call last)",
-      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[47]\u001b[39m\u001b[32m, line 32\u001b[39m\n\u001b[32m     28\u001b[39m     \u001b[38;5;66;03m# Return AI-class probability for each input text\u001b[39;00m\n\u001b[32m     29\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m probabilities[:, \u001b[32m1\u001b[39m].astype(\u001b[38;5;28mfloat\u001b[39m).tolist()\n\u001b[32m---> \u001b[39m\u001b[32m32\u001b[39m predictions = \u001b[43mrun_detection\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmy_detector\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtest_df\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     33\u001b[39m evaluation_result = run_evaluation(predictions, test_df)\n\u001b[32m     35\u001b[39m evaluation_result\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/raid/detect.py:6\u001b[39m, in \u001b[36mrun_detection\u001b[39m\u001b[34m(f, df)\u001b[39m\n\u001b[32m      3\u001b[39m scores_df = df[[\u001b[33m\"\u001b[39m\u001b[33mid\u001b[39m\u001b[33m\"\u001b[39m]].copy()\n\u001b[32m      5\u001b[39m \u001b[38;5;66;03m# Run the detector function on the dataset and put output in score column\u001b[39;00m\n\u001b[32m----> \u001b[39m\u001b[32m6\u001b[39m scores_df[\u001b[33m\"\u001b[39m\u001b[33mscore\u001b[39m\u001b[33m\"\u001b[39m] = \u001b[43mf\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdf\u001b[49m\u001b[43m[\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mgeneration\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m.\u001b[49m\u001b[43mtolist\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m      8\u001b[39m \u001b[38;5;66;03m# Convert scores and ids to dict in 'records' format for seralization\u001b[39;00m\n\u001b[32m      9\u001b[39m \u001b[38;5;66;03m# e.g. [{'id':'...', 'score':0}, {'id':'...', 'score':1}, ...]\u001b[39;00m\n\u001b[32m     10\u001b[39m results = scores_df[[\u001b[33m\"\u001b[39m\u001b[33mid\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mscore\u001b[39m\u001b[33m\"\u001b[39m]].to_dict(orient=\u001b[33m\"\u001b[39m\u001b[33mrecords\u001b[39m\u001b[33m\"\u001b[39m)\n",
-      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[47]\u001b[39m\u001b[32m, line 24\u001b[39m, in \u001b[36mmy_detector\u001b[39m\u001b[34m(texts)\u001b[39m\n\u001b[32m     22\u001b[39m model.eval()\n\u001b[32m     23\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m torch.no_grad():\n\u001b[32m---> \u001b[39m\u001b[32m24\u001b[39m     outputs = \u001b[43mmodel\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43minputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     25\u001b[39m     logits = outputs.logits\n\u001b[32m     26\u001b[39m     probabilities = torch.softmax(logits, dim=-\u001b[32m1\u001b[39m).cpu().numpy()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1736\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1734\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1735\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1736\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1747\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1742\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1743\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1744\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1745\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1746\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1747\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1749\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1750\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/accelerate/utils/operations.py:819\u001b[39m, in \u001b[36mconvert_outputs_to_fp32.<locals>.forward\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m    818\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mforward\u001b[39m(*args, **kwargs):\n\u001b[32m--> \u001b[39m\u001b[32m819\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmodel_forward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/accelerate/utils/operations.py:807\u001b[39m, in \u001b[36mConvertOutputsToFp32.__call__\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m    806\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m__call__\u001b[39m(\u001b[38;5;28mself\u001b[39m, *args, **kwargs):\n\u001b[32m--> \u001b[39m\u001b[32m807\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m convert_to_fp32(\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mmodel_forward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m)\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/amp/autocast_mode.py:44\u001b[39m, in \u001b[36mautocast_decorator.<locals>.decorate_autocast\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m     41\u001b[39m \u001b[38;5;129m@functools\u001b[39m.wraps(func)\n\u001b[32m     42\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mdecorate_autocast\u001b[39m(*args, **kwargs):\n\u001b[32m     43\u001b[39m     \u001b[38;5;28;01mwith\u001b[39;00m autocast_instance:\n\u001b[32m---> \u001b[39m\u001b[32m44\u001b[39m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/peft/peft_model.py:921\u001b[39m, in \u001b[36mPeftModel.forward\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m    919\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m._enable_peft_forward_hooks(*args, **kwargs):\n\u001b[32m    920\u001b[39m     kwargs = {k: v \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m kwargs.items() \u001b[38;5;28;01mif\u001b[39;00m k \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.special_peft_forward_args}\n\u001b[32m--> \u001b[39m\u001b[32m921\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mget_base_model\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1736\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1734\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1735\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1736\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1747\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1742\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1743\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1744\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1745\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1746\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1747\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1749\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1750\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/utils/generic.py:835\u001b[39m, in \u001b[36mcan_return_tuple.<locals>.wrapper\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m    833\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m return_dict_passed \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m    834\u001b[39m     return_dict = return_dict_passed\n\u001b[32m--> \u001b[39m\u001b[32m835\u001b[39m output = \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    836\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m return_dict \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(output, \u001b[38;5;28mtuple\u001b[39m):\n\u001b[32m    837\u001b[39m     output = output.to_tuple()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py:1162\u001b[39m, in \u001b[36mBertForSequenceClassification.forward\u001b[39m\u001b[34m(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, labels, **kwargs)\u001b[39m\n\u001b[32m   1144\u001b[39m \u001b[38;5;129m@can_return_tuple\u001b[39m\n\u001b[32m   1145\u001b[39m \u001b[38;5;129m@auto_docstring\u001b[39m\n\u001b[32m   1146\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mforward\u001b[39m(\n\u001b[32m   (...)\u001b[39m\u001b[32m   1154\u001b[39m     **kwargs: Unpack[TransformersKwargs],\n\u001b[32m   1155\u001b[39m ) -> \u001b[38;5;28mtuple\u001b[39m[torch.Tensor] | SequenceClassifierOutput:\n\u001b[32m   1156\u001b[39m \u001b[38;5;250m    \u001b[39m\u001b[33mr\u001b[39m\u001b[33;03m\"\"\"\u001b[39;00m\n\u001b[32m   1157\u001b[39m \u001b[33;03m    labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\u001b[39;00m\n\u001b[32m   1158\u001b[39m \u001b[33;03m        Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,\u001b[39;00m\n\u001b[32m   1159\u001b[39m \u001b[33;03m        config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If\u001b[39;00m\n\u001b[32m   1160\u001b[39m \u001b[33;03m        `config.num_labels > 1` a classification loss is computed (Cross-Entropy).\u001b[39;00m\n\u001b[32m   1161\u001b[39m \u001b[33;03m    \"\"\"\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1162\u001b[39m     outputs = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mbert\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m   1163\u001b[39m \u001b[43m        \u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1164\u001b[39m \u001b[43m        \u001b[49m\u001b[43mattention_mask\u001b[49m\u001b[43m=\u001b[49m\u001b[43mattention_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1165\u001b[39m \u001b[43m        \u001b[49m\u001b[43mtoken_type_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mtoken_type_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1166\u001b[39m \u001b[43m        \u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1167\u001b[39m \u001b[43m        \u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m=\u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1168\u001b[39m \u001b[43m        \u001b[49m\u001b[43mreturn_dict\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[32m   1169\u001b[39m \u001b[43m        \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   1170\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1172\u001b[39m     pooled_output = outputs[\u001b[32m1\u001b[39m]\n\u001b[32m   1174\u001b[39m     pooled_output = \u001b[38;5;28mself\u001b[39m.dropout(pooled_output)\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1736\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1734\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1735\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1736\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1747\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1742\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1743\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1744\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1745\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1746\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1747\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1749\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1750\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/utils/generic.py:1002\u001b[39m, in \u001b[36mcheck_model_inputs.<locals>.wrapped_fn.<locals>.wrapper\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1000\u001b[39m             outputs = func(\u001b[38;5;28mself\u001b[39m, *args, **kwargs)\n\u001b[32m   1001\u001b[39m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1002\u001b[39m         outputs = \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1003\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m original_exception:\n\u001b[32m   1004\u001b[39m     \u001b[38;5;66;03m# If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\u001b[39;00m\n\u001b[32m   1005\u001b[39m     \u001b[38;5;66;03m# Get a TypeError even after removing the recordable kwargs -> re-raise the original exception\u001b[39;00m\n\u001b[32m   1006\u001b[39m     \u001b[38;5;66;03m# Otherwise -> we're probably missing `**kwargs` in the decorated function\u001b[39;00m\n\u001b[32m   1007\u001b[39m     kwargs_without_recordable = {k: v \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m kwargs.items() \u001b[38;5;28;01mif\u001b[39;00m k \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m recordable_keys}\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py:679\u001b[39m, in \u001b[36mBertModel.forward\u001b[39m\u001b[34m(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, cache_position, **kwargs)\u001b[39m\n\u001b[32m    676\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m cache_position \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m    677\u001b[39m     cache_position = torch.arange(past_key_values_length, past_key_values_length + seq_length, device=device)\n\u001b[32m--> \u001b[39m\u001b[32m679\u001b[39m embedding_output = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43membeddings\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    680\u001b[39m \u001b[43m    \u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    681\u001b[39m \u001b[43m    \u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    682\u001b[39m \u001b[43m    \u001b[49m\u001b[43mtoken_type_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mtoken_type_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    683\u001b[39m \u001b[43m    \u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m=\u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    684\u001b[39m \u001b[43m    \u001b[49m\u001b[43mpast_key_values_length\u001b[49m\u001b[43m=\u001b[49m\u001b[43mpast_key_values_length\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    685\u001b[39m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    687\u001b[39m attention_mask, encoder_attention_mask = \u001b[38;5;28mself\u001b[39m._create_attention_masks(\n\u001b[32m    688\u001b[39m     attention_mask=attention_mask,\n\u001b[32m    689\u001b[39m     encoder_attention_mask=encoder_attention_mask,\n\u001b[32m   (...)\u001b[39m\u001b[32m    693\u001b[39m     past_key_values=past_key_values,\n\u001b[32m    694\u001b[39m )\n\u001b[32m    696\u001b[39m encoder_outputs = \u001b[38;5;28mself\u001b[39m.encoder(\n\u001b[32m    697\u001b[39m     embedding_output,\n\u001b[32m    698\u001b[39m     attention_mask=attention_mask,\n\u001b[32m   (...)\u001b[39m\u001b[32m    705\u001b[39m     **kwargs,\n\u001b[32m    706\u001b[39m )\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1736\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1734\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1735\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1736\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py:1747\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1742\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1743\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1744\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1745\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1746\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1747\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1749\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1750\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py:107\u001b[39m, in \u001b[36mBertEmbeddings.forward\u001b[39m\u001b[34m(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)\u001b[39m\n\u001b[32m    104\u001b[39m embeddings = inputs_embeds + token_type_embeddings\n\u001b[32m    106\u001b[39m position_embeddings = \u001b[38;5;28mself\u001b[39m.position_embeddings(position_ids)\n\u001b[32m--> \u001b[39m\u001b[32m107\u001b[39m embeddings = \u001b[43membeddings\u001b[49m\u001b[43m \u001b[49m\u001b[43m+\u001b[49m\u001b[43m \u001b[49m\u001b[43mposition_embeddings\u001b[49m\n\u001b[32m    109\u001b[39m embeddings = \u001b[38;5;28mself\u001b[39m.LayerNorm(embeddings)\n\u001b[32m    110\u001b[39m embeddings = \u001b[38;5;28mself\u001b[39m.dropout(embeddings)\n",
-      "\u001b[31mOutOfMemoryError\u001b[39m: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 3.68 GiB of which 719.12 MiB is free. Process 2034 has 46.03 MiB memory in use. Process 1961 has 6.78 MiB memory in use. Including non-PyTorch memory, this process has 2.90 GiB memory in use. Of the allocated memory 2.71 GiB is allocated by PyTorch, and 85.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)"
-     ]
-    }
-   ],
-   "source": [
-    "from raid import run_detection, run_evaluation\n",
-    "from raid.utils import load_data\n",
-    "\n",
-    "# Use test split and cap sample size for a quick RAID validation\n",
-    "test_df = load_data(split=\"test\")\n",
-    "sample_size = min(int(len(test_df) * 0.02), 512)\n",
-    "test_df = test_df.sample(n=sample_size, random_state=42)\n",
-    "\n",
-    "print(f\"Using {len(test_df)} samples for RAID quick test\")\n",
-    "\n",
-    "\n",
-    "def my_detector(texts: list[str]) -> list[float]:\n",
-    "    # RAID passes a batch/list of strings and expects a list of scores\n",
-    "    inputs = tokenizer(\n",
-    "        texts,\n",
-    "        max_length=512,\n",
-    "        truncation=True,\n",
-    "        padding=True,\n",
-    "        return_tensors=\"pt\",\n",
-    "    ).to(model.device)\n",
-    "\n",
-    "    model.eval()\n",
-    "    with torch.no_grad():\n",
-    "        outputs = model(**inputs)\n",
-    "        logits = outputs.logits\n",
-    "        probabilities = torch.softmax(logits, dim=-1).cpu().numpy()\n",
-    "\n",
-    "    # Return AI-class probability for each input text\n",
-    "    return probabilities[:, 1].astype(float).tolist()\n",
-    "\n",
-    "\n",
-    "predictions = run_detection(my_detector, test_df)\n",
-    "evaluation_result = run_evaluation(predictions, test_df)\n",
-    "\n",
-    "evaluation_result"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6b6eb543",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "ml",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

notebook/ai_vs_human/mainv2.ipynb

@@ -1,1170 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "464eefd0",
-   "metadata": {},
-   "source": [
-    "# AI vs Human Detector V2\n",
-    "This notebook trains a V2 model that explicitly supports short inputs (including sentences under 50 words) and saves artifacts in `v2_model/`."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0be0e8d9",
-   "metadata": {},
-   "source": [
-    "## ✅ Bug Fixes & Capabilities\n",
-    "\n",
-    "**Fixed Issues:**\n",
-    "1. ✅ Runtime error when calling `trainer.evaluate()` after training (removed duplicate evaluation)\n",
-    "2. ✅ Missing `accelerate` dependency (auto-installs if needed)\n",
-    "3. ✅ Recursive dataset loading from `./DATASET/` folder (supports `.jsonl`, `.json`, `.csv`)\n",
-    "4. ✅ Short sentence support (<50 words) with data augmentation\n",
-    "\n",
-    "**Model Capabilities:**\n",
-    "- ✅ Works with **all sentence types**: very short (1-10 words), short (10-50), medium (50-150), long (150+)\n",
-    "- ✅ Handles edge cases: single words, special characters, numbers, mixed formats\n",
-    "- ✅ Batch prediction support\n",
-    "- ✅ Saves to `v2_model/` with tokenizer, config, and label map\n",
-    "- ✅ Can be loaded independently after saving\n",
-    "\n",
-    "**Architecture:** DistilRoBERTa-base (faster, lighter than BERT)\n",
-    "\n",
-    "**Quick Start:**\n",
-    "1. Run cells 1-7 to prepare data\n",
-    "2. Run cell 8 to train (takes ~15-30 min on GPU)\n",
-    "3. Run cell 9 to save to `v2_model/`\n",
-    "4. Run cells 10-12 to test all sentence types"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3a8134db",
-   "metadata": {},
-   "source": [
-    "## Additional Testing: Extreme Edge Cases & Batch Prediction"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "f400f763",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Note: you may need to restart the kernel to use updated packages.\n"
-     ]
-    }
-   ],
-   "source": [
-    "%pip install -q -U datasets evaluate transformers torch pandas scikit-learn accelerate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "0c3d4d6d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n",
-      "/home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'Could not load this library: /home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?\n",
-      "  warn(\n",
-      "/home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().\n",
-      "  warnings.warn(_BETA_TRANSFORMS_WARNING)\n",
-      "/home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().\n",
-      "  warnings.warn(_BETA_TRANSFORMS_WARNING)\n"
-     ]
-    }
-   ],
-   "source": [
-    "from __future__ import annotations\n",
-    "\n",
-    "from dataclasses import dataclass\n",
-    "from functools import partial\n",
-    "from pathlib import Path\n",
-    "import json\n",
-    "import random\n",
-    "\n",
-    "import datasets\n",
-    "from datasets import Dataset, DatasetDict, concatenate_datasets\n",
-    "import evaluate\n",
-    "import numpy as np\n",
-    "import pandas as pd\n",
-    "import torch\n",
-    "from transformers import (\n",
-    "    AutoModelForSequenceClassification,\n",
-    "    AutoTokenizer,\n",
-    "    BatchEncoding,\n",
-    "    DataCollatorWithPadding,\n",
-    "    PreTrainedTokenizer,\n",
-    "    Trainer,\n",
-    "    TrainingArguments,\n",
-    ")\n",
-    "from packaging import version"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "624d23ba",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Base model: distilroberta-base\n",
-      "Device: cuda\n",
-      "Output path: ./v2_model\n"
-     ]
-    }
-   ],
-   "source": [
-    "@dataclass\n",
-    "class V2Config:\n",
-    "    base_model_name: str = \"distilroberta-base\"\n",
-    "    max_samples: int = 20000\n",
-    "    max_length: int = 256\n",
-    "    short_word_limit: int = 50\n",
-    "    short_aug_ratio: float = 0.35\n",
-    "    output_dir: str = \"./v2_model\"\n",
-    "    seed: int = 42\n",
-    "\n",
-    "\n",
-    "cfg = V2Config()\n",
-    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
-    "random.seed(cfg.seed)\n",
-    "np.random.seed(cfg.seed)\n",
-    "torch.manual_seed(cfg.seed)\n",
-    "\n",
-    "print(f\"Base model: {cfg.base_model_name}\")\n",
-    "print(f\"Device: {DEVICE}\")\n",
-    "print(f\"Output path: {cfg.output_dir}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "0a1f860a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def normalize_text(text: str) -> str:\n",
-    "    return \" \".join(str(text).split()).strip()\n",
-    "\n",
-    "\n",
-    "def count_words(text: str) -> int:\n",
-    "    return len(normalize_text(text).split())\n",
-    "\n",
-    "\n",
-    "def _load_local_file_to_text_labels(file_path: Path) -> tuple[list[str], list[int]]:\n",
-    "    texts: list[str] = []\n",
-    "    labels: list[int] = []\n",
-    "\n",
-    "    try:\n",
-    "        suffix = file_path.suffix.lower()\n",
-    "        if suffix == \".jsonl\":\n",
-    "            df = pd.read_json(file_path, lines=True)\n",
-    "        elif suffix == \".json\":\n",
-    "            df = pd.read_json(file_path)\n",
-    "        elif suffix == \".csv\":\n",
-    "            df = pd.read_csv(file_path)\n",
-    "        else:\n",
-    "            return texts, labels\n",
-    "\n",
-    "        if {\"human_text\", \"ai_text\"}.issubset(df.columns):\n",
-    "            human_texts = [normalize_text(x) for x in df[\"human_text\"].dropna().tolist()]\n",
-    "            ai_texts = [normalize_text(x) for x in df[\"ai_text\"].dropna().tolist()]\n",
-    "            human_texts = [x for x in human_texts if x]\n",
-    "            ai_texts = [x for x in ai_texts if x]\n",
-    "            texts.extend(human_texts)\n",
-    "            labels.extend([0] * len(human_texts))\n",
-    "            texts.extend(ai_texts)\n",
-    "            labels.extend([1] * len(ai_texts))\n",
-    "            return texts, labels\n",
-    "\n",
-    "        # Alternative schema fallback: text + label/ai_gen columns.\n",
-    "        if \"text\" in df.columns and (\"label\" in df.columns or \"ai_gen\" in df.columns):\n",
-    "            label_col = \"label\" if \"label\" in df.columns else \"ai_gen\"\n",
-    "            for _, row in df.iterrows():\n",
-    "                text = normalize_text(row.get(\"text\", \"\"))\n",
-    "                if not text:\n",
-    "                    continue\n",
-    "                val = str(row.get(label_col, \"\")).strip().lower()\n",
-    "                is_ai = val in {\"1\", \"true\", \"ai\", \"ai-generated\", \"ai_generated\"}\n",
-    "                texts.append(text)\n",
-    "                labels.append(1 if is_ai else 0)\n",
-    "            return texts, labels\n",
-    "\n",
-    "    except Exception as error:\n",
-    "        print(f\"Skipped file due to parse error: {file_path} ({error})\")\n",
-    "\n",
-    "    return texts, labels\n",
-    "\n",
-    "\n",
-    "def get_combined_dataset(max_samples: int = 20000, use_local: bool = True) -> DatasetDict:\n",
-    "    all_texts: list[str] = []\n",
-    "    all_labels: list[int] = []\n",
-    "\n",
-    "    try:\n",
-    "        hc3 = datasets.load_dataset(\"Hello-SimpleAI/HC3\", \"all\", split=\"train\")\n",
-    "        for row in hc3:\n",
-    "            for answer in row.get(\"human_answers\", [])[:1]:\n",
-    "                text = normalize_text(answer)\n",
-    "                if text:\n",
-    "                    all_texts.append(text)\n",
-    "                    all_labels.append(0)\n",
-    "            for answer in row.get(\"chatgpt_answers\", [])[:1]:\n",
-    "                text = normalize_text(answer)\n",
-    "                if text:\n",
-    "                    all_texts.append(text)\n",
-    "                    all_labels.append(1)\n",
-    "        print(f\"HC3 samples: {len(all_texts)}\")\n",
-    "    except Exception as error:\n",
-    "        print(f\"HC3 unavailable: {error}\")\n",
-    "\n",
-    "    if use_local:\n",
-    "        dataset_root = Path(\"./DATASET\")\n",
-    "        candidates = list(dataset_root.rglob(\"*.jsonl\")) + list(dataset_root.rglob(\"*.json\")) + list(dataset_root.rglob(\"*.csv\"))\n",
-    "\n",
-    "        local_before = len(all_texts)\n",
-    "        for file_path in candidates:\n",
-    "            texts, labels = _load_local_file_to_text_labels(file_path)\n",
-    "            all_texts.extend(texts)\n",
-    "            all_labels.extend(labels)\n",
-    "\n",
-    "        print(f\"Local recursive files scanned: {len(candidates)}\")\n",
-    "        print(f\"Local samples added: {len(all_texts) - local_before}\")\n",
-    "\n",
-    "    if not all_texts:\n",
-    "        raise ValueError(\"No training data loaded from HC3 or local dataset.\")\n",
-    "\n",
-    "    ds = Dataset.from_dict({\"text\": all_texts, \"label\": all_labels})\n",
-    "    ds = ds.filter(lambda x: x[\"text\"] is not None and len(normalize_text(x[\"text\"])) > 0)\n",
-    "    ds = ds.shuffle(seed=cfg.seed)\n",
-    "    if len(ds) > max_samples:\n",
-    "        ds = ds.select(range(max_samples))\n",
-    "\n",
-    "    split = ds.train_test_split(test_size=0.1, seed=cfg.seed)\n",
-    "    return split\n",
-    "\n",
-    "\n",
-    "def add_short_text_variants(dataset: Dataset, short_word_limit: int = 50, ratio: float = 0.35) -> Dataset:\n",
-    "    short_texts: list[str] = []\n",
-    "    short_labels: list[int] = []\n",
-    "\n",
-    "    for row in dataset:\n",
-    "        text = normalize_text(row[\"text\"])\n",
-    "        label = int(row[\"label\"])\n",
-    "        words = text.split()\n",
-    "\n",
-    "        if len(words) <= short_word_limit:\n",
-    "            if random.random() < ratio:\n",
-    "                short_texts.append(text)\n",
-    "                short_labels.append(label)\n",
-    "            continue\n",
-    "\n",
-    "        # Keep first N words as a short variant to train behavior on short inputs.\n",
-    "        if random.random() < ratio:\n",
-    "            short_text = \" \".join(words[:short_word_limit])\n",
-    "            short_texts.append(short_text)\n",
-    "            short_labels.append(label)\n",
-    "\n",
-    "    if not short_texts:\n",
-    "        return dataset\n",
-    "\n",
-    "    aug = Dataset.from_dict({\"text\": short_texts, \"label\": short_labels})\n",
-    "    return concatenate_datasets([dataset, aug]).shuffle(seed=cfg.seed)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "889c5e58",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "HC3 unavailable: Dataset scripts are no longer supported, but found HC3.py\n",
-      "Skipped file due to parse error: DATASET/test.csv (No columns to parse from file)\n",
-      "Local recursive files scanned: 2\n",
-      "Local samples added: 19940\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Filter: 100%|██████████| 19940/19940 [00:00<00:00, 133317.22 examples/s]\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Train samples: 24213\n",
-      "Eval samples: 1994\n",
-      "Train short (<50 words): 6839\n",
-      "Eval short (<50 words): 569\n"
-     ]
-    }
-   ],
-   "source": [
-    "raw_data = get_combined_dataset(max_samples=cfg.max_samples)\n",
-    "train_data = add_short_text_variants(\n",
-    "    raw_data[\"train\"],\n",
-    "    short_word_limit=cfg.short_word_limit,\n",
-    "    ratio=cfg.short_aug_ratio,\n",
-    ")\n",
-    "eval_data = raw_data[\"test\"]\n",
-    "\n",
-    "short_train = sum(count_words(t) < 50 for t in train_data[\"text\"])\n",
-    "short_eval = sum(count_words(t) < 50 for t in eval_data[\"text\"])\n",
-    "\n",
-    "print(f\"Train samples: {len(train_data)}\")\n",
-    "print(f\"Eval samples: {len(eval_data)}\")\n",
-    "print(f\"Train short (<50 words): {short_train}\")\n",
-    "print(f\"Eval short (<50 words): {short_eval}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "e8a2ff3e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Loading weights: 100%|██████████| 101/101 [00:00<00:00, 8921.80it/s]\n",
-      "\u001b[1mRobertaForSequenceClassification LOAD REPORT\u001b[0m from: distilroberta-base\n",
-      "Key                         | Status     | \n",
-      "----------------------------+------------+-\n",
-      "roberta.pooler.dense.weight | UNEXPECTED | \n",
-      "lm_head.dense.weight        | UNEXPECTED | \n",
-      "roberta.pooler.dense.bias   | UNEXPECTED | \n",
-      "lm_head.layer_norm.bias     | UNEXPECTED | \n",
-      "lm_head.dense.bias          | UNEXPECTED | \n",
-      "lm_head.layer_norm.weight   | UNEXPECTED | \n",
-      "lm_head.bias                | UNEXPECTED | \n",
-      "classifier.out_proj.bias    | MISSING    | \n",
-      "classifier.dense.weight     | MISSING    | \n",
-      "classifier.dense.bias       | MISSING    | \n",
-      "classifier.out_proj.weight  | MISSING    | \n",
-      "\n",
-      "\u001b[3mNotes:\n",
-      "- UNEXPECTED\u001b[3m\t:can be ignored when loading from different task/architecture; not ok if you expect identical arch.\n",
-      "- MISSING\u001b[3m\t:those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.\u001b[0m\n",
-      "Map: 100%|██████████| 24213/24213 [00:01<00:00, 12285.23 examples/s]\n",
-      "Map: 100%|██████████| 1994/1994 [00:00<00:00, 11737.65 examples/s]\n"
-     ]
-    }
-   ],
-   "source": [
-    "tokenizer = AutoTokenizer.from_pretrained(cfg.base_model_name)\n",
-    "model = AutoModelForSequenceClassification.from_pretrained(cfg.base_model_name, num_labels=2).to(DEVICE)\n",
-    "\n",
-    "\n",
-    "def preprocess_batch(batch: dict, tokenizer: PreTrainedTokenizer, max_length: int = 256) -> BatchEncoding:\n",
-    "    encoded = tokenizer(\n",
-    "        batch[\"text\"],\n",
-    "        truncation=True,\n",
-    "        max_length=max_length,\n",
-    "    )\n",
-    "    encoded[\"labels\"] = batch[\"label\"]\n",
-    "    return encoded\n",
-    "\n",
-    "\n",
-    "tokenize_fn = partial(preprocess_batch, tokenizer=tokenizer, max_length=cfg.max_length)\n",
-    "tokenized_train = train_data.map(tokenize_fn, batched=True, remove_columns=[\"text\", \"label\"])\n",
-    "tokenized_eval = eval_data.map(tokenize_fn, batched=True, remove_columns=[\"text\", \"label\"])\n",
-    "\n",
-    "columns = tokenized_train.column_names\n",
-    "tensor_columns = [name for name in [\"input_ids\", \"attention_mask\", \"token_type_ids\", \"labels\"] if name in columns]\n",
-    "tokenized_train.set_format(type=\"torch\", columns=tensor_columns)\n",
-    "tokenized_eval.set_format(type=\"torch\", columns=tensor_columns)\n",
-    "\n",
-    "metric_accuracy = evaluate.load(\"accuracy\")\n",
-    "metric_f1 = evaluate.load(\"f1\")\n",
-    "\n",
-    "\n",
-    "def compute_metrics(eval_pred: tuple[np.ndarray, np.ndarray]) -> dict[str, float]:\n",
-    "    logits, labels = eval_pred\n",
-    "    if isinstance(logits, tuple):\n",
-    "        logits = logits[0]\n",
-    "    preds = np.argmax(logits, axis=1)\n",
-    "    acc = metric_accuracy.compute(predictions=preds, references=labels)\n",
-    "    f1 = metric_f1.compute(predictions=preds, references=labels)\n",
-    "    return {\"accuracy\": float(acc[\"accuracy\"]), \"f1\": float(f1[\"f1\"])}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "00f52ac8",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Start training V2 model...\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "\n",
-       "    <div>\n",
-       "      \n",
-       "      <progress value='4542' max='4542' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
-       "      [4542/4542 20:20, Epoch 3/3]\n",
-       "    </div>\n",
-       "    <table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       " <tr style=\"text-align: left;\">\n",
-       "      <th>Step</th>\n",
-       "      <th>Training Loss</th>\n",
-       "      <th>Validation Loss</th>\n",
-       "      <th>Accuracy</th>\n",
-       "      <th>F1</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <td>200</td>\n",
-       "      <td>0.666410</td>\n",
-       "      <td>0.350684</td>\n",
-       "      <td>0.834504</td>\n",
-       "      <td>0.855390</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>400</td>\n",
-       "      <td>0.598755</td>\n",
-       "      <td>0.256876</td>\n",
-       "      <td>0.897192</td>\n",
-       "      <td>0.904518</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>600</td>\n",
-       "      <td>0.574993</td>\n",
-       "      <td>0.198666</td>\n",
-       "      <td>0.919258</td>\n",
-       "      <td>0.917138</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>800</td>\n",
-       "      <td>0.560090</td>\n",
-       "      <td>0.555182</td>\n",
-       "      <td>0.849047</td>\n",
-       "      <td>0.868040</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1000</td>\n",
-       "      <td>0.387553</td>\n",
-       "      <td>0.203730</td>\n",
-       "      <td>0.929288</td>\n",
-       "      <td>0.930848</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1200</td>\n",
-       "      <td>0.411762</td>\n",
-       "      <td>0.521041</td>\n",
-       "      <td>0.849047</td>\n",
-       "      <td>0.868387</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1400</td>\n",
-       "      <td>0.386610</td>\n",
-       "      <td>0.348940</td>\n",
-       "      <td>0.902708</td>\n",
-       "      <td>0.910434</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1600</td>\n",
-       "      <td>0.244696</td>\n",
-       "      <td>0.346382</td>\n",
-       "      <td>0.916249</td>\n",
-       "      <td>0.921633</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>1800</td>\n",
-       "      <td>0.223823</td>\n",
-       "      <td>0.308763</td>\n",
-       "      <td>0.924774</td>\n",
-       "      <td>0.928977</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2000</td>\n",
-       "      <td>0.249242</td>\n",
-       "      <td>0.358467</td>\n",
-       "      <td>0.919258</td>\n",
-       "      <td>0.924307</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2200</td>\n",
-       "      <td>0.221226</td>\n",
-       "      <td>0.335397</td>\n",
-       "      <td>0.919759</td>\n",
-       "      <td>0.924599</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2400</td>\n",
-       "      <td>0.221417</td>\n",
-       "      <td>0.587722</td>\n",
-       "      <td>0.882648</td>\n",
-       "      <td>0.894973</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2600</td>\n",
-       "      <td>0.191291</td>\n",
-       "      <td>0.329566</td>\n",
-       "      <td>0.928285</td>\n",
-       "      <td>0.931677</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>2800</td>\n",
-       "      <td>0.219115</td>\n",
-       "      <td>0.368331</td>\n",
-       "      <td>0.919759</td>\n",
-       "      <td>0.925164</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>3000</td>\n",
-       "      <td>0.308968</td>\n",
-       "      <td>0.277328</td>\n",
-       "      <td>0.931795</td>\n",
-       "      <td>0.934928</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>3200</td>\n",
-       "      <td>0.131352</td>\n",
-       "      <td>0.585112</td>\n",
-       "      <td>0.891174</td>\n",
-       "      <td>0.901854</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>3400</td>\n",
-       "      <td>0.152614</td>\n",
-       "      <td>0.388915</td>\n",
-       "      <td>0.924273</td>\n",
-       "      <td>0.929208</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>3600</td>\n",
-       "      <td>0.145248</td>\n",
-       "      <td>0.439313</td>\n",
-       "      <td>0.921765</td>\n",
-       "      <td>0.926898</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>3800</td>\n",
-       "      <td>0.086042</td>\n",
-       "      <td>0.467167</td>\n",
-       "      <td>0.920762</td>\n",
-       "      <td>0.926099</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>4000</td>\n",
-       "      <td>0.051121</td>\n",
-       "      <td>0.561893</td>\n",
-       "      <td>0.909729</td>\n",
-       "      <td>0.916898</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>4200</td>\n",
-       "      <td>0.141769</td>\n",
-       "      <td>0.477382</td>\n",
-       "      <td>0.920762</td>\n",
-       "      <td>0.926168</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>4400</td>\n",
-       "      <td>0.016825</td>\n",
-       "      <td>0.506922</td>\n",
-       "      <td>0.918255</td>\n",
-       "      <td>0.924151</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table><p>"
-      ],
-      "text/plain": [
-       "<IPython.core.display.HTML object>"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.48it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.84it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.64it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.02it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.96it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.07it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.79it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.02it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.03it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.03it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.00it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.10it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.59it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.23it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.16it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.19it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.14it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.14it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.00it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.21it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  5.17it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.99it/s]\n",
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.22it/s]\n",
-      "There were missing keys in the checkpoint model loaded: ['roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias'].\n",
-      "There were unexpected keys in the checkpoint model loaded: ['roberta.embeddings.LayerNorm.beta', 'roberta.embeddings.LayerNorm.gamma', 'roberta.encoder.layer.0.attention.output.LayerNorm.beta', 'roberta.encoder.layer.0.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.0.output.LayerNorm.beta', 'roberta.encoder.layer.0.output.LayerNorm.gamma', 'roberta.encoder.layer.1.attention.output.LayerNorm.beta', 'roberta.encoder.layer.1.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.1.output.LayerNorm.beta', 'roberta.encoder.layer.1.output.LayerNorm.gamma', 'roberta.encoder.layer.2.attention.output.LayerNorm.beta', 'roberta.encoder.layer.2.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.2.output.LayerNorm.beta', 'roberta.encoder.layer.2.output.LayerNorm.gamma', 'roberta.encoder.layer.3.attention.output.LayerNorm.beta', 'roberta.encoder.layer.3.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.3.output.LayerNorm.beta', 'roberta.encoder.layer.3.output.LayerNorm.gamma', 'roberta.encoder.layer.4.attention.output.LayerNorm.beta', 'roberta.encoder.layer.4.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.4.output.LayerNorm.beta', 'roberta.encoder.layer.4.output.LayerNorm.gamma', 'roberta.encoder.layer.5.attention.output.LayerNorm.beta', 'roberta.encoder.layer.5.attention.output.LayerNorm.gamma', 'roberta.encoder.layer.5.output.LayerNorm.beta', 'roberta.encoder.layer.5.output.LayerNorm.gamma'].\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Final evaluation...\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "\n",
-       "    <div>\n",
-       "      \n",
-       "      <progress value='250' max='250' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
-       "      [250/250 00:07]\n",
-       "    </div>\n",
-       "    "
-      ],
-      "text/plain": [
-       "<IPython.core.display.HTML object>"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "ename": "RuntimeError",
-     "evalue": "on_train_begin must be called before on_evaluate",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
-      "\u001b[31mRuntimeError\u001b[39m                              Traceback (most recent call last)",
-      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[8]\u001b[39m\u001b[32m, line 55\u001b[39m\n\u001b[32m     52\u001b[39m trainer.train()\n\u001b[32m     54\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33m\"\u001b[39m\u001b[33mFinal evaluation...\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m---> \u001b[39m\u001b[32m55\u001b[39m eval_result = \u001b[43mtrainer\u001b[49m\u001b[43m.\u001b[49m\u001b[43mevaluate\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     56\u001b[39m \u001b[38;5;28mprint\u001b[39m(json.dumps(eval_result, indent=\u001b[32m2\u001b[39m, default=\u001b[38;5;28mstr\u001b[39m))\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/trainer.py:2602\u001b[39m, in \u001b[36mTrainer.evaluate\u001b[39m\u001b[34m(self, eval_dataset, ignore_keys, metric_key_prefix)\u001b[39m\n\u001b[32m   2599\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m DebugOption.TPU_METRICS_DEBUG \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.args.debug:\n\u001b[32m   2600\u001b[39m     xm.master_print(met.metrics_report())\n\u001b[32m-> \u001b[39m\u001b[32m2602\u001b[39m \u001b[38;5;28mself\u001b[39m.control = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mcallback_handler\u001b[49m\u001b[43m.\u001b[49m\u001b[43mon_evaluate\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mstate\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mcontrol\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput\u001b[49m\u001b[43m.\u001b[49m\u001b[43mmetrics\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   2604\u001b[39m \u001b[38;5;28mself\u001b[39m._memory_tracker.stop_and_update_metrics(output.metrics)\n\u001b[32m   2606\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m output.metrics\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/trainer_callback.py:524\u001b[39m, in \u001b[36mCallbackHandler.on_evaluate\u001b[39m\u001b[34m(self, args, state, control, metrics)\u001b[39m\n\u001b[32m    522\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mon_evaluate\u001b[39m(\u001b[38;5;28mself\u001b[39m, args: TrainingArguments, state: TrainerState, control: TrainerControl, metrics):\n\u001b[32m    523\u001b[39m     control.should_evaluate = \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m524\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mcall_event\u001b[49m\u001b[43m(\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mon_evaluate\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstate\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcontrol\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmetrics\u001b[49m\u001b[43m=\u001b[49m\u001b[43mmetrics\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/trainer_callback.py:545\u001b[39m, in \u001b[36mCallbackHandler.call_event\u001b[39m\u001b[34m(self, event, args, state, control, **kwargs)\u001b[39m\n\u001b[32m    543\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mcall_event\u001b[39m(\u001b[38;5;28mself\u001b[39m, event, args, state, control, **kwargs):\n\u001b[32m    544\u001b[39m     \u001b[38;5;28;01mfor\u001b[39;00m callback \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.callbacks:\n\u001b[32m--> \u001b[39m\u001b[32m545\u001b[39m         result = \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mcallback\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mevent\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    546\u001b[39m \u001b[43m            \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    547\u001b[39m \u001b[43m            \u001b[49m\u001b[43mstate\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    548\u001b[39m \u001b[43m            \u001b[49m\u001b[43mcontrol\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    549\u001b[39m \u001b[43m            \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    550\u001b[39m \u001b[43m            \u001b[49m\u001b[43mprocessing_class\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mprocessing_class\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    551\u001b[39m \u001b[43m            \u001b[49m\u001b[43moptimizer\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43moptimizer\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    552\u001b[39m \u001b[43m            \u001b[49m\u001b[43mlr_scheduler\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mlr_scheduler\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    553\u001b[39m \u001b[43m            \u001b[49m\u001b[43mtrain_dataloader\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mtrain_dataloader\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    554\u001b[39m \u001b[43m            \u001b[49m\u001b[43meval_dataloader\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43meval_dataloader\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    555\u001b[39m \u001b[43m            \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    556\u001b[39m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    557\u001b[39m         \u001b[38;5;66;03m# A Callback can skip the return of `control` if it doesn't change it.\u001b[39;00m\n\u001b[32m    558\u001b[39m         \u001b[38;5;28;01mif\u001b[39;00m result \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/utils/notebook.py:354\u001b[39m, in \u001b[36mNotebookProgressCallback.on_evaluate\u001b[39m\u001b[34m(self, args, state, control, metrics, **kwargs)\u001b[39m\n\u001b[32m    353\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mon_evaluate\u001b[39m(\u001b[38;5;28mself\u001b[39m, args, state, control, metrics=\u001b[38;5;28;01mNone\u001b[39;00m, **kwargs):\n\u001b[32m--> \u001b[39m\u001b[32m354\u001b[39m     tt = \u001b[43m_require\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mtraining_tracker\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mon_train_begin must be called before on_evaluate\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[32m    356\u001b[39m     values = {\u001b[33m\"\u001b[39m\u001b[33mTraining Loss\u001b[39m\u001b[33m\"\u001b[39m: \u001b[33m\"\u001b[39m\u001b[33mNo log\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mValidation Loss\u001b[39m\u001b[33m\"\u001b[39m: \u001b[33m\"\u001b[39m\u001b[33mNo log\u001b[39m\u001b[33m\"\u001b[39m}\n\u001b[32m    357\u001b[39m     \u001b[38;5;28;01mfor\u001b[39;00m log \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mreversed\u001b[39m(state.log_history):\n",
-      "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/envs/ml/lib/python3.11/site-packages/transformers/utils/notebook.py:31\u001b[39m, in \u001b[36m_require\u001b[39m\u001b[34m(x, msg)\u001b[39m\n\u001b[32m     29\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m_require\u001b[39m(x: _T | \u001b[38;5;28;01mNone\u001b[39;00m, msg: \u001b[38;5;28mstr\u001b[39m) -> _T:\n\u001b[32m     30\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m x \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m---> \u001b[39m\u001b[32m31\u001b[39m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(msg)\n\u001b[32m     32\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m x\n",
-      "\u001b[31mRuntimeError\u001b[39m: on_train_begin must be called before on_evaluate"
-     ]
-    }
-   ],
-   "source": [
-    "import sys\n",
-    "import subprocess\n",
-    "\n",
-    "\n",
-    "def _ensure_accelerate(min_version: str = \"1.1.0\") -> None:\n",
-    "    try:\n",
-    "        import accelerate  # noqa: F401\n",
-    "        from packaging import version\n",
-    "\n",
-    "        if version.parse(accelerate.__version__) < version.parse(min_version):\n",
-    "            raise ImportError(f\"accelerate version too old: {accelerate.__version__}\")\n",
-    "    except Exception:\n",
-    "        print(\"Installing/upgrading accelerate in current kernel environment...\")\n",
-    "        subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", f\"accelerate>={min_version}\"])\n",
-    "\n",
-    "\n",
-    "_ensure_accelerate()\n",
-    "\n",
-    "train_args = TrainingArguments(\n",
-    "    output_dir=\"./results/v2-distilroberta\",\n",
-    "    num_train_epochs=3,\n",
-    "    learning_rate=2e-5,\n",
-    "    weight_decay=0.01,\n",
-    "    per_device_train_batch_size=8,\n",
-    "    per_device_eval_batch_size=8,\n",
-    "    gradient_accumulation_steps=2,\n",
-    "    fp16=torch.cuda.is_available(),\n",
-    "    eval_strategy=\"steps\",\n",
-    "    eval_steps=200,\n",
-    "    save_strategy=\"steps\",\n",
-    "    save_steps=200,\n",
-    "    save_total_limit=2,\n",
-    "    logging_steps=50,\n",
-    "    metric_for_best_model=\"eval_f1\",\n",
-    "    load_best_model_at_end=True,\n",
-    "    remove_unused_columns=False,\n",
-    "    report_to=\"none\",\n",
-    ")\n",
-    "\n",
-    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
-    "\n",
-    "trainer = Trainer(\n",
-    "    model=model,\n",
-    "    args=train_args,\n",
-    "    train_dataset=tokenized_train,\n",
-    "    eval_dataset=tokenized_eval,\n",
-    "    data_collator=data_collator,\n",
-    "    compute_metrics=compute_metrics,\n",
-    ")\n",
-    "\n",
-    "print(\"Start training V2 model...\")\n",
-    "train_result = trainer.train()\n",
-    "\n",
-    "print(\"\\n✓ Training complete!\")\n",
-    "print(f\"Final training metrics:\")\n",
-    "if hasattr(trainer.state, 'log_history') and trainer.state.log_history:\n",
-    "    # Get the last evaluation metrics from log history\n",
-    "    for log_entry in reversed(trainer.state.log_history):\n",
-    "        if 'eval_loss' in log_entry:\n",
-    "            print(f\"  Eval Loss: {log_entry.get('eval_loss', 'N/A'):.4f}\")\n",
-    "            print(f\"  Eval Accuracy: {log_entry.get('eval_accuracy', 'N/A'):.4f}\")\n",
-    "            print(f\"  Eval F1: {log_entry.get('eval_f1', 'N/A'):.4f}\")\n",
-    "            break"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "1b601515",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.29it/s]"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Saved V2 model to: /mnt/linux-data/Work/aiapi/notebook/ai_vs_human/v2_model\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "save_dir = Path(cfg.output_dir)\n",
-    "save_dir.mkdir(parents=True, exist_ok=True)\n",
-    "trainer.save_model(str(save_dir))\n",
-    "tokenizer.save_pretrained(str(save_dir))\n",
-    "\n",
-    "label_map = {\"0\": \"human\", \"1\": \"ai\"}\n",
-    "(save_dir / \"label_map.json\").write_text(json.dumps(label_map, indent=2), encoding=\"utf-8\")\n",
-    "\n",
-    "print(f\"Saved V2 model to: {save_dir.resolve()}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "93f0e5a0",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "================================================================================\n",
-      "COMPREHENSIVE TEST: All Sentence Types\n",
-      "================================================================================\n",
-      "\n",
-      "1. VERY SHORT SENTENCES (< 10 words):\n",
-      "  [2 words] human: Hello world.\n",
-      "  [3 words] human: AI is powerful.\n",
-      "  [3 words] human: I like coding.\n",
-      "  [4 words] human: Machine learning works well.\n",
-      "\n",
-      "2. SHORT SENTENCES (10-50 words):\n",
-      "  [10 words] human: AI writes fast, but humans add personal experience and emoti...\n",
-      "  [14 words] human: I woke up late, missed the bus, and ran all the way to class...\n",
-      "  [11 words] human: This response was generated by a language model in one pass....\n",
-      "  [17 words] human: The field of data science combines statistics, programming, ...\n",
-      "\n",
-      "3. MEDIUM SENTENCES (50-150 words):\n",
-      "  [74 words] human: Artificial intelligence systems can process massive amounts ...\n",
-      "  [87 words] human: I once tried to learn guitar in a single weekend because I t...\n",
-      "\n",
-      "4. LONG SENTENCES (150+ words):\n",
-      "  [153 words] ai: Machine learning represents a subset of artificial intellige...\n",
-      "\n",
-      "5. EDGE CASES:\n",
-      "  [1 words] human: 'A'\n",
-      "  [4 words] human: 'This is a test.'\n",
-      "  [4 words] human: 'Multiple   spaces   between   words'\n",
-      "\n",
-      "================================================================================\n",
-      "✓ All sentence types tested successfully!\n",
-      "================================================================================\n"
-     ]
-    }
-   ],
-   "source": [
-    "def predict_v2(text: str) -> dict[str, float | int | str]:\n",
-    "    \"\"\"Predict whether text is AI or human-written. Works for all sentence lengths.\"\"\"\n",
-    "    cleaned = normalize_text(text)\n",
-    "    if not cleaned:\n",
-    "        raise ValueError(\"Input text is empty.\")\n",
-    "\n",
-    "    inputs = tokenizer(\n",
-    "        cleaned,\n",
-    "        truncation=True,\n",
-    "        max_length=cfg.max_length,\n",
-    "        return_tensors=\"pt\",\n",
-    "    ).to(model.device)\n",
-    "\n",
-    "    model.eval()\n",
-    "    with torch.no_grad():\n",
-    "        logits = model(**inputs).logits\n",
-    "        probs = torch.softmax(logits, dim=-1).cpu().numpy()[0]\n",
-    "\n",
-    "    pred = int(np.argmax(probs))\n",
-    "    wc = count_words(cleaned)\n",
-    "\n",
-    "    return {\n",
-    "        \"text\": cleaned,\n",
-    "        \"word_count\": wc,\n",
-    "        \"predicted_label\": pred,\n",
-    "        \"predicted_name\": \"ai\" if pred == 1 else \"human\",\n",
-    "        \"probability_human\": float(probs[0]),\n",
-    "        \"probability_ai\": float(probs[1]),\n",
-    "        \"short_text\": wc < 50,\n",
-    "    }\n",
-    "\n",
-    "\n",
-    "print(\"=\" * 80)\n",
-    "print(\"COMPREHENSIVE TEST: All Sentence Types\")\n",
-    "print(\"=\" * 80)\n",
-    "\n",
-    "# Test 1: Very short sentences (under 10 words)\n",
-    "print(\"\\n1. VERY SHORT SENTENCES (< 10 words):\")\n",
-    "very_short = [\n",
-    "    \"Hello world.\",\n",
-    "    \"AI is powerful.\",\n",
-    "    \"I like coding.\",\n",
-    "    \"Machine learning works well.\",\n",
-    "]\n",
-    "for text in very_short:\n",
-    "    result = predict_v2(text)\n",
-    "    print(f\"  [{result['word_count']} words] {result['predicted_name']}: {text[:60]}\")\n",
-    "\n",
-    "# Test 2: Short sentences (10-50 words)\n",
-    "print(\"\\n2. SHORT SENTENCES (10-50 words):\")\n",
-    "short_examples = [\n",
-    "    \"AI writes fast, but humans add personal experience and emotion.\",\n",
-    "    \"I woke up late, missed the bus, and ran all the way to class.\",\n",
-    "    \"This response was generated by a language model in one pass.\",\n",
-    "    \"The field of data science combines statistics, programming, and domain knowledge to extract meaningful insights from data.\",\n",
-    "]\n",
-    "for text in short_examples:\n",
-    "    result = predict_v2(text)\n",
-    "    print(f\"  [{result['word_count']} words] {result['predicted_name']}: {text[:60]}...\")\n",
-    "\n",
-    "# Test 3: Medium sentences (50-150 words)\n",
-    "print(\"\\n3. MEDIUM SENTENCES (50-150 words):\")\n",
-    "medium_examples = [\n",
-    "    \"Artificial intelligence systems can process massive amounts of data extremely quickly compared to humans. They are designed to analyze large datasets, identify patterns, and extract useful insights within seconds or minutes. Using advanced algorithms and machine learning models, AI systems can examine structured and unstructured data such as text, images, audio, and numerical information. By learning from historical data, these systems can recognize complex relationships between variables and make accurate predictions about future outcomes.\",\n",
-    "    \"I once tried to learn guitar in a single weekend because I thought it would be easy. Turns out my fingers had other plans. After two hours of awkward chords and random noises, I realized that music requires patience, practice, and a lot more discipline than I originally expected. My friends laughed when they heard me trying to play, but I kept practicing anyway because I genuinely wanted to improve. Eventually, after weeks of consistent effort, I could finally play a simple song from start to finish.\",\n",
-    "]\n",
-    "for text in medium_examples:\n",
-    "    result = predict_v2(text)\n",
-    "    print(f\"  [{result['word_count']} words] {result['predicted_name']}: {text[:60]}...\")\n",
-    "\n",
-    "# Test 4: Long sentences (150+ words)\n",
-    "print(\"\\n4. LONG SENTENCES (150+ words):\")\n",
-    "long_examples = [\n",
-    "    \"Machine learning represents a subset of artificial intelligence that enables computer systems to automatically learn and improve from experience without being explicitly programmed for every single task. The fundamental idea behind machine learning is to develop algorithms that can receive input data and use statistical analysis to predict an output while updating outputs as new data becomes available. This field has grown exponentially over the past few decades, driven by increases in computational power, the availability of large datasets, and breakthroughs in algorithmic approaches. Modern machine learning systems power everything from recommendation engines on streaming platforms to autonomous vehicles, medical diagnosis tools, and natural language processing applications. The three main categories of machine learning include supervised learning, where models are trained on labeled data; unsupervised learning, where patterns are discovered in unlabeled data; and reinforcement learning, where agents learn to make decisions by receiving rewards or penalties for their actions in an environment.\",\n",
-    "]\n",
-    "for text in long_examples:\n",
-    "    result = predict_v2(text)\n",
-    "    print(f\"  [{result['word_count']} words] {result['predicted_name']}: {text[:60]}...\")\n",
-    "\n",
-    "# Test 5: Edge cases\n",
-    "print(\"\\n5. EDGE CASES:\")\n",
-    "edge_cases = [\n",
-    "    \"A\",  # Single word\n",
-    "    \"This is a test.\",  # Very basic\n",
-    "    \"   Multiple   spaces   between   words   \",  # Extra whitespace\n",
-    "]\n",
-    "for text in edge_cases:\n",
-    "    try:\n",
-    "        result = predict_v2(text)\n",
-    "        print(f\"  [{result['word_count']} words] {result['predicted_name']}: '{text.strip()}'\")\n",
-    "    except Exception as e:\n",
-    "        print(f\"  ERROR: {text.strip()[:30]} - {str(e)}\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\" * 80)\n",
-    "print(\"✓ All sentence types tested successfully!\")\n",
-    "print(\"=\" * 80)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "98ef7c7d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "================================================================================\n",
-      "TESTING SAVED V2 MODEL FROM DISK\n",
-      "================================================================================\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Loading weights: 100%|██████████| 105/105 [00:00<00:00, 8556.64it/s]"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "✓ Loaded model from: v2_model\n",
-      "\n",
-      "Running inference tests:\n",
-      "  [very short  ] human  (AI: 0.50%): Hi there!\n",
-      "  [short       ] human  (AI: 0.09%): I love programming and building cool projects.\n",
-      "  [medium      ] human  (AI: 3.09%): Artificial intelligence has revolutionized many in\n",
-      "\n",
-      "✓ Saved model works correctly for all sentence types!\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Load and test the saved v2_model independently\n",
-    "print(\"=\" * 80)\n",
-    "print(\"TESTING SAVED V2 MODEL FROM DISK\")\n",
-    "print(\"=\" * 80)\n",
-    "\n",
-    "saved_model_path = Path(cfg.output_dir)\n",
-    "if saved_model_path.exists():\n",
-    "    # Load fresh model and tokenizer from saved checkpoint\n",
-    "    saved_tokenizer = AutoTokenizer.from_pretrained(str(saved_model_path))\n",
-    "    saved_model = AutoModelForSequenceClassification.from_pretrained(str(saved_model_path)).to(DEVICE)\n",
-    "    \n",
-    "    print(f\"\\n✓ Loaded model from: {saved_model_path}\")\n",
-    "    \n",
-    "    # Test with diverse examples\n",
-    "    test_cases = [\n",
-    "        (\"Hi there!\", \"very short\"),\n",
-    "        (\"I love programming and building cool projects.\", \"short\"),\n",
-    "        (\"Artificial intelligence has revolutionized many industries by enabling automation, improving decision-making, and creating new opportunities for innovation.\", \"medium\"),\n",
-    "    ]\n",
-    "    \n",
-    "    print(\"\\nRunning inference tests:\")\n",
-    "    for text, category in test_cases:\n",
-    "        inputs = saved_tokenizer(text, truncation=True, max_length=256, return_tensors=\"pt\").to(DEVICE)\n",
-    "        saved_model.eval()\n",
-    "        with torch.no_grad():\n",
-    "            logits = saved_model(**inputs).logits\n",
-    "            probs = torch.softmax(logits, dim=-1).cpu().numpy()[0]\n",
-    "            pred_label = int(np.argmax(probs))\n",
-    "            pred_name = \"ai\" if pred_label == 1 else \"human\"\n",
-    "        \n",
-    "        print(f\"  [{category:12}] {pred_name:6} (AI: {probs[1]:.2%}): {text[:50]}\")\n",
-    "    \n",
-    "    print(\"\\n✓ Saved model works correctly for all sentence types!\")\n",
-    "else:\n",
-    "    print(f\"⚠ Model not found at: {saved_model_path}\")\n",
-    "    print(\"  Run the save cell first to create v2_model/\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "2f63e591",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "================================================================================\n",
-      "EXTREME EDGE CASE TESTING\n",
-      "================================================================================\n",
-      "\n",
-      "Testing extreme edge cases:\n",
-      "  ✓ Single character     [ 1w] human  (99.3%): 'A'\n",
-      "  ✓ Single word          [ 1w] human  (99.4%): 'Hello'\n",
-      "  ✓ Two words            [ 2w] human  (99.6%): 'Hello world'\n",
-      "  ✓ Numbers only         [ 3w] human  (98.7%): '123 456 789'\n",
-      "  ✓ Special chars        [ 4w] human  (99.8%): '!!! ### $$$ ???'\n",
-      "  ✓ Mixed alphanumeric   [ 3w] human  (99.3%): 'Test123 ABC456 xyz789'\n",
-      "  ✓ Very long word       [ 1w] human  (99.1%): 'supercalifragilisticexpialidocious'\n",
-      "  ✓ Repeated words       [ 5w] human  (99.6%): 'test test test test test'\n",
-      "  ✓ Newlines             [ 6w] human  (99.4%): 'Line one\\nLine two\\nLine three'\n",
-      "  ✓ Tabs                 [ 3w] human  (99.5%): 'Col1\\tCol2\\tCol3'\n",
-      "  ✓ Multiple spaces      [ 3w] human  (99.7%): 'Too     many      spaces'\n",
-      "  ✓ Punctuation heavy    [ 5w] human  (99.8%): 'Wow! Really? Yes! No... Maybe?'\n",
-      "  ✗ Empty-like           ERROR: Input text is empty.\n",
-      "  ✓ Mixed case           [ 5w] human  (99.3%): 'ThIs Is MiXeD cAsE tExT'\n",
-      "  ✓ All caps             [ 4w] human  (99.3%): 'THIS IS ALL CAPITALS'\n",
-      "  ✓ All lower            [ 4w] human  (99.9%): 'this is all lowercase'\n",
-      "\n",
-      "Result: 15 passed, 1 failed\n",
-      "\n",
-      "================================================================================\n",
-      "BATCH PREDICTION TEST\n",
-      "================================================================================\n",
-      "\n",
-      "Predicting batch of mixed-length sentences:\n",
-      "\n",
-      "  Sentence 1 (1 words):\n",
-      "    Text: Short....\n",
-      "    Prediction: human\n",
-      "    Confidence: AI=0.1%, Human=99.9%\n",
-      "\n",
-      "  Sentence 2 (9 words):\n",
-      "    Text: This is a medium length sentence with some content....\n",
-      "    Prediction: human\n",
-      "    Confidence: AI=0.1%, Human=99.9%\n",
-      "\n",
-      "  Sentence 3 (29 words):\n",
-      "    Text: This is a longer sentence that contains more words and provi...\n",
-      "    Prediction: human\n",
-      "    Confidence: AI=0.1%, Human=99.9%\n",
-      "\n",
-      "================================================================================\n",
-      "✓ ALL EDGE CASES AND BATCH TESTS COMPLETE!\n",
-      "================================================================================\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"=\" * 80)\n",
-    "print(\"EXTREME EDGE CASE TESTING\")\n",
-    "print(\"=\" * 80)\n",
-    "\n",
-    "# Test various edge cases that might break the model\n",
-    "edge_test_cases = {\n",
-    "    \"Single character\": \"A\",\n",
-    "    \"Single word\": \"Hello\",\n",
-    "    \"Two words\": \"Hello world\",\n",
-    "    \"Numbers only\": \"123 456 789\",\n",
-    "    \"Special chars\": \"!!! ### $$$ ???\",\n",
-    "    \"Mixed alphanumeric\": \"Test123 ABC456 xyz789\",\n",
-    "    \"Very long word\": \"supercalifragilisticexpialidocious\",\n",
-    "    \"Repeated words\": \"test test test test test\",\n",
-    "    \"Newlines\": \"Line one\\nLine two\\nLine three\",\n",
-    "    \"Tabs\": \"Col1\\tCol2\\tCol3\",\n",
-    "    \"Multiple spaces\": \"Too     many      spaces\",\n",
-    "    \"Punctuation heavy\": \"Wow! Really? Yes! No... Maybe?\",\n",
-    "    \"Empty-like\": \"   \",\n",
-    "    \"Mixed case\": \"ThIs Is MiXeD cAsE tExT\",\n",
-    "    \"All caps\": \"THIS IS ALL CAPITALS\",\n",
-    "    \"All lower\": \"this is all lowercase\",\n",
-    "}\n",
-    "\n",
-    "print(\"\\nTesting extreme edge cases:\")\n",
-    "passed = 0\n",
-    "failed = 0\n",
-    "\n",
-    "for case_name, text in edge_test_cases.items():\n",
-    "    try:\n",
-    "        result = predict_v2(text)\n",
-    "        wc = result['word_count']\n",
-    "        pred = result['predicted_name']\n",
-    "        conf = result['probability_ai'] if pred == 'ai' else result['probability_human']\n",
-    "        \n",
-    "        # Handle display of text with special characters\n",
-    "        display_text = text.replace('\\n', '\\\\n').replace('\\t', '\\\\t')[:40]\n",
-    "        print(f\"  ✓ {case_name:20} [{wc:2}w] {pred:6} ({conf:.1%}): '{display_text}'\")\n",
-    "        passed += 1\n",
-    "    except Exception as e:\n",
-    "        print(f\"  ✗ {case_name:20} ERROR: {str(e)[:50]}\")\n",
-    "        failed += 1\n",
-    "\n",
-    "print(f\"\\nResult: {passed} passed, {failed} failed\")\n",
-    "\n",
-    "# Batch prediction test\n",
-    "print(\"\\n\" + \"=\" * 80)\n",
-    "print(\"BATCH PREDICTION TEST\")\n",
-    "print(\"=\" * 80)\n",
-    "\n",
-    "batch_texts = [\n",
-    "    \"Short.\",\n",
-    "    \"This is a medium length sentence with some content.\",\n",
-    "    \"This is a longer sentence that contains more words and provides more context for the model to analyze and make predictions based on the patterns it learned during training.\",\n",
-    "]\n",
-    "\n",
-    "print(\"\\nPredicting batch of mixed-length sentences:\")\n",
-    "batch_results = [predict_v2(text) for text in batch_texts]\n",
-    "\n",
-    "for i, (text, result) in enumerate(zip(batch_texts, batch_results), 1):\n",
-    "    print(f\"\\n  Sentence {i} ({result['word_count']} words):\")\n",
-    "    print(f\"    Text: {text[:60]}...\")\n",
-    "    print(f\"    Prediction: {result['predicted_name']}\")\n",
-    "    print(f\"    Confidence: AI={result['probability_ai']:.1%}, Human={result['probability_human']:.1%}\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\" * 80)\n",
-    "print(\"✓ ALL EDGE CASES AND BATCH TESTS COMPLETE!\")\n",
-    "print(\"=\" * 80)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "ml",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.14"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

notebook/ai_vs_human_nepali/notebook/Nepali_Ai_vs_Human.ipynb

@@ -1,1429 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "code",
-      "execution_count": 1,
-      "id": "901fc22d",
-      "metadata": {
-        "id": "901fc22d"
-      },
-      "outputs": [
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/home/pujan/miniconda3/envs/ml/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-            "  from .autonotebook import tqdm as notebook_tqdm\n"
-          ]
-        }
-      ],
-      "source": [
-        "import os\n",
-        "os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'\n",
-        "\n",
-        "import math\n",
-        "import pandas as pd\n",
-        "import torch\n",
-        "from torch.utils.data import Dataset, DataLoader\n",
-        "from transformers import AutoTokenizer, AutoModel, get_linear_schedule_with_warmup\n",
-        "from sklearn.model_selection import train_test_split\n",
-        "from sklearn.metrics import classification_report, f1_score, accuracy_score\n",
-        "import torch.nn as nn\n",
-        "from torch.optim import AdamW"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 2,
-      "id": "70d3c048",
-      "metadata": {},
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Columns: ['human_text', 'ai_generated_text']\n",
-            "Prepared dataset shape: (1986, 2)\n",
-            "label\n",
-            "1    996\n",
-            "0    990\n",
-            "Name: count, dtype: int64\n"
-          ]
-        },
-        {
-          "data": {
-            "text/html": [
-              "<div>\n",
-              "<style scoped>\n",
-              "    .dataframe tbody tr th:only-of-type {\n",
-              "        vertical-align: middle;\n",
-              "    }\n",
-              "\n",
-              "    .dataframe tbody tr th {\n",
-              "        vertical-align: top;\n",
-              "    }\n",
-              "\n",
-              "    .dataframe thead th {\n",
-              "        text-align: right;\n",
-              "    }\n",
-              "</style>\n",
-              "<table border=\"1\" class=\"dataframe\">\n",
-              "  <thead>\n",
-              "    <tr style=\"text-align: right;\">\n",
-              "      <th></th>\n",
-              "      <th>text</th>\n",
-              "      <th>label</th>\n",
-              "    </tr>\n",
-              "  </thead>\n",
-              "  <tbody>\n",
-              "    <tr>\n",
-              "      <th>0</th>\n",
-              "      <td>हामीले पार्टी एकतापछि कि दुबै पार्टीको सिद्धान...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>1</th>\n",
-              "      <td>एमाले प्रतिनिधिसभाको प्रत्यक्षतर्फ ८० समानुपात...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>2</th>\n",
-              "      <td>नेकपा माओवादी केन्द्रका नेता रामनारायण विडारील...</td>\n",
-              "      <td>1</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>3</th>\n",
-              "      <td>प्रदेश नं २ का मुख्यमन्त्रीको रूपमा संघीय समाज...</td>\n",
-              "      <td>1</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>4</th>\n",
-              "      <td>बिहीबार एमालेका अध्यक्ष केपी शर्मा ओली र माओवा...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "  </tbody>\n",
-              "</table>\n",
-              "</div>"
-            ],
-            "text/plain": [
-              "                                                text  label\n",
-              "0  हामीले पार्टी एकतापछि कि दुबै पार्टीको सिद्धान...      0\n",
-              "1  एमाले प्रतिनिधिसभाको प्रत्यक्षतर्फ ८० समानुपात...      0\n",
-              "2  नेकपा माओवादी केन्द्रका नेता रामनारायण विडारील...      1\n",
-              "3  प्रदेश नं २ का मुख्यमन्त्रीको रूपमा संघीय समाज...      1\n",
-              "4  बिहीबार एमालेका अध्यक्�� केपी शर्मा ओली र माओवा...      0"
-            ]
-          },
-          "execution_count": 2,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "# Load Dataset and convert to binary classification format\n",
-        "DATA_PATH = '../DATASET/new_data.csv'\n",
-        "raw_df = pd.read_csv(DATA_PATH)\n",
-        "print('Columns:', raw_df.columns.tolist())\n",
-        "\n",
-        "required_cols = ['human_text', 'ai_generated_text']\n",
-        "missing = [c for c in required_cols if c not in raw_df.columns]\n",
-        "if missing:\n",
-        "    raise ValueError(f'Missing required columns: {missing}')\n",
-        "\n",
-        "# Build unified training dataframe: text + label (0=Human, 1=AI)\n",
-        "df_human = raw_df[['human_text']].dropna().rename(columns={'human_text': 'text'})\n",
-        "df_human['label'] = 0\n",
-        "\n",
-        "df_ai = raw_df[['ai_generated_text']].dropna().rename(columns={'ai_generated_text': 'text'})\n",
-        "df_ai['label'] = 1\n",
-        "\n",
-        "df = pd.concat([df_human, df_ai], ignore_index=True)\n",
-        "df['text'] = df['text'].astype(str).str.strip()\n",
-        "df = df[df['text'].str.len() > 10].drop_duplicates(subset=['text']).sample(frac=1, random_state=42).reset_index(drop=True)\n",
-        "\n",
-        "print('Prepared dataset shape:', df.shape)\n",
-        "print(df['label'].value_counts())\n",
-        "df.head()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 3,
-      "id": "f93d4c7a",
-      "metadata": {
-        "id": "f93d4c7a"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Nulls in text: 0\n",
-            "Nulls in label: 0\n",
-            "Example text sample:\n",
-            "हामीले पार्टी एकतापछि कि दुबै पार्टीको सिद्धान्त राख्ने कि राख्ने माओवाद र जबज दुबै नराख्ने भन्दा उहाँहरु मान्नु भएन । एमालेका साथीहरुले जवजको विषय उठाउन चाहनुभएन । सिद्धान्तको विषय नै नमिलेपछि पार्टी एकता संयोजन समितिको बैठक रोकियो कार्यदलका एक सदस्\n"
-          ]
-        }
-      ],
-      "source": [
-        "# Quick sanity checks\n",
-        "print('Nulls in text:', int(df['text'].isnull().sum()))\n",
-        "print('Nulls in label:', int(df['label'].isnull().sum()))\n",
-        "print('Example text sample:')\n",
-        "print(df.loc[0, 'text'][:250])"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 4,
-      "id": "ba4a933f",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 206
-        },
-        "id": "ba4a933f",
-        "outputId": "9bf5f0a5-c547-43f1-b8f2-a580024d74a9"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "label\n",
-            "AI       0.501511\n",
-            "Human    0.498489\n",
-            "Name: proportion, dtype: float64\n"
-          ]
-        },
-        {
-          "data": {
-            "text/plain": [
-              "label       \n",
-              "0      count    990.000000\n",
-              "       mean     455.551515\n",
-              "       std       56.825837\n",
-              "       min      299.000000\n",
-              "       25%      418.000000\n",
-              "       50%      458.000000\n",
-              "       75%      494.000000\n",
-              "       max      629.000000\n",
-              "1      count    996.000000\n",
-              "       mean     284.231928\n",
-              "       std       67.165254\n",
-              "       min      103.000000\n",
-              "       25%      238.000000\n",
-              "       50%      282.000000\n",
-              "       75%      331.000000\n",
-              "       max      433.000000\n",
-              "Name: text, dtype: float64"
-            ]
-          },
-          "execution_count": 4,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "# Class balance\n",
-        "print(df['label'].value_counts(normalize=True).rename({0: 'Human', 1: 'AI'}))\n",
-        "df.groupby('label')['text'].apply(lambda s: s.str.len().describe())"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 5,
-      "id": "d7b48175",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 206
-        },
-        "id": "d7b48175",
-        "outputId": "08bc4562-874c-40c1-d554-1d809a6d0e31"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "<matplotlib.legend.Legend at 0x7fef748b5290>"
-            ]
-          },
-          "execution_count": 5,
-          "metadata": {},
-          "output_type": "execute_result"
-        },
-        {
-          "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvwAAAGHCAYAAADMVYYQAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAARoNJREFUeJzt3X1cVGX+//H3gMMIAt7HTaJiIeVduVomVmgFu5aurtXWWq1Wa5bdaO5+7WtWDrsFZuVSa9nPttS2yG03c7Wvd6SJlbmp5epqmbspmkpkoaAoDMz1+6OYHAEdYGCY4+v5eMxDz3Wuuc7nzGdGPlxec47NGGMEAAAAwJJCAh0AAAAAgMZDwQ8AAABYGAU/AAAAYGEU/AAAAICFUfADAAAAFkbBDwAAAFgYBT8AAABgYRT8AAAAgIVR8AMAAAAWRsEPIOjZbDafHmvXrvXL8Q4cOCCn06ktW7b41H/t2rWy2Wz6+9//7pfj+1tpaamcTmeNr4/T6ZTNZtOhQ4fqNfbYsWO9ctCqVSt17dpVP//5zzVv3jyVlZVVe87gwYM1ePDgOh1nx44dcjqd2rNnT52ed+qx9uzZI5vNpqeffrpO45xJZmamFi9eXK296r3hr/cmANSkRaADAICG+uijj7y2//CHP+i9997TmjVrvNp79Ojhl+MdOHBAGRkZ6tq1qy6++GK/jBlIpaWlysjIkKQ6F9q+CA8P9+Ti+PHj2rdvn5YvX65x48bpmWee0YoVK9SpUydP/xdeeKHOx9ixY4cyMjI0ePBgde3a1efn1edY9ZGZmakbbrhBI0eO9Gr/yU9+oo8++shv700AqAkFP4Cgd9lll3ltd+zYUSEhIdXaERg15eLXv/61br/9dg0bNkw33HCDNmzY4NnXFMVvaWmpIiIiAl5oR0dH8z4F0OhY0gPgrFBeXq7HH39cF1xwgRwOhzp27Kjbb79d33zzjafPjBkzFBISoqVLl3o9d+zYsYqIiNC2bdu0du1aXXLJJZKk22+/3bNUxel0NjjGgoICjR8/Xp06dVJYWJgSExOVkZGhiooKT5+Tl5zMmjVLiYmJioyM1MCBA72K5iovvfSSunfvLofDoR49eignJ0djx471zILv2bNHHTt2lCRlZGR4zmfs2LFe43z99df61a9+pdatWysmJkZ33HGHjhw50qDzTU9P17hx4/TPf/5T69at87TXtKRnzpw5uuiiixQZGamoqChdcMEFevjhhyVJ8+fP14033ihJGjJkiOcc5s+f7xmvV69eWrdunVJSUhQREaE77rij1mNJktvt1hNPPKHOnTurZcuW6t+/v1avXu3V5+TX8WRVy6Cq2Gw2HTt2TAsWLPDEVnXM2pb0LFmyRAMHDlRERISioqKUlpZW7X+yqo6zfft2v+cGgLVQ8AOwPLfbrREjRmjGjBkaPXq0/u///k8zZsxQbm6uBg8erOPHj0uSHnroIQ0dOlRjxoxRfn6+JGnevHlasGCB/vSnP6l37976yU9+onnz5kmSHnnkEX300Uf66KOP9Jvf/KZBMRYUFOjSSy/VypUr9dhjj2n58uW68847lZWVpXHjxlXr//zzzys3N1fZ2dl6/fXXdezYMV177bVehd7cuXN11113qU+fPlq0aJEeeeQRZWRkeBWXcXFxWrFihSTpzjvv9JzPo48+6nW866+/Xt27d9dbb72l//3f/1VOTo4efPDBBp2zJP385z+XJK+C/1QLFy7UhAkTlJqaqrfffluLFy/Wgw8+qGPHjkmSrrvuOmVmZnpel6pzuO666zxjHDx4ULfeeqtGjx6tZcuWacKECaeNa/bs2VqxYoWys7P12muvKSQkREOHDq1WdPvio48+Unh4uK699lpPbKdbSpSTk6MRI0YoOjpab7zxhl5++WUVFRVp8ODB+uCDD6r1b6zcALAQAwAWM2bMGNOqVSvP9htvvGEkmbfeesur38aNG40k88ILL3jaDh06ZDp16mQuvfRS88knn5iIiAhz66231vi8efPm+RTPe++9ZySZv/3tb7X2GT9+vImMjDT5+fle7U8//bSRZLZv326MMWb37t1Gkundu7epqKjw9Pv444+NJPPGG28YY4yprKw0sbGxZsCAAV7j5efnG7vdbrp06eJp++abb4wkM3369GpxTZ8+3UgyM2fO9GqfMGGCadmypXG73ac991NzcarPPvvMSDL33HOPpy01NdWkpqZ6tu+77z7Tpk2b0x7nb3/7m5Fk3nvvvWr7UlNTjSSzevXqGvedfKyq1zc+Pt4cP37c015cXGzatWtnrrnmGq9zO/l1rFL1mp2sVatWZsyYMdX6Vr03quKurKw08fHxpnfv3qaystLTr6SkxJxzzjkmJSWl2nHqmxsAZw9m+AFY3jvvvKM2bdpo+PDhqqio8DwuvvhixcbGes14t2/fXn/961/1ySefKCUlRZ07d9aLL77YJDEOGTJE8fHxXjEOHTpUkpSXl+fV/7rrrlNoaKhnu0+fPpLk+Z+JnTt3qqCgQL/85S+9nte5c2cNGjSozvFVzcSffLwTJ06osLCwzmOdzBhzxj6XXnqpDh8+rF/96lf6xz/+Ua8rBrVt21ZXXXWVz/1HjRqlli1berajoqI0fPhwrVu3TpWVlXU+vq927typAwcO6LbbblNIyI8/oiMjI3X99ddrw4YNKi0t9XpOY+UGgHVQ8AOwvK+//lqHDx9WWFiY7Ha716OgoKBaATlgwAD17NlTJ06c0D333KNWrVo1SYxLly6tFl/Pnj0lqVqM7du399p2OByS5Fme9O2330qSYmJiqh2rprYzOdPx6qvqF5T4+Pha+9x222165ZVXlJ+fr+uvv17nnHOOBgwYoNzcXJ+PExcXV6e4YmNja2wrLy/X0aNH6zRWXVTlraZ44+Pj5Xa7VVRU5NXeWLkBYB1cpQeA5XXo0EHt27f3rFU/VVRUlNf29OnTtW3bNvXr10+PPfaYhg0bpm7dujV6jH369NETTzxR4/7TFcQ1qSoCv/7662r7CgoK6h5gI1myZImkM18O9Pbbb9ftt9+uY8eOad26dZo+fbqGDRumL774Ql26dDnjcU7+Eq0vanqNCgoKFBYWpsjISElSy5Yta7yPQH3vWSD9mLeDBw9W23fgwAGFhISobdu29R4fwNmJGX4Aljds2DB9++23qqysVP/+/as9kpOTPX1zc3OVlZWlRx55RLm5uWrdurVuuukmlZeXe/o0xgzqsGHD9O9//1vnnXdejTHWteBPTk5WbGys3nzzTa/2vXv3av369V5tgZoRzs3N1Z///GelpKTo8ssv9+k5rVq10tChQzVt2jSVl5dr+/btkvx/DosWLdKJEyc82yUlJVq6dKmuuOIKz1Kqrl27qrCw0OuXqvLycq1cubLaeA6Hw6fYkpOTde655yonJ8drudOxY8f01ltvea7cAwB1wQw/AMu7+eab9frrr+vaa6/VxIkTdemll8put+urr77Se++9pxEjRugXv/iF50ouqampmj59ukJCQvTXv/5VV155paZMmaLs7GxJ0nnnnafw8HC9/vrruvDCCxUZGan4+PgzFuU1XTZTklJTU/X73/9eubm5SklJ0QMPPKDk5GSdOHFCe/bs0bJly/Tiiy963ZzqTEJCQpSRkaHx48frhhtu0B133KHDhw8rIyNDcXFxXuvDo6Ki1KVLF/3jH//Q1VdfrXbt2qlDhw51uoHV6bjdbs+5l5WVae/evVq+fLnefPNNXXjhhdV+KTnVuHHjFB4erkGDBikuLk4FBQXKyspS69atPZdI7dWrl6Tvr0wUFRWlli1bKjExsdpyF1+FhoYqLS1NkydPltvt1pNPPqni4mLPDcok6aabbtJjjz2mm2++Wf/zP/+jEydO6LnnnqtxjX/v3r21du1aLV26VHFxcYqKivL6RbNKSEiIZs6cqVtuuUXDhg3T+PHjVVZWpqeeekqHDx/WjBkz6nU+AM5ygf7WMAD4W01XhnG5XObpp582F110kWnZsqWJjIw0F1xwgRk/frzZtWuXqaioMKmpqSYmJsYcPHjQ67lPPfWUkWTefvttT9sbb7xhLrjgAmO322u9wk2Vqiux1PaoukLLN998Yx544AGTmJho7Ha7adeunenXr5+ZNm2aOXr0qDHmx6vIPPXUU9WOU1Mcc+fONeeff74JCwsz3bt3N6+88ooZMWKE6du3r1e/d9991/Tt29c4HA4jyXNFmaorwXzzzTde/efNm2ckmd27d9d63sZ8n4uTzzU8PNx07tzZDB8+3LzyyiumrKys2nNOvXLOggULzJAhQ0xMTIwJCwsz8fHx5pe//KXZunWr1/Oys7NNYmKiCQ0N9bqKUmpqqunZs2eN8dV2lZ4nn3zSZGRkmE6dOpmwsDDTt29fs3LlymrPX7Zsmbn44otNeHi46datm5k9e3aNV+nZsmWLGTRokImIiDCSPMc89So9VRYvXmwGDBhgWrZsaVq1amWuvvpq8+GHH3r1aWhuAJw9bMb4cIkEAIAlHD58WN27d9fIkSM1d+7cQIcDAGgCLOkBAIsqKCjQE088oSFDhqh9+/bKz8/XH//4R5WUlGjixImBDg8A0EQo+AHAohwOh/bs2aMJEybou+++U0REhC677DK9+OKLnst9AgCsjyU9AAAAgIVxWU4AAADAwij4AQAAAAuj4AcAAAAszPJf2nW73Tpw4ICioqLqfGt1AAAAoLkyxqikpETx8fFeN1Q8leUL/gMHDighISHQYQAAAACNYt++fae9G7vlC/6oqChJ378Q0dHRjX48l8ulVatWKT09XXa7vdGPh6ZHjq2N/FofObY28mt95PhHxcXFSkhI8NS7tbF8wV+1jCc6OrrJCv6IiAhFR0ef9W9CqyLH1kZ+rY8cWxv5tT5yXN2Zlq3zpV0AAADAwij4AQAAAAuj4AcAAAAszPJr+AEAAFB3lZWVcrlcgQ6jGpfLpRYtWujEiROqrKwMdDiNKjQ0VC1atGjwpeUp+AEAAODl6NGj+uqrr2SMCXQo1RhjFBsbq3379p0V91iKiIhQXFycwsLC6j0GBT8AAAA8Kisr9dVXXykiIkIdO3ZsdkW12+3W0aNHFRkZedqbTQU7Y4zKy8v1zTffaPfu3UpKSqr3+VLwAwAAwMPlcskYo44dOyo8PDzQ4VTjdrtVXl6uli1bWrrgl6Tw8HDZ7Xbl5+d7zrk+rP0qAQAAoF6a28z+2cofv9RQ8AMAAAAWRsEPAAAAWBgFPwAAAGBhfGkXAACc0dRF23zqlzWqdyNHgkDx9T3gL3V9L40dO1aHDx/W4sWLvdrXrl2rIUOGqKioSG3atPFfgEGEGX4AAADAwij4AQAAcFZwOp26+OKLvdqys7PVtWtXz/bYsWM1cuRIZWZmKiYmRm3atFFGRoYqKir0P//zP2rXrp06deqkV155xWuchx56SN27d1dERIS6deumRx991OtOxVXH/stf/qKuXbuqdevWuvnmm1VSUtKYpyyJgh8AAADwsmbNGh04cEDr1q3TrFmz5HQ6NWzYMLVt21b//Oc/dffdd+vuu+/Wvn37PM+JiorS/PnztWPHDj377LN66aWX9Mc//tFr3P/+979avHix3nnnHb3zzjvKy8vTjBkzGv18KPgBAABgCe+8844iIyO9HkOHDq3zOO3atdNzzz2n5ORk3XHHHUpOTlZpaakefvhhJSUlaerUqQoLC9OHH37oec4jjzyilJQUde3aVcOHD9dvf/tbvfnmm17jut1uzZ8/X7169dIVV1yh2267TatXr27weZ9JQAv+rl27ymazVXvce++9kr6/pbDT6VR8fLzCw8M1ePBgbd++PZAhAwAAoJkaMmSItmzZ4vX485//XOdxevbs6XXDq5iYGPXu/eOXiENDQ9W+fXsVFhZ62v7+97/r8ssvV2xsrCIjI/Xoo49q7969XuN27dpVUVFRnu24uDivMRpLQAv+jRs36uDBg55Hbm6uJOnGG2+UJM2cOVOzZs3S7NmztXHjRsXGxiotLa1J1joBAAAguLRq1Urnn3++1+Pcc8/17A8JCZExxus5J6+zr2K32722bTZbjW1ut1uStGHDBt18880aOnSo3nnnHX366aeaNm2aysvLzzhu1RiNKaCX5ezYsaPX9owZM3TeeecpNTVVxhhlZ2dr2rRpGjVqlCRpwYIFiomJUU5OjsaPHx+IkAEAABCkOnbsqIKCAhljZLPZJElbtmxp8LgffvihunTpomnTpnna8vPzGzyuvzSb6/CXl5frtdde0+TJk2Wz2fTll1+qoKBA6enpnj4Oh0Opqalav359rQV/WVmZysrKPNvFxcWSvv/trabf4Pyt6hhNcSwEBjm2NvJrfeS4fkLl2yxkoF9X8ttwLpdLxhi53W7v2edTZsYbW20z31Uz9FUxntx+atvJ47jdbl155ZX65ptv9OSTT+r666/XypUrtXz5ckVHR3v61TbO6dq6deumvXv3KicnR5dccomWLVumt99+2+v4VXGfGvPpzrVqnzFGLpdLoaGhXvt8fZ83m4J/8eLFOnz4sMaOHStJKigokPT9mqmTxcTEnPY3pqysLGVkZFRrX7VqlSIiIvwX8BlULU+CdZFjayO/1keO6+aS0DP3kaRly/Y0ahy+Ir/116JFC8XGxuro0aNeS1Ieurpzk8ZRNWlbm1OXeLtcLlVUVFR7Xmlpqaf/ueeeq6efflqzZs3S448/ruHDh+vee+/VggULvCaJTx2noqJC5eXlXm1ut1snTpxQcXGxhgwZonvuuUf333+/ysvLlZaWpt/97neaMWOG5zllZWWqrKz0GuPEiRNyu92nPdfy8nIdP35c69atU0VFRY3ndiY2c+pCpgD56U9/qrCwMC1dulSStH79eg0aNEgHDhxQXFycp9+4ceO0b98+rVixosZxaprhT0hI0KFDhxQdHd24J6Hv3yS5ublKS0urtk4L1kCOrY38Wh85rp+MpTt86jd9eI9GjuT0yG/DnThxQvv27VPXrl3VsmXLQIdTjTFGJSUlioqK8izLsbITJ05oz549SkhIqJaP4uJidejQQUeOHDltndssZvjz8/P17rvvatGiRZ622NhYSd/P9J9c8BcWFlab9T+Zw+GQw+Go1m6325v0g9/Ux0PTI8fWRn6tjxzXTaWP1/loLq8p+a2/yspK2Ww2hYSEeF2pprmoWv5SFaPVhYSEeL40fOp72tf3eLN4lebNm6dzzjlH1113nactMTFRsbGxXv8lV15erry8PKWkpAQiTAAAACDoBHyG3+12a968eRozZoxatPgxHJvNpkmTJikzM1NJSUlKSkpSZmamIiIiNHr06ABGDAAAAASPgBf87777rvbu3as77rij2r4pU6bo+PHjmjBhgoqKijRgwACtWrXK64YFAAAAAGoX8II/PT292g0QqthsNjmdTjmdzqYNCgAAALCIZrGGHwAAAEDjoOAHAAAALIyCHwAAALAwCn4AAADAwgL+pV0AAAAEgaUTm/Z4w59t2uNZGDP8AAAAsIz169crNDRUP/vZz7za9+zZI5vNpi1btgQmsACi4AcAAIBlvPLKK7r//vv1wQcfaO/evYEOp1mg4AcAAIAlHDt2TG+++abuueceDRs2TPPnzw90SM0CBT8AAAAs4a9//auSk5OVnJysW2+9VfPmzav1Bq9nEwp+AAAAWMLLL7+sW2+9VZL0s5/9TEePHtXq1asDHFXgUfADAAAg6O3cuVMff/yxbr75ZklSixYtdNNNN+mVV14JcGSBx2U5AQAAEPRefvllVVRU6Nxzz/W0GWNkt9tVVFQUwMgCjxl+AAAABLWKigq9+uqreuaZZ7RlyxbP41//+pe6dOmi119/PdAhBhQz/AAAAAhq77zzjoqKinTnnXeqdevWXvtuuOEGvfzyyxo2bFiAogs8Cn4AAACcWTO+8+3LL7+sa665plqxL0nXX3+9MjMz9d133wUgsuaBgh8AAABBbenSpbXu+8lPfuK5NOfZeolO1vADAAAAFkbBDwAAAFgYBT8AAABgYRT8AAAAgIVR8AMAAKCas/ULrs2NP/JAwQ8AAACP0NBQSVJ5eXmAI4EklZaWSpLsdnu9x+CynAAAAPBo0aKFIiIi9M0338hutyskpHnND7vdbpWXl+vEiRPNLjZ/MsaotLRUhYWFatOmjecXsfqg4AcAAICHzWZTXFycdu/erfz8/ECHU40xRsePH1d4eLhsNlugw2l0bdq0UWxsbIPGoOAHAACAl7CwMCUlJTXLZT0ul0vr1q3TlVde2aBlLsHAbrc3aGa/CgU/AABnsamLtgU6BDRTISEhatmyZaDDqCY0NFQVFRVq2bKl5Qt+f7HuwicAAAAAFPwAAACAlVHwAwAAABYW8IJ///79uvXWW9W+fXtFRETo4osv1ubNmz37jTFyOp2Kj49XeHi4Bg8erO3btwcwYgAAACB4BLTgLyoq0qBBg2S327V8+XLt2LFDzzzzjNq0aePpM3PmTM2aNUuzZ8/Wxo0bFRsbq7S0NJWUlAQucAAAACBIBPQqPU8++aQSEhI0b948T1vXrl09fzfGKDs7W9OmTdOoUaMkSQsWLFBMTIxycnI0fvz4pg4ZAAAACCoBLfiXLFmin/70p7rxxhuVl5enc889VxMmTNC4ceMkSbt371ZBQYHS09M9z3E4HEpNTdX69etrLPjLyspUVlbm2S4uLpb0/TVbXS5XI5+RPMdoimMhMMixtZFf6yPH3kLl9ut4gX5dya/1keMf+foa2IwxppFjqVXVtV0nT56sG2+8UR9//LEmTZqk//f//p9+/etfa/369Ro0aJD279+v+Ph4z/Puuusu5efna+XKldXGdDqdysjIqNaek5OjiIiIxjsZAAAAoAmVlpZq9OjROnLkiKKjo2vtF9AZfrfbrf79+yszM1OS1LdvX23fvl1z5szRr3/9a0+/U2+bbIyp9VbKU6dO1eTJkz3bxcXFSkhIUHp6+mlfCH9xuVzKzc1VWloaN4OwqKDK8fKHfOs39MnGjSOIBFV+US/k2FvG0h1+HW/68B5+Ha+uyK/1keMfVa1kOZOAFvxxcXHq0cP7H4YLL7xQb731liQpNjZWklRQUKC4uDhPn8LCQsXExNQ4psPhkMPhqNZut9ub9E3R1MdD0wuKHNsqfevX3M8jAIIiv2gQcvy9Sj9fv6O5vKbk1/rIse+ft4BepWfQoEHauXOnV9sXX3yhLl26SJISExMVGxur3Nxcz/7y8nLl5eUpJSWlSWMFAAAAglFAZ/gffPBBpaSkKDMzU7/85S/18ccfa+7cuZo7d66k75fyTJo0SZmZmUpKSlJSUpIyMzMVERGh0aNHBzJ0AAAAICgEtOC/5JJL9Pbbb2vq1Kn6/e9/r8TERGVnZ+uWW27x9JkyZYqOHz+uCRMmqKioSAMGDNCqVasUFRUVwMgBAACA4BDQgl+Shg0bpmHDhtW632azyel0yul0Nl1QAAAAgEUEdA0/AAAAgMZFwQ8AAABYGAU/AAAAYGEU/AAAAICFUfADAAAAFkbBDwAAAFhYwC/LCeAstnSib/2GP9u4cQBBZOqibT71yxrVu5EjARAsmOEHAAAALIyCHwAAALAwCn4AAADAwij4AQAAAAuj4AcAAAAsjIIfAAAAsDAKfgAAAMDCuA4/AABoctxPAGg6zPADAAAAFkbBDwAAAFgYBT8AAABgYazhB4CGWDrRt37Dn23cOAAAqAUz/AAAAICFUfADAAAAFkbBDwAAAFgYBT8AAABgYRT8AAAAgIVR8AMAAAAWRsEPAAAAWBjX4QcAoBmYumhboEMAYFHM8AMAAAAWRsEPAAAAWBgFPwAAAGBhAS34nU6nbDab1yM2Ntaz3xgjp9Op+Ph4hYeHa/Dgwdq+fXsAIwYAAACCS8Bn+Hv27KmDBw96Htu2/filpZkzZ2rWrFmaPXu2Nm7cqNjYWKWlpamkpCSAEQMAAADBI+AFf4sWLRQbG+t5dOzYUdL3s/vZ2dmaNm2aRo0apV69emnBggUqLS1VTk5OgKMGAAAAgkPAL8u5a9cuxcfHy+FwaMCAAcrMzFS3bt20e/duFRQUKD093dPX4XAoNTVV69ev1/jx42scr6ysTGVlZZ7t4uJiSZLL5ZLL5Wrck/nhOCf/CesJqhybUN/6BepcmmF8dc5vMzwHnF5z/QyHyu3X8Xw9v+Z+3LrmqbnmF/5Djn/k62tgM8aYRo6lVsuXL1dpaam6d++ur7/+Wo8//rg+//xzbd++XTt37tSgQYO0f/9+xcfHe55z1113KT8/XytXrqxxTKfTqYyMjGrtOTk5ioiIaLRzAQAAAJpSaWmpRo8erSNHjig6OrrWfgEt+E917NgxnXfeeZoyZYouu+wyDRo0SAcOHFBcXJynz7hx47Rv3z6tWLGixjFqmuFPSEjQoUOHTvtC+IvL5VJubq7S0tJkt9sb/XhoekGV4+UP+dZv6JONG0dtmmF8dc5vMzwHnF5z/QxnLN3h1/GmD+9hieP6Ol6V5ppf+A85/lFxcbE6dOhwxoI/4Et6TtaqVSv17t1bu3bt0siRIyVJBQUFXgV/YWGhYmJiah3D4XDI4XBUa7fb7U36pmjq46HpBUWObZW+9QvUeTTj+HzObzM+B5xec/sMV/r5a3W+nltzP259c9Tc8gv/I8e+fz4C/qXdk5WVlemzzz5TXFycEhMTFRsbq9zcXM/+8vJy5eXlKSUlJYBRAgAAAMEjoDP8v/vd7zR8+HB17txZhYWFevzxx1VcXKwxY8bIZrNp0qRJyszMVFJSkpKSkpSZmamIiAiNHj06kGEDOBssf8j32XvgNKYu2nbmTgDQiAJa8H/11Vf61a9+pUOHDqljx4667LLLtGHDBnXp0kWSNGXKFB0/flwTJkxQUVGRBgwYoFWrVikqKiqQYQMAAABBI6AF/8KFC0+732azyel0yul0Nk1AAAAAgMU0qzX8AAAAAPyrWV2lB0Azt3Sib/2GP9u4cQAAAJ8xww8AAABYGAU/AAAAYGEU/AAAAICFsYYfAAA0W3W9j0Go3LokVMpYuqPGu/lmjertr9CAoMEMPwAAAGBhFPwAAACAhVHwAwAAABbGGn4AACyormvfAVgXM/wAAACAhVHwAwAAABZGwQ8AAABYGGv4ATR/Syf61m/4s40bBwAAQYgZfgAAAMDCKPgBAAAAC6PgBwAAACyMNfwAzi5n+j6ACZWU2iShAFbE9f+B5ocZfgAAAMDCKPgBAAAAC6PgBwAAACyMgh8AAACwML60C8D/fL1RllWOCwBAM8YMPwAAAGBhFPwAAACAhdWr4O/WrZu+/fbbau2HDx9Wt27dGhwUAAAAAP+o1xr+PXv2qLKyslp7WVmZ9u/f3+CgAABo7rjBFIBgUaeCf8mSJZ6/r1y5Uq1bt/ZsV1ZWavXq1eratavfggMAAADQMHUq+EeOHClJstlsGjNmjNc+u92url276plnnvFbcAAAAAAapk4Fv9vtliQlJiZq48aN6tChQ6MEBQAAAMA/6rWGf/fu3f6OQ1lZWXr44Yc1ceJEZWdnS5KMMcrIyNDcuXNVVFSkAQMG6Pnnn1fPnj39fnwAAGB9vn73ImtU70aOBGg69b7x1urVq7V69WoVFhZ6Zv6rvPLKK3Uaa+PGjZo7d6769Onj1T5z5kzNmjVL8+fPV/fu3fX4448rLS1NO3fuVFRUVH1DBwAAAM4a9bosZ0ZGhtLT07V69WodOnRIRUVFXo+6OHr0qG655Ra99NJLatu2rafdGKPs7GxNmzZNo0aNUq9evbRgwQKVlpYqJyenPmEDAAAAZ516zfC/+OKLmj9/vm677bYGB3Dvvffquuuu0zXXXKPHH3/c0757924VFBQoPT3d0+ZwOJSamqr169dr/PjxNY5XVlamsrIyz3ZxcbEkyeVyyeVyNTjeM6k6RlMcC4ERVDk2ob718/VcfB0viLl+OEeXv881GN4vZwl/fYZD5T5zJzS5kB/yEtLA/ATFv/FnqaD6OdzIfH0N6lXwl5eXKyUlpT5P9bJw4UJ98skn2rhxY7V9BQUFkqSYmBiv9piYGOXn59c6ZlZWljIyMqq1r1q1ShEREQ2M2He5ublNdiwERnDkONW3bsuW+Xc8C8jV5ZLx44A+v8ZoKg39DF9i/d9/g1q/0L0Nev6yZXv8EwgaTXD8HG5cpaWlPvWrV8H/m9/8Rjk5OXr00Ufr83RJ0r59+zRx4kStWrVKLVu2rLWfzWbz2jbGVGs72dSpUzV58mTPdnFxsRISEpSenq7o6Oh6x+srl8ul3NxcpaWlyW63N/rx0PSCKsfLH/Kt39An/TteEHOZUOXqcqXpA9lt1W8wWG++vsZodP76DGcs3eHHqOAvIXKrX+heba7sLHf9Vi5LkqYP7+HHqOBPQfVzuJFVrWQ5k3oV/CdOnNDcuXP17rvvqk+fPtVe7FmzZp1xjM2bN6uwsFD9+vXztFVWVmrdunWaPXu2du7cKen7mf64uDhPn8LCwmqz/idzOBxyOBzV2u12e5O+KZr6eGh6QZFjXwtWX8/DnwVwc2Yku63SvwV/c3+vnIUa+hmubEAxicbnVkiDctTs/31HcPwcbmS+nn+9Cv6tW7fq4osvliT9+9//9tp3utn3k1199dXats370li33367LrjgAj300EPq1q2bYmNjlZubq759+0r6filRXl6ennySmTIAAADAF/Uq+N97770GHzgqKkq9evXyamvVqpXat2/vaZ80aZIyMzOVlJSkpKQkZWZmKiIiQqNHj27w8QEAAICzQb2vw98UpkyZouPHj2vChAmeG2+tWrWKa/ADAAAAPqpXwT9kyJDTLt1Zs2ZNvYJZu3at17bNZpPT6ZTT6azXeAAAAMDZrl4Ff9X6/Soul0tbtmzRv//9b40ZM8YfcQEAAADwg3oV/H/84x9rbHc6nTp69GiDAgIAAADgP369ptitt96qV155xZ9DAgAAAGgAvxb8H3300WlvogUAAACgadVrSc+oUaO8to0xOnjwoDZt2tSgu+8CAAAA8K96FfytW7f22g4JCVFycrJ+//vfKz093S+BAQAQCFMXbTtzJwAIIvUq+OfNm+fvOAAAAAA0ggbdeGvz5s367LPPZLPZ1KNHD/Xt29dfcQEAAADwg3oV/IWFhbr55pu1du1atWnTRsYYHTlyREOGDNHChQvVsWNHf8cJAAAAoB7qVfDff//9Ki4u1vbt23XhhRdKknbs2KExY8bogQce0BtvvOHXIAGcYunEQEcAAACCRL0K/hUrVujdd9/1FPuS1KNHDz3//PN8aRcAAABoRup1HX632y273V6t3W63y+12NzgoAAAAAP5Rr4L/qquu0sSJE3XgwAFP2/79+/Xggw/q6quv9ltwAAAAABqmXkt6Zs+erREjRqhr165KSEiQzWbT3r171bt3b7322mv+jhFAY+M7AQAAWFa9Cv6EhAR98sknys3N1eeffy5jjHr06KFrrrnG3/EBAAAAaIA6LelZs2aNevTooeLiYklSWlqa7r//fj3wwAO65JJL1LNnT73//vuNEigAAACAuqtTwZ+dna1x48YpOjq62r7WrVtr/PjxmjVrlt+CAwAAANAwdVrS869//UtPPvlkrfvT09P19NNPNzgoAMAZ+Pq9i+HPNm4cAIBmr04z/F9//XWNl+Os0qJFC33zzTcNDgoAAACAf9Sp4D/33HO1bdu2Wvdv3bpVcXFxDQ4KAAAAgH/UqeC/9tpr9dhjj+nEiRPV9h0/flzTp0/XsGHD/BYcAAAAgIap0xr+Rx55RIsWLVL37t113333KTk5WTabTZ999pmef/55VVZWatq0aY0VKxC8WG8N3gMBN3XR9/9DHSq3LgmVMpbuUGX97j8JAEGlTgV/TEyM1q9fr3vuuUdTp06VMUaSZLPZ9NOf/lQvvPCCYmJiGiVQAAAAAHVX5xtvdenSRcuWLVNRUZH+85//yBijpKQktW3btjHiAwAAANAA9brTriS1bdtWl1xyiT9jAQAAAOBnLF4EAAAALIyCHwAAALAwCn4AAADAwij4AQAAAAuj4AcAAAAsLKAF/5w5c9SnTx9FR0crOjpaAwcO1PLlyz37jTFyOp2Kj49XeHi4Bg8erO3btwcwYgAAACC4BLTg79Spk2bMmKFNmzZp06ZNuuqqqzRixAhPUT9z5kzNmjVLs2fP1saNGxUbG6u0tDSVlJQEMmwAAAAgaAS04B8+fLiuvfZade/eXd27d9cTTzyhyMhIbdiwQcYYZWdna9q0aRo1apR69eqlBQsWqLS0VDk5OYEMGwAAAAga9b7xlr9VVlbqb3/7m44dO6aBAwdq9+7dKigoUHp6uqePw+FQamqq1q9fr/Hjx9c4TllZmcrKyjzbxcXFkiSXyyWXy9W4J/HDcU7+E9ZTrxybUF8H9+94qDPXD6+tK1Cvsb/fA/xb5BEqtyQp5JQ/YS3+yi8/x5svaq0f+foa2IwxppFjOa1t27Zp4MCBOnHihCIjI5WTk6Nrr71W69ev16BBg7R//37Fx8d7+t91113Kz8/XypUraxzP6XQqIyOjWntOTo4iIiIa7TwAAACAplRaWqrRo0fryJEjio6OrrVfwGf4k5OTtWXLFh0+fFhvvfWWxowZo7y8PM9+m83m1d8YU63tZFOnTtXkyZM928XFxUpISFB6evppXwh/cblcys3NVVpamux2e6MfD02vXjle/pBv/YY+6d/xUGcuE6pcXa40fSC7rbLpA/D3e8DX8c4CGUt3SPp+5rdf6F5truwsNxersxx/5Xf68B5+jAr+RK31o6qVLGcS8II/LCxM559/viSpf//+2rhxo5599lk99ND3P8wKCgoUFxfn6V9YWKiYmJhax3M4HHI4HNXa7XZ7k74pmvp4aHp1yrGvhaO/x0P9GMluqwxMwR+o99RZoPKU4s+tkGptsI6G5pef4c0ftZbv79Nm9y+dMUZlZWVKTExUbGyscnNzPfvKy8uVl5enlJSUAEYIAAAABI+AzvA//PDDGjp0qBISElRSUqKFCxdq7dq1WrFihWw2myZNmqTMzEwlJSUpKSlJmZmZioiI0OjRowMZNgCguVo6sdZdI7/6TpLktrVQYeeRTRQQrG7qom1n7JM1qncTRALULqAF/9dff63bbrtNBw8eVOvWrdWnTx+tWLFCaWlpkqQpU6bo+PHjmjBhgoqKijRgwACtWrVKUVFRgQwbAAAACBoBLfhffvnl0+632WxyOp1yOp1NExAAAABgMc1uDT8AAAAA/wn4VXoAAI3oNGvavQx/tnHjqE1zjw8ALIAZfgAAAMDCKPgBAAAAC6PgBwAAACyMNfwAgKB28nXQq661DwD4ETP8AAAAgIVR8AMAAAAWRsEPAAAAWBhr+AEAAE5x8ndDgGDHDD8AAABgYRT8AAAAgIVR8AMAAAAWxhp+AIDf+LruOWtU70aOBABQhRl+AAAAwMIo+AEAAAALo+AHAAAALIw1/EBzsnRioCMAmkTVWv+RX31Xa58Bie1+3DjNZ+N0YwAAmOEHAAAALI2CHwAAALAwCn4AAADAwij4AQAAAAuj4AcAAAAsjIIfAAAAsDAKfgAAAMDCuA4/UJPTXQ/fhEpKlZY/JP18VpOFBASTkV/NDHQIAIAfMMMPAAAAWBgFPwAAAGBhFPwAAACAhQV0DX9WVpYWLVqkzz//XOHh4UpJSdGTTz6p5ORkTx9jjDIyMjR37lwVFRVpwIABev7559WzZ88ARg4AjeR03x85y/xz93eBDgFoUlMXbfOpX9ao3o0cCawmoDP8eXl5uvfee7Vhwwbl5uaqoqJC6enpOnbsmKfPzJkzNWvWLM2ePVsbN25UbGys0tLSVFJSEsDIAQAAgOAQ0Bn+FStWeG3PmzdP55xzjjZv3qwrr7xSxhhlZ2dr2rRpGjVqlCRpwYIFiomJUU5OjsaPHx+IsAEAAICg0awuy3nkyBFJUrt27SRJu3fvVkFBgdLT0z19HA6HUlNTtX79+hoL/rKyMpWVlXm2i4uLJUkul0sul6sxw/cc5+Q/EaRMaK27XD/sc5lQydc8n2Y8NC9e+T2b+OnfrFC5JUluW7P68eKlKraQH2KFtVTltTnl19eaINTHmM/2GoNa60e+vgY2Y4xp5Fh8YozRiBEjVFRUpPfff1+StH79eg0aNEj79+9XfHy8p+9dd92l/Px8rVy5sto4TqdTGRkZ1dpzcnIUERHReCcAAAAANKHS0lKNHj1aR44cUXR0dK39ms0UzH333aetW7fqgw8+qLbPZrN5bRtjqrVVmTp1qiZPnuzZLi4uVkJCgtLT00/7QviLy+VSbm6u0tLSZLfbG/14aCTLH6p1l8uEKleXK00fyH5tZoPHQ/PilV9bZaDDaTpDn/TLMBlLd0iSrtuf7ZfxGoPb1kKHEoapw753FGIqTtv3/86d1DRBwW9C5Fa/0L3aXNlZbotejHD68B6BDiGgqLV+VLWS5UyaRcF///33a8mSJVq3bp06derkaY+NjZUkFRQUKC4uztNeWFiomJiYGsdyOBxyOBzV2u12e5O+KZr6ePCzMxV6RrLbKn3P8dlUOFpBVX7Pprz56d+ryh8KrDMV0s1BiKk4Y5yVFi0YzwZuhVg2f9QX36PW8v29ENBPgjFG9913nxYtWqQ1a9YoMTHRa39iYqJiY2OVm5vraSsvL1deXp5SUlKaOlwAAAAg6AR0hv/ee+9VTk6O/vGPfygqKkoFBQWSpNatWys8PFw2m02TJk1SZmamkpKSlJSUpMzMTEVERGj06NGBDB1Nxddrkg9/tnHjAKyOzxoAWFZAC/45c+ZIkgYPHuzVPm/ePI0dO1aSNGXKFB0/flwTJkzw3Hhr1apVioqKauJoAQAAgOAT0ILflwsE2Ww2OZ1OOZ3Oxg8IAAAAsBhrfpsFAAAAgKRmcpUeoMn4uk4ZAADAIpjhBwAAACyMgh8AAACwMAp+AAAAwMJYww8A8N0Zvgcz8qvvmigQAICvmOEHAAAALIyCHwAAALAwCn4AAADAwij4AQAAAAvjS7uwBm6oBdTLP3fzJVsAsDpm+AEAAAALo+AHAAAALIyCHwAAALAw1vADDcF3BwAATWzqom0+9csa1buRI0GwYIYfAAAAsDAKfgAAAMDCKPgBAAAAC6PgBwAAACyMgh8AAACwMAp+AAAAwMIo+AEAAAAL4zr8AADUYuRXM33qt7jTlEaOBKg7rtePKszwAwAAABZGwQ8AAABYGAU/AAAAYGEU/AAAAICFUfADAAAAFkbBDwAAAFgYBT8AAABgYQEt+NetW6fhw4crPj5eNptNixcv9tpvjJHT6VR8fLzCw8M1ePBgbd++PTDBAgAAAEEooAX/sWPHdNFFF2n27Nk17p85c6ZmzZql2bNna+PGjYqNjVVaWppKSkqaOFIAAAAgOAX0TrtDhw7V0KFDa9xnjFF2dramTZumUaNGSZIWLFigmJgY5eTkaPz48U0ZKgAAABCUAlrwn87u3btVUFCg9PR0T5vD4VBqaqrWr19fa8FfVlamsrIyz3ZxcbEkyeVyyeVyNW7QPxzn5D/RQCY00BFU4/ohJlczjA0Nd7bl121rtj8GGk3VOfvz3EPl9ttYaJiQH3IRQk58Fmw1C7XWj3x9DZrtv/QFBQWSpJiYGK/2mJgY5efn1/q8rKwsZWRkVGtftWqVIiIi/BvkaeTm5jbZsawtNdAB1CpXl0sm0FGgsZw1+e0c6AAC51DCML+NdYn2+G0s+Ee/0L2BDiFoLFu2J9Ah1Au1llRaWupTv2Zb8Fex2Wxe28aYam0nmzp1qiZPnuzZLi4uVkJCgtLT0xUdHd1ocVZxuVzKzc1VWlqa7HZ7ox/P8pY/FOgIqnGZUOXqcqXpA9ltlYEOB352tuV3U35RoENocm5bCx1KGKYO+95RiKnwy5j/d+4kv4yDhguRW/1C92pzZWe5uRihT6YP7xHoEOqEWutHVStZzqTZFvyxsbGSvp/pj4uL87QXFhZWm/U/mcPhkMPhqNZut9ub9E3R1MezrOZacBnJbqs8KwrCs9JZlF9/FbzBKMRU+O38Kyksmx23QsiLj4K1XqHW8j13zfaTkJiYqNjYWK//rikvL1deXp5SUlICGBkAAAAQPAI6w3/06FH95z//8Wzv3r1bW7ZsUbt27dS5c2dNmjRJmZmZSkpKUlJSkjIzMxUREaHRo0cHMGr4xdKJgY4AAADgrBDQgn/Tpk0aMmSIZ7tq7f2YMWM0f/58TZkyRcePH9eECRNUVFSkAQMGaNWqVYqKigpUyAAAAEBQCWjBP3jwYBlT+2UwbDabnE6nnE5n0wUFAAAAWEizXcMPAAAAoOGa7VV6AAAA0PimLtrmU7+sUb0bORI0Fmb4AQAAAAuj4AcAAAAsjIIfAAAAsDDW8MO/uL4+UC//3P2dT/0GJLZr5EgAAFbDDD8AAABgYRT8AAAAgIVR8AMAAAAWxhp++LbufvizjR8HgDNirT+AQOF6/cGLGX4AAADAwij4AQAAAAuj4AcAAAAsjDX8TcHXa9OzTh6An/i61h/+MfKrmT71W9xpSiNHAgQPvhPQdJjhBwAAACyMgh8AAACwMAp+AAAAwMJYw9+c+Hutv6/jNfVYgAWwRh4AECyY4QcAAAAsjIIfAAAAsDAKfgAAAMDCKPgBAAAAC+NLuwBwkk35RVLn7/8MMRWBDgcAgo6vN9RC02GGHwAAALAwCn4AAADAwij4AQAAAAtjDT8AAE1k5Fczfeq3uNOURo4EwNmEGX4AAADAwij4AQAAAAuj4AcAAAAsLCjW8L/wwgt66qmndPDgQfXs2VPZ2dm64oorAh1W4CydGOgIAAAAmhVfr/+fNap3QMYLpGY/w//Xv/5VkyZN0rRp0/Tpp5/qiiuu0NChQ7V3795AhwYAAAA0e82+4J81a5buvPNO/eY3v9GFF16o7OxsJSQkaM6cOYEODQAAAGj2mvWSnvLycm3evFn/+7//69Wenp6u9evX1/icsrIylZWVebaPHDkiSfruu+/kcrkaL9gfuFwulZaW6ttvv5Xdbv++8VhFox8XTcdljEpVqm9VIbutMtDhwM9KTrhVWlqqkhNuhRh3oMNBI3Dbmn+OK0qLAx1C0HLLrdLQUrkqi+Vu/vOa8MG3337rtV1jrSXfPzenjlcbf4/XGEpKSiRJxpjT9mvWBf+hQ4dUWVmpmJgYr/aYmBgVFBTU+JysrCxlZGRUa09MTGyUGHG2+lOgA0CjejXQAaDRNfcc5wQ6AKDZePosG68+SkpK1Lp161r3N+uCv4rNZvPaNsZUa6sydepUTZ482bPtdrv13XffqX379rU+x5+Ki4uVkJCgffv2KTo6utGPh6ZHjq2N/FofObY28mt95PhHxhiVlJQoPj7+tP2adcHfoUMHhYaGVpvNLywsrDbrX8XhcMjhcHi1tWnTprFCrFV0dPRZ/ya0OnJsbeTX+sixtZFf6yPH3zvdzH6VZr24LSwsTP369VNubq5Xe25urlJSUgIUFQAAABA8mvUMvyRNnjxZt912m/r376+BAwdq7ty52rt3r+6+++5AhwYAAAA0e82+4L/pppv07bff6ve//70OHjyoXr16admyZerSpUugQ6uRw+HQ9OnTqy0rgnWQY2sjv9ZHjq2N/FofOa47mznTdXwAAAAABK1mvYYfAAAAQMNQ8AMAAAAWRsEPAAAAWBgFPwAAAGBhFPw+WLdunYYPH674+HjZbDYtXrzYa78xRk6nU/Hx8QoPD9fgwYO1fft2rz5lZWW6//771aFDB7Vq1Uo///nP9dVXXzXhWaA2WVlZuuSSSxQVFaVzzjlHI0eO1M6dO736kOPgNmfOHPXp08dzk5aBAwdq+fLlnv3k11qysrJks9k0adIkTxs5Dm5Op1M2m83rERsb69lPfq1h//79uvXWW9W+fXtFRETo4osv1ubNmz37yXP9UfD74NixY7rooos0e/bsGvfPnDlTs2bN0uzZs7Vx40bFxsYqLS1NJSUlnj6TJk3S22+/rYULF+qDDz7Q0aNHNWzYMFVWVjbVaaAWeXl5uvfee7Vhwwbl5uaqoqJC6enpOnbsmKcPOQ5unTp10owZM7Rp0yZt2rRJV111lUaMGOH5QUF+rWPjxo2aO3eu+vTp49VOjoNfz549dfDgQc9j27Ztnn3kN/gVFRVp0KBBstvtWr58uXbs2KFnnnlGbdq08fQhzw1gUCeSzNtvv+3ZdrvdJjY21syYMcPTduLECdO6dWvz4osvGmOMOXz4sLHb7WbhwoWePvv37zchISFmxYoVTRY7fFNYWGgkmby8PGMMObaqtm3bmj//+c/k10JKSkpMUlKSyc3NNampqWbixInGGD7DVjB9+nRz0UUX1biP/FrDQw89ZC6//PJa95PnhmGGv4F2796tgoICpaene9ocDodSU1O1fv16SdLmzZvlcrm8+sTHx6tXr16ePmg+jhw5Iklq166dJHJsNZWVlVq4cKGOHTumgQMHkl8Luffee3Xdddfpmmuu8Wonx9awa9cuxcfHKzExUTfffLO+/PJLSeTXKpYsWaL+/fvrxhtv1DnnnKO+ffvqpZde8uwnzw1Dwd9ABQUFkqSYmBiv9piYGM++goIChYWFqW3btrX2QfNgjNHkyZN1+eWXq1evXpLIsVVs27ZNkZGRcjgcuvvuu/X222+rR48e5NciFi5cqE8++URZWVnV9pHj4DdgwAC9+uqrWrlypV566SUVFBQoJSVF3377Lfm1iC+//FJz5sxRUlKSVq5cqbvvvlsPPPCAXn31VUl8jhuqRaADsAqbzea1bYyp1nYqX/qgad13333aunWrPvjgg2r7yHFwS05O1pYtW3T48GG99dZbGjNmjPLy8jz7yW/w2rdvnyZOnKhVq1apZcuWtfYjx8Fr6NChnr/37t1bAwcO1HnnnacFCxbosssuk0R+g53b7Vb//v2VmZkpSerbt6+2b9+uOXPm6Ne//rWnH3muH2b4G6jqKgGn/uZYWFjo+S00NjZW5eXlKioqqrUPAu/+++/XkiVL9N5776lTp06ednJsDWFhYTr//PPVv39/ZWVl6aKLLtKzzz5Lfi1g8+bNKiwsVL9+/dSiRQu1aNFCeXl5eu6559SiRQtPjsixdbRq1Uq9e/fWrl27+AxbRFxcnHr06OHVduGFF2rv3r2S+FncUBT8DZSYmKjY2Fjl5uZ62srLy5WXl6eUlBRJUr9+/WS32736HDx4UP/+9789fRA4xhjdd999WrRokdasWaPExESv/eTYmowxKisrI78WcPXVV2vbtm3asmWL59G/f3/dcsst2rJli7p160aOLaasrEyfffaZ4uLi+AxbxKBBg6pdEvuLL75Qly5dJPGzuMEC8EXhoFNSUmI+/fRT8+mnnxpJZtasWebTTz81+fn5xhhjZsyYYVq3bm0WLVpktm3bZn71q1+ZuLg4U1xc7Bnj7rvvNp06dTLvvvuu+eSTT8xVV11lLrroIlNRURGo08IP7rnnHtO6dWuzdu1ac/DgQc+jtLTU04ccB7epU6eadevWmd27d5utW7eahx9+2ISEhJhVq1YZY8ivFZ18lR5jyHGw++1vf2vWrl1rvvzyS7NhwwYzbNgwExUVZfbs2WOMIb9W8PHHH5sWLVqYJ554wuzatcu8/vrrJiIiwrz22muePuS5/ij4ffDee+8ZSdUeY8aMMcZ8f6mo6dOnm9jYWONwOMyVV15ptm3b5jXG8ePHzX333WfatWtnwsPDzbBhw8zevXsDcDY4VU25lWTmzZvn6UOOg9sdd9xhunTpYsLCwkzHjh3N1Vdf7Sn2jSG/VnRqwU+Og9tNN91k4uLijN1uN/Hx8WbUqFFm+/btnv3k1xqWLl1qevXqZRwOh7ngggvM3LlzvfaT5/qzGWNMYP5vAQAAAEBjYw0/AAAAYGEU/AAAAICFUfADAAAAFkbBDwAAAFgYBT8AAABgYRT8AAAAgIVR8AMAAAAWRsEPAAAAWBgFPwCcBWw2mxYvXhzoMAAAAUDBDwAWUFBQoPvvv1/dunWTw+FQQkKChg8frtWrVwc6tDMaO3asRo4cGegwAMCyWgQ6AABAw+zZs0eDBg1SmzZtNHPmTPXp00cul0srV67Uvffeq88//7xRjlteXq6wsLBGGbs+mls8ANBcMMMPAEFuwoQJstls+vjjj3XDDTeoe/fu6tmzpyZPnqwNGzZ4+h06dEi/+MUvFBERoaSkJC1ZssSzr7KyUnfeeacSExMVHh6u5ORkPfvss17HqZqJz8rKUnx8vLp37y5Jeu2119S/f39FRUUpNjZWo0ePVmFhoddzt2/fruuuu07R0dGKiorSFVdcof/+979yOp1asGCB/vGPf8hms8lms2nt2rWSpP379+umm25S27Zt1b59e40YMUJ79uw5YzwvvPCCkpKS1LJlS8XExOiGG27w58sNAEGHGX4ACGLfffedVqxYoSeeeEKtWrWqtr9Nmzaev2dkZGjmzJl66qmn9Kc//Um33HKL8vPz1a5dO7ndbnXq1ElvvvmmOnTooPXr1+uuu+5SXFycfvnLX3rGWL16taKjo5WbmytjjKTvZ9b/8Ic/KDk5WYWFhXrwwQc1duxYLVu2TNL3hfuVV16pwYMHa82aNYqOjtaHH36oiooK/e53v9Nnn32m4uJizZs3T5LUrl07lZaWasiQIbriiiu0bt06tWjRQo8//rh+9rOfaevWrZ6Z/FPj2bRpkx544AH95S9/UUpKir777ju9//77jfXyA0BwMACAoPXPf/7TSDKLFi06bT9J5pFHHvFsHz161NhsNrN8+fJanzNhwgRz/fXXe7bHjBljYmJiTFlZ2WmP9fHHHxtJpqSkxBhjzNSpU01iYqIpLy+vsf+YMWPMiBEjvNpefvllk5ycbNxut6etrKzMhIeHm5UrV9Yaz1tvvWWio6NNcXHxaWMEgLMJS3oAIIiZH2bZbTbbGfv26dPH8/dWrVopKirKa+nNiy++qP79+6tjx46KjIzUSy+9pL1793qN0bt372rr5D/99FONGDFCXbp0UVRUlAYPHixJnudu2bJFV1xxhex2u8/ntXnzZv3nP/9RVFSUIiMjFRkZqXbt2unEiRP673//W2s8aWlp6tKli7p166bbbrtNr7/+ukpLS30+LgBYEQU/AASxpKQk2Ww2ffbZZ2fse2rBbbPZ5Ha7JUlvvvmmHnzwQd1xxx1atWqVtmzZottvv13l5eVezzl12dCxY8eUnp6uyMhIvfbaa9q4caPefvttSfI8Nzw8vM7n5Xa71a9fP23ZssXr8cUXX2j06NG1xhMVFaVPPvlEb7zxhuLi4vTYY4/poosu0uHDh+scAwBYBQU/AASxdu3a6ac//amef/55HTt2rNp+Xwvd999/XykpKZowYYL69u2r888/32smvTaff/65Dh06pBkzZuiKK67QBRdcUO0Lu3369NH7778vl8tV4xhhYWGqrKz0avvJT36iXbt26ZxzztH555/v9WjduvVpY2rRooWuueYazZw5U1u3btWePXu0Zs2aM54LAFgVBT8ABLkXXnhBlZWVuvTSS/XWW29p165d+uyzz/Tcc89p4MCBPo1x/vnna9OmTVq5cqW++OILPfroo9q4ceMZn9e5c2eFhYXpT3/6k7788kstWbJEf/jDH7z63HfffSouLtbNN9+sTZs2adeuXfrLX/6inTt3SpK6du2qrVu3aufOnTp06JBcLpduueUWdejQQSNGjND777+v3bt3Ky8vTxMnTtRXX31VazzvvPOOnnvuOW3ZskX5+fl69dVX5Xa7lZyc7NPrAABWRMEPAEEuMTFRn3zyiYYMGaLf/va36tWrl9LS0rR69WrNmTPHpzHuvvtujRo1SjfddJMGDBigb7/9VhMmTDjj8zp27Kj58+frb3/7m3r06KEZM2bo6aef9urTvn17rVmzRkePHlVqaqr69eunl156ybPEaNy4cUpOTvZ8f+DDDz9URESE1q1bp86dO2vUqFG68MILdccdd+j48eOKjo6uNZ42bdpo0aJFuuqqq3ThhRfqxRdf1BtvvKGePXv69DoAgBXZTNU3vgAAAABYDjP8AAAAgIVR8AMAAAAWRsEPAAAAWBgFPwAAAGBhFPwAAACAhVHwAwAAABZGwQ8AAABYGAU/AAAAYGEU/AAAAICFUfADAAAAFkbBDwAAAFjY/wfznNHMialmyAAAAABJRU5ErkJggg==",
-            "text/plain": [
-              "<Figure size 900x400 with 1 Axes>"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        }
-      ],
-      "source": [
-        "# Visualize text-length distributions by class\n",
-        "df['text_len'] = df['text'].str.len()\n",
-        "ax = df[df['label'] == 0]['text_len'].hist(bins=40, alpha=0.6, label='Human', figsize=(9, 4))\n",
-        "df[df['label'] == 1]['text_len'].hist(bins=40, alpha=0.6, label='AI', ax=ax)\n",
-        "ax.set_title('Text Length Distribution')\n",
-        "ax.set_xlabel('Characters')\n",
-        "ax.set_ylabel('Count')\n",
-        "ax.legend()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 6,
-      "id": "59fe88ce",
-      "metadata": {
-        "id": "59fe88ce"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/html": [
-              "<div>\n",
-              "<style scoped>\n",
-              "    .dataframe tbody tr th:only-of-type {\n",
-              "        vertical-align: middle;\n",
-              "    }\n",
-              "\n",
-              "    .dataframe tbody tr th {\n",
-              "        vertical-align: top;\n",
-              "    }\n",
-              "\n",
-              "    .dataframe thead th {\n",
-              "        text-align: right;\n",
-              "    }\n",
-              "</style>\n",
-              "<table border=\"1\" class=\"dataframe\">\n",
-              "  <thead>\n",
-              "    <tr style=\"text-align: right;\">\n",
-              "      <th></th>\n",
-              "      <th>text</th>\n",
-              "      <th>label</th>\n",
-              "    </tr>\n",
-              "  </thead>\n",
-              "  <tbody>\n",
-              "    <tr>\n",
-              "      <th>0</th>\n",
-              "      <td>हामीले पार्टी एकतापछि कि दुबै पार्टीको सिद्धान...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>1</th>\n",
-              "      <td>एमाले प्रतिनिधिसभाको प्रत्यक्षतर्फ ८० समानुपात...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>2</th>\n",
-              "      <td>नेकपा माओवादी केन्द्रका नेता रामनारायण विडारील...</td>\n",
-              "      <td>1</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>3</th>\n",
-              "      <td>प्रदेश नं २ का मुख्यमन्त्रीको रूपमा संघीय समाज...</td>\n",
-              "      <td>1</td>\n",
-              "    </tr>\n",
-              "    <tr>\n",
-              "      <th>4</th>\n",
-              "      <td>बिहीबार एमालेका अध्यक्ष केपी शर्मा ओली र माओवा...</td>\n",
-              "      <td>0</td>\n",
-              "    </tr>\n",
-              "  </tbody>\n",
-              "</table>\n",
-              "</div>"
-            ],
-            "text/plain": [
-              "                                                text  label\n",
-              "0  हामीले पार्टी एकतापछि कि दुबै पार्टीको सिद्धान...      0\n",
-              "1  एमाले प्रतिनिधिसभाको प्रत्यक्षतर्फ ८० समानुपात...      0\n",
-              "2  नेकपा माओवादी केन्द्रका नेता रामनारायण विडारील...      1\n",
-              "3  प्रदेश नं २ का मुख्यमन्त्रीको रूपमा संघीय समाज...      1\n",
-              "4  बिहीबार एमालेका अध्यक्ष केपी शर्मा ओली र माओवा...      0"
-            ]
-          },
-          "execution_count": 6,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "# Keep only columns needed for training\n",
-        "df = df[['text', 'label']].copy()\n",
-        "df.head()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 7,
-      "id": "434df9a2",
-      "metadata": {
-        "id": "434df9a2"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Using model: distilbert-base-multilingual-cased\n"
-          ]
-        }
-      ],
-      "source": [
-        "# Model/tokenizer config (smaller multilingual model for low-VRAM GPU)\n",
-        "MODEL_NAME = 'distilbert-base-multilingual-cased'\n",
-        "MAX_LEN = 96\n",
-        "\n",
-        "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
-        "print('Using model:', MODEL_NAME)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 8,
-      "id": "ef7d53f9",
-      "metadata": {
-        "id": "ef7d53f9"
-      },
-      "outputs": [],
-      "source": [
-        "class NepaliDataset(Dataset):\n",
-        "    def __init__(self, texts, labels):\n",
-        "        self.texts = texts\n",
-        "        self.labels = labels\n",
-        "\n",
-        "    def __len__(self):\n",
-        "        return len(self.texts)\n",
-        "\n",
-        "    def __getitem__(self, idx):\n",
-        "        return {\n",
-        "            'text': self.texts[idx],\n",
-        "            'label': int(self.labels[idx]),\n",
-        "        }"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 9,
-      "id": "134a3fc1",
-      "metadata": {
-        "id": "134a3fc1"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Train: 1588 | Val: 398\n"
-          ]
-        }
-      ],
-      "source": [
-        "# Train/Validation Split\n",
-        "train_texts, val_texts, train_labels, val_labels = train_test_split(\n",
-        "    df['text'].tolist(),\n",
-        "    df['label'].tolist(),\n",
-        "    test_size=0.2,\n",
-        "    random_state=42,\n",
-        "    stratify=df['label'].tolist(),\n",
-        ")\n",
-        "print(f'Train: {len(train_texts)} | Val: {len(val_texts)}')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 10,
-      "id": "dd226ed1",
-      "metadata": {
-        "id": "dd226ed1"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch size: 2 | Max length: 96\n"
-          ]
-        }
-      ],
-      "source": [
-        "train_dataset = NepaliDataset(train_texts, train_labels)\n",
-        "val_dataset = NepaliDataset(val_texts, val_labels)\n",
-        "\n",
-        "def collate_batch(batch):\n",
-        "    texts = [item['text'] for item in batch]\n",
-        "    labels = torch.tensor([item['label'] for item in batch], dtype=torch.long)\n",
-        "    enc = tokenizer(\n",
-        "        texts,\n",
-        "        padding=True,\n",
-        "        truncation=True,\n",
-        "        max_length=MAX_LEN,\n",
-        "        return_tensors='pt',\n",
-        "    )\n",
-        "    return {\n",
-        "        'input_ids': enc['input_ids'],\n",
-        "        'attention_mask': enc['attention_mask'],\n",
-        "        'labels': labels,\n",
-        "    }\n",
-        "\n",
-        "BATCH_SIZE = 2\n",
-        "train_loader = DataLoader(\n",
-        "    train_dataset,\n",
-        "    batch_size=BATCH_SIZE,\n",
-        "    shuffle=True,\n",
-        "    collate_fn=collate_batch,\n",
-        "    pin_memory=(torch.cuda.is_available()),\n",
-        ")\n",
-        "val_loader = DataLoader(\n",
-        "    val_dataset,\n",
-        "    batch_size=BATCH_SIZE,\n",
-        "    shuffle=False,\n",
-        "    collate_fn=collate_batch,\n",
-        "    pin_memory=(torch.cuda.is_available()),\n",
-        ")\n",
-        "print('Batch size:', BATCH_SIZE, '| Max length:', MAX_LEN)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 11,
-      "id": "51320951",
-      "metadata": {
-        "id": "51320951"
-      },
-      "outputs": [],
-      "source": [
-        "# === Model ===\n",
-        "class IndicBERTClassifier(nn.Module):\n",
-        "    def __init__(self, dropout=0.2):\n",
-        "        super(IndicBERTClassifier, self).__init__()\n",
-        "        self.bert = AutoModel.from_pretrained(MODEL_NAME)\n",
-        "        if hasattr(self.bert, 'gradient_checkpointing_enable'):\n",
-        "            self.bert.gradient_checkpointing_enable()\n",
-        "        self.dropout = nn.Dropout(dropout)\n",
-        "        self.classifier = nn.Linear(self.bert.config.hidden_size, 2)\n",
-        "\n",
-        "    def forward(self, input_ids, attention_mask):\n",
-        "        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)\n",
-        "        cls_output = outputs.last_hidden_state[:, 0, :]\n",
-        "        cls_output = self.dropout(cls_output)\n",
-        "        return self.classifier(cls_output)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 12,
-      "id": "944f918e",
-      "metadata": {
-        "id": "944f918e"
-      },
-      "outputs": [],
-      "source": [
-        "# Step 8: Create a custom Dataset class\n",
-        "class NepaliTextDataset(Dataset):\n",
-        "    def __init__(self, input_ids, attention_mask, labels):\n",
-        "        self.input_ids = input_ids\n",
-        "        self.attention_mask = attention_mask\n",
-        "        self.labels = labels\n",
-        "\n",
-        "    def __len__(self):\n",
-        "        return len(self.labels)\n",
-        "\n",
-        "    def __getitem__(self, idx):\n",
-        "        return {\n",
-        "            'input_ids': torch.tensor(self.input_ids[idx]),\n",
-        "            'attention_mask': torch.tensor(self.attention_mask[idx]),\n",
-        "            'labels': torch.tensor(self.labels[idx])\n",
-        "        }"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 13,
-      "id": "a9d426e1",
-      "metadata": {
-        "id": "a9d426e1"
-      },
-      "outputs": [
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "Loading weights: 100%|██████████| 100/100 [00:00<00:00, 11666.08it/s]\n",
-            "\u001b[1mDistilBertModel LOAD REPORT\u001b[0m from: distilbert-base-multilingual-cased\n",
-            "Key                     | Status     |  | \n",
-            "------------------------+------------+--+-\n",
-            "vocab_layer_norm.bias   | UNEXPECTED |  | \n",
-            "vocab_transform.weight  | UNEXPECTED |  | \n",
-            "vocab_layer_norm.weight | UNEXPECTED |  | \n",
-            "vocab_transform.bias    | UNEXPECTED |  | \n",
-            "vocab_projector.bias    | UNEXPECTED |  | \n",
-            "\n",
-            "\u001b[3mNotes:\n",
-            "- UNEXPECTED\u001b[3m\t:can be ignored when loading from different task/architecture; not ok if you expect identical arch.\u001b[0m\n"
-          ]
-        }
-      ],
-      "source": [
-        "\n",
-        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
-        "model = IndicBERTClassifier().to(device)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 14,
-      "id": "2740c14a",
-      "metadata": {
-        "id": "2740c14a"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Grad accumulation steps: 4\n"
-          ]
-        }
-      ],
-      "source": [
-        "# === Optimizer, Scheduler & Loss ===\n",
-        "optimizer = AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)\n",
-        "loss_fn = nn.CrossEntropyLoss()\n",
-        "\n",
-        "max_epochs = 6\n",
-        "grad_accum_steps = 4  # effective batch = BATCH_SIZE * grad_accum_steps\n",
-        "steps_per_epoch = math.ceil(len(train_loader) / grad_accum_steps)\n",
-        "total_steps = steps_per_epoch * max_epochs\n",
-        "warmup_steps = int(0.1 * total_steps)\n",
-        "scheduler = get_linear_schedule_with_warmup(\n",
-        "    optimizer,\n",
-        "    num_warmup_steps=warmup_steps,\n",
-        "    num_training_steps=total_steps,\n",
-        ")\n",
-        "print('Grad accumulation steps:', grad_accum_steps)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 15,
-      "id": "14ce04bd",
-      "metadata": {
-        "id": "14ce04bd"
-      },
-      "outputs": [],
-      "source": [
-        "# === Training Loop ===\n",
-        "def train(model, loader):\n",
-        "    model.train()\n",
-        "    total_loss = 0\n",
-        "    for batch in loader:\n",
-        "        input_ids = batch['input_ids'].to(device)\n",
-        "        attention_mask = batch['attention_mask'].to(device)\n",
-        "        labels = batch['labels'].to(device)\n",
-        "\n",
-        "        optimizer.zero_grad()\n",
-        "        outputs = model(input_ids, attention_mask)\n",
-        "        loss = loss_fn(outputs, labels)\n",
-        "        loss.backward()\n",
-        "        optimizer.step()\n",
-        "        total_loss += loss.item()\n",
-        "    return total_loss / len(loader)\n",
-        "\n",
-        "# === Evaluation ===\n",
-        "def evaluate(model, loader):\n",
-        "    model.eval()\n",
-        "    preds, true = [], []\n",
-        "    with torch.no_grad():\n",
-        "        for batch in loader:\n",
-        "            input_ids = batch['input_ids'].to(device)\n",
-        "            attention_mask = batch['attention_mask'].to(device)\n",
-        "            labels = batch['labels'].to(device)\n",
-        "\n",
-        "            outputs = model(input_ids, attention_mask)\n",
-        "            pred_labels = torch.argmax(outputs, dim=1)\n",
-        "            preds.extend(pred_labels.cpu().numpy())\n",
-        "            true.extend(labels.cpu().numpy())\n",
-        "\n",
-        "    print(classification_report(true, preds, target_names=[\"Human\", \"AI\"]))\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "d24e91b7",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "d24e91b7",
-        "outputId": "33ef8227-5c71-4c0d-88e7-b1a9e30b45f4"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "Epoch 1/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:4: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
-            "  scaler = GradScaler(enabled=use_amp)\n",
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 0.8206\n",
-            "Batch 50 | Loss: 0.8677\n",
-            "Batch 100 | Loss: 0.8435\n",
-            "Batch 150 | Loss: 0.6523\n",
-            "Batch 200 | Loss: 0.7219\n",
-            "Batch 250 | Loss: 0.5793\n",
-            "Batch 300 | Loss: 0.6833\n",
-            "Batch 350 | Loss: 0.5742\n",
-            "Batch 400 | Loss: 0.4844\n",
-            "Batch 450 | Loss: 0.5671\n",
-            "Batch 500 | Loss: 0.5363\n",
-            "Batch 550 | Loss: 0.5386\n",
-            "Batch 600 | Loss: 0.5520\n",
-            "Batch 650 | Loss: 0.7692\n",
-            "Batch 700 | Loss: 0.4680\n",
-            "Batch 750 | Loss: 0.6353\n",
-            "Train      | Loss: 0.6600 | Acc: 0.5913 | F1: 0.5895\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Validation | Loss: 0.5192 | Acc: 0.8015 | F1: 0.7812\n",
-            "              precision    recall  f1-score   support\n",
-            "\n",
-            "       Human       0.75      0.90      0.82       198\n",
-            "          AI       0.88      0.70      0.78       200\n",
-            "\n",
-            "    accuracy                           0.80       398\n",
-            "   macro avg       0.81      0.80      0.80       398\n",
-            "weighted avg       0.81      0.80      0.80       398\n",
-            "\n",
-            "Saved improved checkpoint: model_best.pth\n",
-            "\n",
-            "Epoch 2/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 0.6078\n",
-            "Batch 50 | Loss: 1.1135\n",
-            "Batch 100 | Loss: 0.3297\n",
-            "Batch 150 | Loss: 0.8473\n",
-            "Batch 200 | Loss: 0.9326\n",
-            "Batch 250 | Loss: 0.5112\n",
-            "Batch 300 | Loss: 0.1645\n",
-            "Batch 350 | Loss: 0.2250\n",
-            "Batch 400 | Loss: 0.7142\n",
-            "Batch 450 | Loss: 0.3741\n",
-            "Batch 500 | Loss: 0.3084\n",
-            "Batch 550 | Loss: 0.1472\n",
-            "Batch 600 | Loss: 0.0679\n",
-            "Batch 650 | Loss: 0.1234\n",
-            "Batch 700 | Loss: 1.1370\n",
-            "Batch 750 | Loss: 0.8843\n",
-            "Train      | Loss: 0.4817 | Acc: 0.7720 | F1: 0.7665\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Validation | Loss: 0.3708 | Acc: 0.8417 | F1: 0.8225\n",
-            "              precision    recall  f1-score   support\n",
-            "\n",
-            "       Human       0.78      0.95      0.86       198\n",
-            "          AI       0.94      0.73      0.82       200\n",
-            "\n",
-            "    accuracy                           0.84       398\n",
-            "   macro avg       0.86      0.84      0.84       398\n",
-            "weighted avg       0.86      0.84      0.84       398\n",
-            "\n",
-            "Saved improved checkpoint: model_best.pth\n",
-            "\n",
-            "Epoch 3/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 0.0415\n",
-            "Batch 50 | Loss: 0.0845\n",
-            "Batch 100 | Loss: 0.0336\n",
-            "Batch 150 | Loss: 0.6389\n",
-            "Batch 200 | Loss: 1.6021\n",
-            "Batch 250 | Loss: 0.0696\n",
-            "Batch 300 | Loss: 0.5184\n",
-            "Batch 350 | Loss: 0.0569\n",
-            "Batch 400 | Loss: 0.8119\n",
-            "Batch 450 | Loss: 1.5121\n",
-            "Batch 500 | Loss: 0.0330\n",
-            "Batch 550 | Loss: 0.0208\n",
-            "Batch 600 | Loss: 1.1329\n",
-            "Batch 650 | Loss: 0.7745\n",
-            "Batch 700 | Loss: 0.0740\n",
-            "Batch 750 | Loss: 1.4907\n",
-            "Train      | Loss: 0.3830 | Acc: 0.8495 | F1: 0.8488\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Validation | Loss: 0.3527 | Acc: 0.8668 | F1: 0.8515\n",
-            "              precision    recall  f1-score   support\n",
-            "\n",
-            "       Human       0.80      0.97      0.88       198\n",
-            "          AI       0.97      0.76      0.85       200\n",
-            "\n",
-            "    accuracy                           0.87       398\n",
-            "   macro avg       0.88      0.87      0.87       398\n",
-            "weighted avg       0.88      0.87      0.87       398\n",
-            "\n",
-            "Saved improved checkpoint: model_best.pth\n",
-            "\n",
-            "Epoch 4/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 1.2321\n",
-            "Batch 50 | Loss: 0.0369\n",
-            "Batch 100 | Loss: 0.0161\n",
-            "Batch 150 | Loss: 0.2000\n",
-            "Batch 200 | Loss: 0.0035\n",
-            "Batch 250 | Loss: 2.3207\n",
-            "Batch 300 | Loss: 0.0022\n",
-            "Batch 350 | Loss: 2.2738\n",
-            "Batch 400 | Loss: 0.0011\n",
-            "Batch 450 | Loss: 0.0075\n",
-            "Batch 500 | Loss: 2.4454\n",
-            "Batch 550 | Loss: 0.3863\n",
-            "Batch 600 | Loss: 0.0038\n",
-            "Batch 650 | Loss: 0.0061\n",
-            "Batch 700 | Loss: 0.0005\n",
-            "Batch 750 | Loss: 0.0182\n",
-            "Train      | Loss: 0.4209 | Acc: 0.8923 | F1: 0.8903\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Validation | Loss: 0.4601 | Acc: 0.8769 | F1: 0.8831\n",
-            "              precision    recall  f1-score   support\n",
-            "\n",
-            "       Human       0.92      0.83      0.87       198\n",
-            "          AI       0.84      0.93      0.88       200\n",
-            "\n",
-            "    accuracy                           0.88       398\n",
-            "   macro avg       0.88      0.88      0.88       398\n",
-            "weighted avg       0.88      0.88      0.88       398\n",
-            "\n",
-            "Saved improved checkpoint: model_best.pth\n",
-            "\n",
-            "Epoch 5/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 0.0010\n",
-            "Batch 50 | Loss: 0.0061\n",
-            "Batch 100 | Loss: 0.0047\n",
-            "Batch 150 | Loss: 0.0201\n",
-            "Batch 200 | Loss: 0.0023\n",
-            "Batch 250 | Loss: 0.0395\n",
-            "Batch 300 | Loss: 0.0011\n",
-            "Batch 350 | Loss: 0.0002\n",
-            "Batch 400 | Loss: 3.2169\n",
-            "Batch 450 | Loss: 4.4883\n",
-            "Batch 500 | Loss: 0.0002\n",
-            "Batch 550 | Loss: 0.0003\n",
-            "Batch 600 | Loss: 0.0000\n",
-            "Batch 650 | Loss: 0.0002\n",
-            "Batch 700 | Loss: 0.0000\n",
-            "Batch 750 | Loss: 4.6367\n",
-            "Train      | Loss: 0.5447 | Acc: 0.9011 | F1: 0.8990\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Validation | Loss: 0.5331 | Acc: 0.9271 | F1: 0.9266\n",
-            "              precision    recall  f1-score   support\n",
-            "\n",
-            "       Human       0.92      0.94      0.93       198\n",
-            "          AI       0.94      0.92      0.93       200\n",
-            "\n",
-            "    accuracy                           0.93       398\n",
-            "   macro avg       0.93      0.93      0.93       398\n",
-            "weighted avg       0.93      0.93      0.93       398\n",
-            "\n",
-            "Saved improved checkpoint: model_best.pth\n",
-            "\n",
-            "Epoch 6/6\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/tmp/ipykernel_155548/4183901742.py:17: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
-            "  with autocast(enabled=use_amp):\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Batch 0 | Loss: 0.0000\n"
-          ]
-        }
-      ],
-      "source": [
-        "from torch.cuda.amp import autocast, GradScaler\n",
-        "\n",
-        "use_amp = device.type == 'cuda'\n",
-        "scaler = GradScaler(enabled=use_amp)\n",
-        "\n",
-        "def train_one_epoch(model, loader):\n",
-        "    model.train()\n",
-        "    total_loss = 0.0\n",
-        "    all_preds, all_true = [], []\n",
-        "\n",
-        "    optimizer.zero_grad(set_to_none=True)\n",
-        "    for batch_idx, batch in enumerate(loader):\n",
-        "        input_ids = batch['input_ids'].to(device, non_blocking=True)\n",
-        "        attention_mask = batch['attention_mask'].to(device, non_blocking=True)\n",
-        "        labels = batch['labels'].to(device, non_blocking=True)\n",
-        "\n",
-        "        with autocast(enabled=use_amp):\n",
-        "            logits = model(input_ids, attention_mask=attention_mask)\n",
-        "            loss = loss_fn(logits, labels) / grad_accum_steps\n",
-        "\n",
-        "        scaler.scale(loss).backward()\n",
-        "\n",
-        "        if (batch_idx + 1) % grad_accum_steps == 0 or (batch_idx + 1) == len(loader):\n",
-        "            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
-        "            scaler.step(optimizer)\n",
-        "            scaler.update()\n",
-        "            scheduler.step()\n",
-        "            optimizer.zero_grad(set_to_none=True)\n",
-        "\n",
-        "        total_loss += loss.item() * grad_accum_steps\n",
-        "        preds = torch.argmax(logits, dim=1)\n",
-        "        all_preds.extend(preds.detach().cpu().numpy())\n",
-        "        all_true.extend(labels.detach().cpu().numpy())\n",
-        "\n",
-        "        if batch_idx % 50 == 0:\n",
-        "            print(f'Batch {batch_idx} | Loss: {(loss.item() * grad_accum_steps):.4f}')\n",
-        "\n",
-        "    avg_loss = total_loss / max(len(loader), 1)\n",
-        "    train_acc = accuracy_score(all_true, all_preds)\n",
-        "    train_f1 = f1_score(all_true, all_preds)\n",
-        "    return avg_loss, train_acc, train_f1\n",
-        "\n",
-        "\n",
-        "def evaluate(model, loader):\n",
-        "    model.eval()\n",
-        "    all_preds, all_true = [], []\n",
-        "    total_loss = 0.0\n",
-        "\n",
-        "    with torch.no_grad():\n",
-        "        for batch in loader:\n",
-        "            input_ids = batch['input_ids'].to(device, non_blocking=True)\n",
-        "            attention_mask = batch['attention_mask'].to(device, non_blocking=True)\n",
-        "            labels = batch['labels'].to(device, non_blocking=True)\n",
-        "\n",
-        "            with autocast(enabled=use_amp):\n",
-        "                logits = model(input_ids, attention_mask=attention_mask)\n",
-        "                loss = loss_fn(logits, labels)\n",
-        "\n",
-        "            total_loss += loss.item()\n",
-        "            preds = torch.argmax(logits, dim=1)\n",
-        "            all_preds.extend(preds.cpu().numpy())\n",
-        "            all_true.extend(labels.cpu().numpy())\n",
-        "\n",
-        "    val_loss = total_loss / max(len(loader), 1)\n",
-        "    val_acc = accuracy_score(all_true, all_preds)\n",
-        "    val_f1 = f1_score(all_true, all_preds)\n",
-        "\n",
-        "    print(f'Validation | Loss: {val_loss:.4f} | Acc: {val_acc:.4f} | F1: {val_f1:.4f}')\n",
-        "    print(classification_report(all_true, all_preds, target_names=['Human', 'AI']))\n",
-        "    return val_loss, val_acc, val_f1\n",
-        "\n",
-        "\n",
-        "# Training with early stopping on validation F1\n",
-        "patience = 2\n",
-        "best_val_f1 = 0.0\n",
-        "epochs_without_improve = 0\n",
-        "\n",
-        "for epoch in range(1, max_epochs + 1):\n",
-        "    print(f'\\nEpoch {epoch}/{max_epochs}')\n",
-        "    if device.type == 'cuda':\n",
-        "        torch.cuda.empty_cache()\n",
-        "\n",
-        "    train_loss, train_acc, train_f1 = train_one_epoch(model, train_loader)\n",
-        "    print(f'Train      | Loss: {train_loss:.4f} | Acc: {train_acc:.4f} | F1: {train_f1:.4f}')\n",
-        "\n",
-        "    val_loss, val_acc, val_f1 = evaluate(model, val_loader)\n",
-        "\n",
-        "    if val_f1 > best_val_f1:\n",
-        "        best_val_f1 = val_f1\n",
-        "        epochs_without_improve = 0\n",
-        "        torch.save(model.state_dict(), 'model_best.pth')\n",
-        "        print('Saved improved checkpoint: model_best.pth')\n",
-        "    else:\n",
-        "        epochs_without_improve += 1\n",
-        "        if epochs_without_improve >= patience:\n",
-        "            print('Early stopping triggered.')\n",
-        "            break\n",
-        "\n",
-        "print(f'Best validation F1: {best_val_f1:.4f}')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "wBIT-kPaswqy",
-      "metadata": {
-        "id": "wBIT-kPaswqy"
-      },
-      "outputs": [],
-      "source": [
-        "# Optional: save current in-memory weights as latest checkpoint\n",
-        "torch.save(model.state_dict(), 'model_latest.pth')\n",
-        "print('Saved: model_latest.pth')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "19b9652c",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "19b9652c",
-        "outputId": "e1b12835-b081-4d46-a909-c92cb3b6d230"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "('./nepali_xlmr_classifier/tokenizer_config.json',\n",
-              " './nepali_xlmr_classifier/special_tokens_map.json',\n",
-              " './nepali_xlmr_classifier/sentencepiece.bpe.model',\n",
-              " './nepali_xlmr_classifier/added_tokens.json',\n",
-              " './nepali_xlmr_classifier/tokenizer.json')"
-            ]
-          },
-          "execution_count": 41,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "tokenizer.save_pretrained(\"./nepali_xlmr_classifier\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "eAnrw316iRw8",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "eAnrw316iRw8",
-        "outputId": "04885bb5-4f06-459b-a83c-40f5e00703fe"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "0\n"
-          ]
-        }
-      ],
-      "source": [
-        "def predict(text):\n",
-        "    model.eval()\n",
-        "    inputs = tokenizer(\n",
-        "        text,\n",
-        "        return_tensors='pt',\n",
-        "        truncation=True,\n",
-        "        padding=True,\n",
-        "        max_length=MAX_LEN,\n",
-        "    )\n",
-        "    inputs = {k: v.to(device) for k, v in inputs.items()}\n",
-        "\n",
-        "    with torch.no_grad():\n",
-        "        logits = model(inputs['input_ids'], inputs['attention_mask'])\n",
-        "        probs = torch.softmax(logits, dim=1)\n",
-        "        pred = torch.argmax(probs, dim=1).item()\n",
-        "        confidence = probs[0, pred].item()\n",
-        "\n",
-        "    label = 'AI' if pred == 1 else 'Human'\n",
-        "    return label, confidence\n",
-        "\n",
-        "sample = 'अख्तियार दुरुपयोग अनुसन्धान आयोगले सिन्धुपाल्चोक–२ बाट प्रतिनिधिसभा सदस्य निर्वाचित सांसद तथा पूर्वमन्त्री बस्नेतसहित १६ जना र २ कम्पनी विरुद्ध ३ अर्ब २१ करोडभन्दा बढी बिगो कायम गरी बिहीबार विशेष अदालतमा भ्रष्टाचार मुद्दा दायर गरेको छ ।'\n",
-        "label, conf = predict(sample)\n",
-        "print(f'Prediction: {label} | Confidence: {conf:.4f}')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "lqGrqG51NiQV",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "lqGrqG51NiQV",
-        "outputId": "6bdae59b-2684-4bd0-f804-d16ebd8272db"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "1\n",
-            "0\n"
-          ]
-        }
-      ],
-      "source": [
-        "print(predict(\"इन्टरनेटको सुरुवात सन् १९६९ मा अमेरिकी रक्षा मन्त्रालयले निर्माण गरेको ARPANET नामक प्रोजेक्टबाट भएको हो, जसको उद्देश्य आपसी संचारलाई सहज बनाउने थियो र जसले भविष्यमा इन्टरनेटको रूप लियो\"))\n",
-        "\n",
-        "print(predict(\"सुरुमा इन्टरनेट केही वैज्ञानिक तथा सरकारी संस्थाहरूमा सीमित रहेको भए पनि, समयक्रममा यसको पहुँच आम नागरिक, विद्यालय, र व्यवसायिक क्षेत्रमा विस्तार हुँदै गयो\"))\n",
-        "\n",
-        "print(predict(\"ARPANETले कम्प्युटरहरूलाई आपसमा जोड्ने सफल प्रयोग गरेपछि इन्टरनेटको सम्भावना प्रमाणित भयो, जसले गर्दा विश्वभरका अनुसन्धानकर्ताहरू यसप्रति आकर्षित हुन थाले\"))\n",
-        "\n",
-        "print(predict(\"सन् १९९० को दशकमा विश्वव्यापी रूपमा इन्टरनेट विस्तार हुन थालेपछि मानिसहरू सूचनाको आदान–प्रदान, इमेल, र वेबसाइटहरूको प्रयोगमार्फत डिजिटल संसारमा प्रवेश गर्न थाले।\"))\n",
-        "\n",
-        "print(predict(\"इन्टरनेटले शिक्षा, स्वास्थ्य, सञ्चार, मनोरञ्जन, तथा व्यापारजस्ता धेरै क्षेत्रहरूमा अभूतपूर्व परिवर्तन ल्याएको छ, जसले गर्दा मानव जीवन सरल, छरितो र प्रभावकारी बनेको छ।\"))\n",
-        "\n",
-        "print(predict(\"समयसँगै इन्टरनेट एक अत्यावश्यक सेवाको रूपमा विकास भएको छ, जसबिनाको आधुनिक जीवन लगभग असम्भवजस्तै लाग्ने अवस्था सिर्जना भएको छ।\"))\n",
-        "\n",
-        "print(predict(\"आजको युगमा इन्टरनेट केवल सूचना प्राप्तिको माध्यम मात्र नभई ज्ञानको भण्डार, रचनात्मकता प्रदर्शन गर्ने मंच, तथा रोजगार सृजनाको स्रोत पनि बनिसकेको छ।\"))\n",
-        "\n",
-        "print(predict(\"इन्टरनेटको प्रभाव त्यति गहिरो भएको छ कि विद्यालयका बालबालिकादेखि वृद्धसम्म यसको प्रयोगमा संलग्न छन्, जसले डिजिटल विभाजनको अवधारणा जन्माएको छ।\"))\n",
-        "\n",
-        "print(predict(\"इन्टरनेटले विश्वलाई एउटा सानो गाउँमा रूपान्तरण गरेको छ, जहाँ मानिसहरू हजारौं किलोमिटर टाढा भएर पनि एकअर्कासँग प्रत्यक्ष संवाद गर्न सक्छन्।\"))\n",
-        "\n",
-        "print(predict(\"संसदीय समितिले समन्वयकारी भूमिका निर्वाह गर्दै मनसुनजन्य विपद् जोखिम न्यूनीकरण, विपद् प्रतिकार्यका लागि तयारी गर्न तीन तहकै सरकारलाई निर्देशन दिएको छ।\"))\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "X2ePCc5Disrt",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 35
-        },
-        "id": "X2ePCc5Disrt",
-        "outputId": "a4d27689-28cb-43c0-8333-67f2d3a6e097"
-      },
-      "outputs": [
-        {
-          "data": {
-            "application/vnd.google.colaboratory.intrinsic+json": {
-              "type": "string"
-            },
-            "text/plain": [
-              "'/content/classifier.zip'"
-            ]
-          },
-          "execution_count": 42,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "import shutil\n",
-        "\n",
-        "# Replace 'my_folder' with your folder name or path\n",
-        "folder_path = '/content/nepali_xlmr_classifier'\n",
-        "zip_path = '/content/classifier.zip'\n",
-        "\n",
-        "shutil.make_archive(zip_path.replace('.zip', ''), 'zip', folder_path)\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "4BDzVg2gN7xi",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 17
-        },
-        "id": "4BDzVg2gN7xi",
-        "outputId": "ef31798e-24f5-45ad-900f-7528b32ae39f"
-      },
-      "outputs": [
-        {
-          "data": {
-            "application/javascript": "\n    async function download(id, filename, size) {\n      if (!google.colab.kernel.accessAllowed) {\n        return;\n      }\n      const div = document.createElement('div');\n      const label = document.createElement('label');\n      label.textContent = `Downloading \"${filename}\": `;\n      div.appendChild(label);\n      const progress = document.createElement('progress');\n      progress.max = size;\n      div.appendChild(progress);\n      document.body.appendChild(div);\n\n      const buffers = [];\n      let downloaded = 0;\n\n      const channel = await google.colab.kernel.comms.open(id);\n      // Send a message to notify the kernel that we're ready.\n      channel.send({})\n\n      for await (const message of channel.messages) {\n        // Send a message to notify the kernel that we're ready.\n        channel.send({})\n        if (message.buffers) {\n          for (const buffer of message.buffers) {\n            buffers.push(buffer);\n            downloaded += buffer.byteLength;\n            progress.value = downloaded;\n          }\n        }\n      }\n      const blob = new Blob(buffers, {type: 'application/binary'});\n      const a = document.createElement('a');\n      a.href = window.URL.createObjectURL(blob);\n      a.download = filename;\n      div.appendChild(a);\n      a.click();\n      div.remove();\n    }\n  ",
-            "text/plain": [
-              "<IPython.core.display.Javascript object>"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        },
-        {
-          "data": {
-            "application/javascript": "download(\"download_33034c8f-76d5-48d0-b7cd-3d066ac8e32f\", \"classifier.zip\", 6596694)",
-            "text/plain": [
-              "<IPython.core.display.Javascript object>"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        }
-      ],
-      "source": [
-        "from google.colab import files\n",
-        "\n",
-        "files.download(zip_path)\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "2jJkcOlw_R1k",
-      "metadata": {
-        "id": "2jJkcOlw_R1k"
-      },
-      "outputs": [],
-      "source": [
-        "torch.save(model.state_dict(), \"final_model.pth\")  # AFTER training with classification head\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "xnHr1IDABebZ",
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "xnHr1IDABebZ",
-        "outputId": "95761a2d-56fa-418c-de03-d66d1ae662ee"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "The text is predicted to be: Human\n",
-            "1\n",
-            "0\n",
-            "1\n"
-          ]
-        }
-      ],
-      "source": [
-        "# prompt: How to load the model and classifier and use it ? if no other code is in top of this\n",
-        "\n",
-        "# Define the device\n",
-        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
-        "\n",
-        "# Instantiate the model\n",
-        "model = IndicBERTClassifier().to(device)\n",
-        "\n",
-        "# Load the saved state dictionary\n",
-        "# Make sure the path to your saved model file is correct\n",
-        "model_path = \"final_model.pth\" # Or \"model_95_acc.pth\" if you saved that one last\n",
-        "model.load_state_dict(torch.load(model_path, map_location=device))\n",
-        "\n",
-        "# Set the model to evaluation mode\n",
-        "model.eval()\n",
-        "\n",
-        "# Load the tokenizer\n",
-        "tokenizer_path = \"./nepali_xlmr_classifier\" # Make sure this path is correct\n",
-        "tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)\n",
-        "\n",
-        "# Now the model and tokenizer are loaded and ready to be used for predictions.\n",
-        "# You can use the existing `predict` function or write a new one.\n",
-        "\n",
-        "# Example of using the predict function with the loaded model and tokenizer\n",
-        "def predict(text):\n",
-        "    model.eval() # Ensure model is in evaluation mode\n",
-        "    inputs = tokenizer(text, return_tensors='pt', truncation=True, padding=True, max_length=512)\n",
-        "    inputs = {k: v.to(device) for k, v in inputs.items()}\n",
-        "    with torch.no_grad():\n",
-        "        outputs = model(**inputs)\n",
-        "\n",
-        "        # Handle if output is tensor (some versions/models return logits directly)\n",
-        "        logits = outputs if isinstance(outputs, torch.Tensor) else outputs.logits\n",
-        "\n",
-        "        pred = torch.argmax(logits, dim=1).item()\n",
-        "    return pred\n",
-        "\n",
-        "# Example usage with some text\n",
-        "text_to_predict = \"This is a test sentence.\" # Replace with your Nepali text\n",
-        "predicted_class = predict(text_to_predict)\n",
-        "\n",
-        "# Interpret the prediction (assuming 0 for Human, 1 for AI based on your previous code)\n",
-        "class_label = \"Human\" if predicted_class == 0 else \"AI\"\n",
-        "print(f\"The text is predicted to be: {class_label}\")\n",
-        "\n",
-        "# You can test with more examples as you did before\n",
-        "print(predict(\"यी सबै वाक्यहरू इन्टरनेटको विकास, प्रभाव, र चुनौतीहरूको गहिरो सन्दर्भ समेटेर तयार पारिएका छन्। यदि तिमीलाई चाहिएको खण्डमा विशेष विषय (जस्तै शिक्षा, साइबर सुरक्षा, ग्रामीण प्रभाव आदि) चाहिएको हो भने, म त्यही विषयमा केन्द्रित लामो वाक्यहरू पनि दिन सक्छु।\"))\n",
-        "print(predict(\"अख्तियार दुरुपयोग अनुसन्धान आयोगले सिन्धुपाल्चोक–२ बाट प्रतिनिधिसभा सदस्य निर्वाचित सांसद तथा पूर्वमन्त्री बस्नेतसहित १६ जना र २ कम्पनी विरुद्ध ३ अर्ब २१ करोडभन्दा बढी बिगो कायम गरी बिहीबार विशेष अदालतमा भ्रष्टाचार मुद्दा दायर गरेको छ । योसँगै बस्नेत सांसद पदबाट स्वतः निलम्बनमा परेका छन् ।\"))\n",
-        "print(predict(\"इन्टरनेटको सुरुवात सन् १९६९ मा अमेरिकी रक्षा मन्त्रालयले निर्माण गरेको ARPANET नामक प्रोजेक्टबाट भएको हो, जसको उद्देश्य आपसी संचारलाई सहज बनाउने थियो र जसले भविष्यमा इन्टरनेटको रूप लियो\"))\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "gG8fnbqyDUpm",
-      "metadata": {
-        "id": "gG8fnbqyDUpm"
-      },
-      "outputs": [],
-      "source": []
-    }
-  ],
-  "metadata": {
-    "accelerator": "TPU",
-    "colab": {
-      "gpuType": "V28",
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "ml",
-      "language": "python",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.11.14"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 5
-}

notebook/ai_vs_human_nepali/notebook/documentation.md

@@ -1,435 +0,0 @@
-# Nepali AI vs Human Notebook Documentation
-
-This folder contains a small notebook series for building an AI-vs-human text detector for Nepali text. The notebooks are not identical copies; they represent the evolution of the project from a lightweight scikit-learn baseline to a stronger hybrid model and a transformer-based experiment.
-
-## Notebook Inventory
-
-The notebooks in this directory are:
-
-- [main.ipynb](main.ipynb)
-- [working model.ipynb](working%20model.ipynb)
-- [Nepali_Ai_vs_Human.ipynb](Nepali_Ai_vs_Human.ipynb)
-- [final_main.ipynb](final_main.ipynb)
-
-## Shared Goal
-
-All notebooks solve the same binary classification task:
-
-- Class 0 = Human-written Nepali text
-- Class 1 = AI-generated Nepali text
-
-The notebooks differ in how they prepare the data, which features they extract, and which model family they train.
-
-## Shared Data Sources
-
-Across the notebooks, the dataset is built from one or more CSV files under the notebook dataset folders. The common column pattern is:
-
-- human_text
-- ai_generated_text
-
-Some notebooks also use:
-
-- title
-- label
-- paragraph
-
-The data preparation usually performs some combination of:
-
-- dropping null rows
-- stripping whitespace
-- removing duplicates
-- converting two source columns into one text column plus one label column
-- balancing classes by sampling
-- splitting long texts into smaller chunks
-
-## Notebook Relationship
-
-The notebooks form a progression:
-
-1. main.ipynb is the first lightweight sklearn baseline.
-2. working model.ipynb refines the baseline with better text chunking.
-3. Nepali_Ai_vs_Human.ipynb switches to a transformer-style neural model.
-4. final_main.ipynb is the most complete hybrid notebook and is the closest thing to a production workflow.
-
-## main.ipynb
-
-### Purpose
-
-This is the earliest baseline notebook. It focuses on a CPU-friendly approach using TF-IDF plus hand-crafted text features, then compares several classic machine learning models.
-
-### Data Preparation
-
-The notebook loads several CSV files and concatenates them into one dataframe. The data is drawn from:
-
-- ../DATASET/data.csv
-- ../DATASET/new_data.csv
-- /mnt/linux-data/Work/aiapi/notebook/ai_vs_human_nepali/news_scrap_new2.fixed.csv
-
-The notebook creates separate cleaned columns for human text and AI text, then stacks them into a single training dataframe with labels.
-
-Important preprocessing steps:
-
-- remove URLs
-- keep only Nepali Unicode characters and whitespace
-- lowercase the text
-- remove consecutive repeated words
-
-### Feature Engineering
-
-The notebook combines two feature families:
-
-- Word-level TF-IDF with 1-2 gram features
-- Dense, hand-crafted features based on text structure
-
-The hand-crafted features include:
-
-- burstiness statistics from sentence lengths
-- average word length
-- average sentence length
-- lexical diversity
-- punctuation ratio
-- repeated bigram ratio
-- Devanagari diacritic density
-
-The sparse TF-IDF matrix is concatenated with the dense feature matrix using horizontal stacking.
-
-### Models Trained
-
-The notebook compares several standard classifiers:
-
-- LogisticRegressionCV
-- RidgeClassifierCV
-- MultinomialNB
-- BernoulliNB
-- RandomForestClassifier
-- GradientBoostingClassifier
-- LinearSVC
-- KNeighborsClassifier
-
-Dense conversion is applied only where needed, such as for LinearSVC and KNeighbors.
-
-### Evaluation
-
-The notebook evaluates the models with:
-
-- validation accuracy
-- weighted F1 score
-- classification reports
-- confusion matrices
-- ROC curves
-
-The top models are selected by validation accuracy and re-used in later prediction cells.
-
-### Prediction Demo
-
-The notebook includes several sample Nepali texts for inference. It prints per-model predictions and, where possible, confidence values.
-
-### Saved Artifacts
-
-Each model is saved as a pickle file in a local saved_models directory.
-
-### Known Issues
-
-- Several cells are duplicated, especially the dataset loading cells.
-- The vectorizer and the feature builder are not saved with the models, so full reloading is incomplete.
-- There are repeated prediction sections, which makes the notebook harder to maintain.
-- Some cells appear to be placeholders or empty.
-
-## working model.ipynb
-
-### Purpose
-
-This notebook is a refinement of main.ipynb. It keeps the same overall classifier strategy but improves how long Nepali articles are handled.
-
-### Main Difference From main.ipynb
-
-The key improvement is sentence chunking:
-
-- long texts are split into smaller chunks
-- chunk boundaries prefer Nepali danda punctuation
-- each chunk is limited to a small number of sentences or words
-
-This makes the dataset more granular and helps the classifier train on smaller, more uniform samples.
-
-### Preprocessing
-
-The notebook defines:
-
-- clean_text
-- remove_auto_repeating
-- split_into_sentence_chunks
-- expand_texts_to_chunks
-
-These functions preserve sentence punctuation for chunking, then normalize the cleaned chunks for downstream training.
-
-### Feature Engineering and Models
-
-The rest of the pipeline is essentially the same as main.ipynb:
-
-- TF-IDF word n-grams
-- burstiness and stylometric features
-- concatenated sparse + dense representation
-- the same family of sklearn classifiers
-
-### Evaluation and Inference
-
-The notebook follows the same model comparison, ROC plotting, confusion matrix plotting, and sample prediction pattern as the baseline notebook.
-
-### Saved Artifacts
-
-Like main.ipynb, the fitted sklearn models are stored under saved_models as individual pickle files.
-
-### Known Issues
-
-- The notebook has redundant cells and repeated code blocks.
-- It still does not serialize the vectorizer and feature transformer together with the model artifacts.
-- Some prediction logic is repeated more than once.
-
-## Nepali_Ai_vs_Human.ipynb
-
-### Purpose
-
-This notebook is the deep learning branch of the project. Instead of hand-crafted features plus classical classifiers, it uses a transformer encoder with a classification head.
-
-### Data Preparation
-
-The notebook reads one CSV file and converts the two-column source format into a single text-label dataframe.
-
-Important preparation steps:
-
-- validate required columns
-- drop nulls
-- build a unified dataframe with text and label
-- filter short texts
-- drop duplicate text rows
-- shuffle the dataset
-
-The notebook keeps the raw text mostly intact rather than applying aggressive regex cleaning.
-
-### Model Architecture
-
-The model pipeline is built around Hugging Face transformers and PyTorch:
-
-- tokenizer from a multilingual BERT-style model
-- AutoModel backbone
-- classification head with dropout
-- binary output layer
-
-The notebook defines a custom PyTorch module named IndicBERTClassifier.
-
-### Training Setup
-
-The notebook uses:
-
-- train/validation split with stratification
-- DataLoader-based batching
-- AdamW optimizer
-- cross-entropy loss
-- linear warmup scheduler
-- gradient accumulation
-- mixed precision when CUDA is available
-- early stopping on validation F1
-
-This makes it more GPU-oriented than the sklearn notebooks.
-
-### Evaluation
-
-Per-epoch evaluation includes:
-
-- accuracy
-- F1 score
-- classification report
-
-The notebook also saves improved checkpoints when validation F1 improves.
-
-### Prediction Demo
-
-The notebook defines a predict function that:
-
-- tokenizes the input text
-- runs the transformer model
-- applies softmax
-- returns the predicted class and confidence
-
-Several sample Nepali sentences are passed through the predictor at the end of the notebook.
-
-### Saved Artifacts
-
-The notebook saves:
-
-- model_best.pth
-- model_latest.pth
-- tokenizer files in ./nepali_xlmr_classifier
-
-There is also a Colab-oriented zip export section.
-
-### Known Issues
-
-- The notebook mixes local notebook execution with Colab-specific code.
-- Some cells show CUDA or environment-related warnings.
-- The training flow is more complex and less polished than the final hybrid notebook.
-- Paths are hard-coded in a few places.
-
-## final_main.ipynb
-
-### Purpose
-
-This is the most complete notebook in the folder. It combines semantic embeddings from Sentence Transformers with stylometric features, then trains a linear model and an XGBoost model on the fused feature vector.
-
-### Data Preparation
-
-The notebook reads the dataset from:
-
-- ../DATASET/Final_data/final_news345.csv
-- /mnt/linux-data/Work/aiapi/notebook/ai_vs_human_nepali/Final_data/final_news345.csv
-
-The notebook expects a label column with string values and maps them to binary classes.
-
-It also includes a preprocessing utility that can:
-
-- split very long Nepali texts into chunks
-- preserve danda-based sentence boundaries
-- filter out extremely short chunks
-- balance the dataset by sampling each class to the same count
-
-### Visualization
-
-The notebook includes exploratory plots for:
-
-- class distribution
-- character count distribution
-- word count distribution
-- sentence count distribution
-- cleaned text length distribution
-- stylometric feature comparison plots
-
-This makes it the most documented and inspection-friendly notebook in the folder.
-
-### Text Cleaning
-
-The notebook defines clean_nepali_text, which:
-
-- lowercases the text
-- normalizes Nepali and common Unicode punctuation
-- removes unwanted characters
-- collapses repeated whitespace
-- trims the result
-
-This cleaned text is used for both embeddings and stylometric extraction.
-
-### Stylometric Features
-
-The notebook uses six hand-crafted features:
-
-- word_count
-- sentence_count
-- avg_word_length
-- avg_sentence_length
-- type_token_ratio
-- punctuation_ratio
-
-These features are extracted from the cleaned text and then standardized with StandardScaler.
-
-### Semantic Embeddings
-
-The notebook uses the Sentence Transformers model:
-
-- sentence-transformers/paraphrase-multilingual-mpnet-base-v2
-
-This produces 768-dimensional multilingual sentence embeddings. The notebook loads the embedder on CPU to reduce CUDA memory pressure.
-
-### Feature Fusion
-
-The final feature matrix is built by concatenating:
-
-- 768 embedding dimensions
-- 6 scaled stylometric dimensions
-
-So each sample becomes a 774-dimensional vector.
-
-### Models Trained
-
-Two models are trained on the fused features:
-
-- Logistic Regression
-- XGBoost
-
-XGBoost is configured with class imbalance handling through scale_pos_weight.
-
-### Evaluation
-
-The notebook evaluates both models using:
-
-- accuracy
-- precision
-- recall
-- F1 score
-- confusion matrices
-- ROC curves and AUC
-
-It also computes and visualizes XGBoost feature importance.
-
-### Prediction Flow
-
-The prediction function follows this exact sequence:
-
-1. clean the input
-2. extract stylometric features
-3. build the sentence embedding
-4. scale the stylometric vector
-5. concatenate the two feature blocks
-6. predict with XGBoost
-
-The function returns a dictionary containing the label, numeric class id, and probability.
-
-### Saved Artifacts
-
-The notebook saves a joblib bundle at:
-
-- ../models/ai_text_detector_model.pkl
-
-The saved artifact includes:
-
-- xgb_model
-- lr_model
-- scaler
-- embed_model name string
-- stylo_cols
-- label_map
-
-### Known Issues
-
-- The XGBoost fit call uses the test set as an eval_set, which is acceptable for monitoring but not ideal if you want strict validation separation.
-- The embedding model name is saved, but the embedder itself is not serialized.
-- The notebook is the strongest production candidate, but it still lacks a separate load-and-predict helper for end users.
-
-## Comparison Summary
-
-| Notebook | Main Approach | Strength | Weakness |
-|---|---|---|---|
-| main.ipynb | TF-IDF + stylometry + classic ML | Simple baseline, easy to inspect | Repetitive and not fully serializable |
-| working model.ipynb | TF-IDF + stylometry + chunking | Better handling of long text | Still mostly a baseline notebook |
-| Nepali_Ai_vs_Human.ipynb | Transformer classifier | Strong semantic modeling | Heavier, more environment-sensitive |
-| final_main.ipynb | Sentence embeddings + stylometry + XGBoost | Best balance of performance, clarity, and deployability | Uses a saved model name string instead of serializing the embedder |
-
-## Recommended Reading Order
-
-If you want to understand the project evolution, read the notebooks in this order:
-
-1. main.ipynb
-2. working model.ipynb
-3. Nepali_Ai_vs_Human.ipynb
-4. final_main.ipynb
-
-If you only want the most useful notebook for reuse or deployment, start with final_main.ipynb.
-
-## Practical Notes
-
-- Several notebooks contain duplicated or stale cells from experimentation.
-- Not every cell has been executed successfully.
-- Paths are sometimes hard-coded for the local workspace, so moving the folder may require path cleanup.
-- The project alternates between three styles of modeling: classical sklearn, transformer fine-tuning, and hybrid embedding-based classification.
-
-## Suggested Next Step
-
-If you want, the next useful document to add is an inference guide that explains how to load the saved model bundle from final_main.ipynb and run predictions on new Nepali text.

notebook/ai_vs_human_nepali/topic_scrapper.ipynb

@@ -1,455 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "4b53d4bc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# # Groq Nepali Rewriter\n",
-    "\n",
-    "# This notebook loads the dataset, builds a Nepali rewrite prompt, tests one sample, and then saves a batch output CSV using the Groq API.\n",
-    "\n",
-    "# Requirements:\n",
-    "# - `GROQ_API_KEY` must be available in `.env`\n",
-    "# - the input file must contain a `paragraph` column"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "6c8dc1cb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "True"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "import os\n",
-    "import re\n",
-    "import time\n",
-    "from concurrent.futures import ThreadPoolExecutor, as_completed\n",
-    "\n",
-    "import pandas as pd\n",
-    "from dotenv import load_dotenv\n",
-    "from groq import Groq\n",
-    "\n",
-    "load_dotenv()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "019adfa8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "api_key = os.getenv(\"GROQ_API_KEY2\")\n",
-    "if not api_key:\n",
-    "    raise ValueError(\"GROQ_API_KEY not found in .env or environment.\")\n",
-    "\n",
-    "client = Groq(api_key=api_key)\n",
-    "MODEL_NAME = \"llama-3.3-70b-versatile\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "4b4d2bbe",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data =pd.read_csv(\"DATASET/topics_1000.csv\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "c36cfbbf",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "<div>\n",
-       "<style scoped>\n",
-       "    .dataframe tbody tr th:only-of-type {\n",
-       "        vertical-align: middle;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe tbody tr th {\n",
-       "        vertical-align: top;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe thead th {\n",
-       "        text-align: right;\n",
-       "    }\n",
-       "</style>\n",
-       "<table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       "    <tr style=\"text-align: right;\">\n",
-       "      <th></th>\n",
-       "      <th>id</th>\n",
-       "      <th>topic</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <th>0</th>\n",
-       "      <td>1</td>\n",
-       "      <td>नेपालमा कृत्रिम बुद्धिमत्ता विकासको वर्तमान अव...</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>1</th>\n",
-       "      <td>2</td>\n",
-       "      <td>नेपालको शिक्षा प्रणालीमा डिजिटल प्रविधिको प्रभाव</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>2</th>\n",
-       "      <td>3</td>\n",
-       "      <td>काठमाडौँ उपत्यकाको वायु प्रदूषण समस्या</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>3</th>\n",
-       "      <td>4</td>\n",
-       "      <td>नेपालमा जलवायु परिवर्तनका असरहरू</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>4</th>\n",
-       "      <td>5</td>\n",
-       "      <td>ग्रामीण क्षेत्रमा इन्टरनेट पहुँचको विस्तार</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table>\n",
-       "</div>"
-      ],
-      "text/plain": [
-       "   id                                              topic\n",
-       "0   1  नेपालमा कृत्रिम बुद्धिमत्ता विकासको वर्तमान अव...\n",
-       "1   2   नेपालको शिक्षा प्रणालीमा डिजिटल प्रविधिको प्रभाव\n",
-       "2   3             काठमाडौँ उपत्यकाको वायु प्रदूषण समस्या\n",
-       "3   4                   नेपालमा जलवायु परिवर्तनका असरहरू\n",
-       "4   5         ग्रामीण क्षेत्रमा इन्टरनेट पहुँचको विस्तार"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "data.head()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "b6e226b8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "def build_prompt(paragraph):\n",
-    "    style = [\n",
-    "        \"Use simple and clear language.\",\n",
-    "        \"Make it engaging and interesting to read.\",\n",
-    "        \"Use a conversational tone.\",\n",
-    "        \"Keep the original meaning intact.\",\n",
-    "        \"Avoid complex jargon and technical terms.\",\n",
-    "        \"Use short sentences and paragraphs.\",\n",
-    "        \"Add examples or anecdotes to illustrate points.\",\n",
-    "        \"Use active voice instead of passive voice.\",\n",
-    "        \"Include a call to action or a thought-provoking question at the end.\",\n",
-    "    ]\n",
-    "    selected_style_random_single =  np.random.choice(style, size=len(style), replace=False)  # Select the first 5 style guidelines\n",
-    "    prompt = f\"\"\"\n",
-    "    give me an essay  for the following topics puree nepali ok no enlgish language:\n",
-    "    {paragraph}\n",
-    "    Rewrite the above paragraph in Nepali, following these style guidelines:\n",
-    "    {', '.join(selected_style_random_single)}\n",
-    "    \"\"\"\n",
-    "    return prompt.strip()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "cf16922b",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "नेपालमा कृत्रिम बुद्धिमत्ता विकासको वर्तमान अवस्था\n",
-      "\n",
-      "कृत्रिम बुद्धिमत्ता विकास नेपालको लागि एक नयाँ युग हो । यो प्राविधिक क्षेत्र दिन-प्रतिदिन विकसित हुने क्रममा छ । नेपालमा कृत्रिम बुद्धिमत्ताले विभिन्न क्षेत्रमा परिवर्तन ल्याउने क्षमता राख्दछ । जस्तै: स्वास्थ्य सेवामा, शिक्षामा, वित्तीय सेवामा, तथा उत्पादन क्षेत्रमा ।\n",
-      "\n",
-      "नेपालमा कृत्रिम बुद्धिमत्ताको विकासले नयाँ अवस्था प्राप्त गरिरहेको छ । यो देशमा विभिन्न प्राविधिक कम्पनीहरुले कृत्रिम बुद्धिमत्ताको विकासमा लगनशील छन् । तसर्थ, यसले नेपालमा रोजगारीको अवसर पनि बढाउने छ । उदाहरणको लागि, कृत्रिम बुद्धिमत्ताले स्वास्थ्य सेवामा रोग निदान गर्ने, रोगको उपचार सुझाउने, तथा व्यक्तिको स्वास्थ्य जाँच गर्ने काम गर्नसक्ने छ ।\n",
-      "\n",
-      "कृत्रिम बुद्धिमत्ताको विकासले नेपालको अर्थतन्त्रमा पनि परिवर्तन ल्याउने छ । यसले व्यवसायिक क्षेत्रमा उत्पादनशीलता बढाउने, उत्पादन मुल्य कम गर्ने, तथा गुणस्तर मापन गर्ने काम गर्नसक्ने छ । उदाहरणको लागि, कृत्रिम बुद्धिमत्ताले वित्तीय सेवामा लेनदेनको निरीक्षण गर्ने, धोकाधोकाको मुल्यांकन गर्ने, तथा वित्तीय संस्थाहरुलाई सुझाव दिने काम गर्नसक्ने छ ।\n",
-      "\n",
-      "नेपालमा कृत्रिम बुद्धिमत्ता विकासको वर्तमान अवस्थाले देशलाई एक नयाँ दिशामा लम्बने क्षमता राख्दछ । तर, यसको विकासमा चुनौतिहरु पनि छन् । जस्तै: डाटा सुरक्षा, निजताको हनन, तथा श्रमिकहरुको प्रतिस्पर्धी क्षमता । तसर्थ, नेपालमा कृत्रिम बुद्धिमत्ताको विकासलाई प्रोत्साहित गर्नको लागि, हामीले यसको विकासमा लगनशील कम्पनीहरुलाई साथ दिनु पर्छ । हामीले पनि कृत्रिम बुद्धिमत्ता���ो विकासमा योगदान पुर्याउनुपर्छ ।\n",
-      "\n",
-      "आह, नेपालमा कृत्रिम बुद्धिमत्ता विकासको वर्तमान अवस्थाले देशलाई एक नयाँ दिशामा लम्बने क्षमता राख्दछ । तर, यसको विकासमा हामी के गरिरहेका छौ? हामीले कृत्रिम बुद्धिमत्ताको विकासमा योगदान पुर्याउने छौ कि? हामीले यसको विकासमा चुनौतिहरुलाई मात गर्ने छौ कि? यस प्रश्नको उत्तर हामीसँग छ । आउनうभ, हामी नेपालमा कृत्रिम बुद्धिमत्ताको विकासलाई प्रोत्साहित गरौं । आउनूभ, हामी देशलाई एक नयाँ दिशामा लम्बौं ।\n"
-     ]
-    }
-   ],
-   "source": [
-    "build_prompt = build_prompt\n",
-    "\n",
-    "sample_title = str(data.iloc[0][\"topic\"])\n",
-    "\n",
-    "sample_response = client.chat.completions.create(\n",
-    "    model=MODEL_NAME,\n",
-    "    messages=[{\"role\": \"user\", \"content\": build_prompt(sample_title)}],\n",
-    ")\n",
-    "\n",
-    "generated_text = sample_response.choices[0].message.content.strip()\n",
-    "print(generated_text)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c709f126",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def grok_step3_5_scraper(\n",
-    "    input_file,\n",
-    "    output_file=\"step3_5_grok_nepali.csv\",\n",
-    "    limit=100,\n",
-    "    model=MODEL_NAME,\n",
-    "    requests_per_second=2,\n",
-    "    max_workers=2,\n",
-    "    max_retries=3,\n",
-    "):\n",
-    "    working_df = pd.read_csv(input_file)\n",
-    "    if limit is not None:\n",
-    "        working_df = working_df.head(limit)\n",
-    "\n",
-    "    cols = set(working_df.columns)\n",
-    "    if \"Title\" in cols or \"शीर्षक\" in cols:\n",
-    "        title_col = \"Title\" if \"Title\" in cols else \"शीर्षक\"\n",
-    "        prompt_col = title_col\n",
-    "        if \"Paragraph\" in cols:\n",
-    "            human_col = \"Paragraph\"\n",
-    "        elif \"विवरण\" in cols:\n",
-    "            human_col = \"विवरण\"\n",
-    "        elif \"paragraph\" in cols:\n",
-    "            human_col = \"paragraph\"\n",
-    "        else:\n",
-    "            human_col = prompt_col\n",
-    "    elif \"paragraph\" in cols or \"Paragraph\" in cols or \"विवरण\" in cols:\n",
-    "        prompt_col = (\n",
-    "            \"paragraph\" if \"paragraph\" in cols\n",
-    "            else (\"Paragraph\" if \"Paragraph\" in cols else \"विवरण\")\n",
-    "        )\n",
-    "        human_col = prompt_col\n",
-    "        title_col = prompt_col\n",
-    "    else:\n",
-    "        raise ValueError(\n",
-    "            \"No supported text columns found. Expected one of: Title/शीर्षक with Paragraph/विवरण, or paragraph.\"\n",
-    "        )\n",
-    "\n",
-    "    working_df = working_df.dropna(subset=[human_col]).copy()\n",
-    "\n",
-    "    total_input_rows = len(working_df)\n",
-    "    already_done = 0\n",
-    "\n",
-    "    if os.path.exists(output_file):\n",
-    "        try:\n",
-    "            existing_df = pd.read_csv(output_file)\n",
-    "            already_done = len(existing_df)\n",
-    "        except pd.errors.EmptyDataError:\n",
-    "            already_done = 0\n",
-    "\n",
-    "    if already_done >= total_input_rows:\n",
-    "        print(\n",
-    "            f\"Nothing to do. {already_done} rows already exist in {output_file} (input rows: {total_input_rows}).\"\n",
-    "        )\n",
-    "        return\n",
-    "\n",
-    "    if already_done > 0:\n",
-    "        working_df = working_df.iloc[already_done:].copy()\n",
-    "        print(\n",
-    "            f\"Resuming from row {already_done}. Processing remaining {len(working_df)} rows out of {total_input_rows}.\"\n",
-    "        )\n",
-    "    else:\n",
-    "        print(f\"Loaded {total_input_rows} rows from {input_file}\")\n",
-    "        print(\n",
-    "            f\"Using title column: {title_col} | prompt column: {prompt_col} | human column: {human_col}\"\n",
-    "        )\n",
-    "\n",
-    "    results = []\n",
-    "\n",
-    "    bad_markers = [\n",
-    "        \"error\",\n",
-    "        \"invalid\",\n",
-    "        \"not found\",\n",
-    "        \"decommissioned\",\n",
-    "        \"rate limit\",\n",
-    "        \"api key\",\n",
-    "    ]\n",
-    "\n",
-    "    def is_valid_ai_text(text: str) -> bool:\n",
-    "        if not text:\n",
-    "            return False\n",
-    "        clean_text = text.strip()\n",
-    "        if len(clean_text) < 20:\n",
-    "            return False\n",
-    "        lower_text = clean_text.lower()\n",
-    "        return not any(marker in lower_text for marker in bad_markers)\n",
-    "\n",
-    "    def extract_retry_wait_seconds(error_text: str) -> float:\n",
-    "        match = re.search(r\"try again in\\s*(\\d+)ms\", error_text, re.IGNORECASE)\n",
-    "        if match:\n",
-    "            return int(match.group(1)) / 1000.0 + 0.2\n",
-    "        return 1.5\n",
-    "\n",
-    "    def process_one(idx, title_text, prompt_text, human_text):\n",
-    "        local_client = Groq(api_key=api_key)\n",
-    "\n",
-    "        for attempt in range(max_retries + 1):\n",
-    "            try:\n",
-    "                completion = local_client.chat.completions.create(\n",
-    "                    model=model,\n",
-    "                    messages=[{\"role\": \"user\", \"content\": build_prompt(str(prompt_text))}],\n",
-    "                    temperature=0.2,\n",
-    "                    max_tokens=500,\n",
-    "                )\n",
-    "                ai_text = completion.choices[0].message.content.strip()\n",
-    "\n",
-    "                if not is_valid_ai_text(ai_text):\n",
-    "                    if attempt < max_retries:\n",
-    "                        continue\n",
-    "                    return {\n",
-    "                        \"idx\": idx,\n",
-    "                        \"ok\": False,\n",
-    "                        \"reason\": \"invalid_or_error_text\",\n",
-    "                        \"ai_text\": ai_text,\n",
-    "                    }\n",
-    "\n",
-    "                return {\n",
-    "                    \"idx\": idx,\n",
-    "                    \"ok\": True,\n",
-    "                    \"title\": str(title_text),\n",
-    "                    \"human_text\": str(human_text),\n",
-    "                    \"ai_generated_text\": ai_text,\n",
-    "                }\n",
-    "            except Exception as error:\n",
-    "                error_text = str(error)\n",
-    "                is_rate_limited = (\n",
-    "                    \"rate_limit_exceeded\" in error_text.lower()\n",
-    "                    or \"rate limit reached\" in error_text.lower()\n",
-    "                    or \"429\" in error_text\n",
-    "                )\n",
-    "\n",
-    "                if is_rate_limited and attempt < max_retries:\n",
-    "                    wait_seconds = extract_retry_wait_seconds(error_text)\n",
-    "                    print(\n",
-    "                        f\"Row {idx} rate-limited, retry {attempt + 1}/{max_retries} after {wait_seconds:.2f}s\"\n",
-    "                    )\n",
-    "                    time.sleep(wait_seconds)\n",
-    "                    continue\n",
-    "\n",
-    "                return {\n",
-    "                    \"idx\": idx,\n",
-    "                    \"ok\": False,\n",
-    "                    \"reason\": error_text,\n",
-    "                    \"ai_text\": \"\",\n",
-    "                }\n",
-    "\n",
-    "    rows = list(working_df[[title_col, prompt_col, human_col]].itertuples(index=True, name=None))\n",
-    "    total = len(rows)\n",
-    "\n",
-    "    for start in range(0, total, requests_per_second):\n",
-    "        window = rows[start : start + requests_per_second]\n",
-    "        tick_start = time.time()\n",
-    "\n",
-    "        with ThreadPoolExecutor(max_workers=max_workers) as executor:\n",
-    "            futures = {\n",
-    "                executor.submit(process_one, idx, title_text, prompt_text, human_text): idx\n",
-    "                for idx, title_text, prompt_text, human_text in window\n",
-    "            }\n",
-    "\n",
-    "            for future in as_completed(futures):\n",
-    "                out = future.result()\n",
-    "                if out[\"ok\"]:\n",
-    "                    # Save as id + ai_gen only\n",
-    "                    results.append({\n",
-    "                        \"id\": out[\"idx\"],\n",
-    "                        \"ai_gen\": out[\"ai_generated_text\"]\n",
-    "                    })\n",
-    "                    print(\n",
-    "                        f\"Row {out['idx']}: generated {len(out['ai_generated_text'].split())} words\"\n",
-    "                    )\n",
-    "                else:\n",
-    "                    print(f\"Row {out['idx']} skipped: {out['reason']}\")\n",
-    "\n",
-    "        if len(results) >= 10:\n",
-    "            pd.DataFrame(results)[[\"id\", \"ai_gen\"]].to_csv(\n",
-    "                output_file,\n",
-    "                index=False,\n",
-    "                mode=\"a\",\n",
-    "                header=not os.path.exists(output_file),\n",
-    "            )\n",
-    "            print(f\"Saved {len(results)} valid rows to {output_file}\")\n",
-    "            results = []\n",
-    "\n",
-    "        elapsed = time.time() - tick_start\n",
-    "        if elapsed < 1:\n",
-    "            time.sleep(1 - elapsed)\n",
-    "\n",
-    "    if results:\n",
-    "        pd.DataFrame(results)[[\"id\", \"ai_gen\"]].to_csv(\n",
-    "            output_file,\n",
-    "            index=False,\n",
-    "            mode=\"a\",\n",
-    "            header=not os.path.exists(output_file),\n",
-    "        )\n",
-    "\n",
-    "    print(f\"Finished. Output saved to {output_file}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "357ccb81",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "ml",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.14"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}