MVP Milestone 3

IMPLEMENTATION_PLAN.md

@@ -609,7 +609,7 @@ Included requirements: `PROJECT_SPEC.md / Dataset Specification / Article Datase
 Tasks:
 
 - Implement article model preserving exact source field names.
-- Validate required keys, field types, non-empty strings, and DOI uniqueness.
+- Validate required keys, strict field types, non-empty DOI and Title strings, and DOI uniqueness. `Abstract` is required and must be a string, but may be empty.
 - Report invalid records without modifying source records.
 - Add valid and invalid article fixtures.
 
@@ -619,14 +619,14 @@ Tests:
 
 - Valid article loading.
 - Missing required fields.
-- Empty DOI/title/abstract.
-- Non-integer Year.
+- Empty DOI and Title. Empty `Abstract` is allowed and must be preserved exactly.
+- Non-integer Year, including Boolean values.
 - Duplicate DOI.
 - Source values preserved exactly.
 
 Acceptance criteria:
 
-- Current full-data smoke test confirms `data/articles.json` validates with 468 unique DOI records.
+- Current full-data smoke test confirms `data/articles.json` contains 468 source records, 467 records with valid non-empty unique DOI values, exactly one invalid record at index 467 due to empty DOI, and 14 otherwise valid records with empty `Abstract` values.
 - Invalid fixtures produce explicit errors.
 - No normalization or rewriting occurs.
 

MVP_PROGRESS.md

@@ -100,3 +100,84 @@ Deviations from approved plan:
 Remaining risks or assumptions:
 - The loader remains fully dynamic and does not hard-code vocabulary names, UUIDs, paths, branches, or record counts, but full-data smoke tests intentionally assert current-file counts.
 - Future approved vocabulary rules would be needed before accepting any UUID-less non-root hierarchy node.
+
+## Milestone 3: Article Loading and Validation
+
+Status: completed
+
+Scope completed:
+- Deterministic loader for `data/articles.json`
+- Strict source article model preserving exact serialized fields: `DOI`, `Title`, `Year`, and `Abstract`
+- Read/decode, individual record validation, duplicate DOI detection, and aggregate result return paths
+- Record-level validation errors with source index, code, field, message, and DOI when available
+- Valid and invalid article fixtures
+- Full-data smoke test for the current `data/articles.json`
+
+Implemented article model and loader APIs:
+- `ArticleRecord` in `src/gcmd_classifier/models.py`
+- `ArticleValidationIssue` and `ArticleLoadResult` in `src/gcmd_classifier/models.py`
+- `read_article_json`, `validate_article_record`, `validate_article_records`, and `load_articles` in `src/gcmd_classifier/articles/loader.py`
+
+Validation and error-collection policy:
+- File-level conditions raise typed exceptions: missing file, invalid JSON, and top-level JSON value that is not a list.
+- Record-level conditions are collected in `ArticleLoadResult.errors` while all remaining records continue to be validated.
+- Duplicate DOI records are reported with `DUPLICATE_DOI` and excluded from `ArticleLoadResult.articles` so returned records have unique DOI values.
+
+Extra-field policy:
+- Unexpected fields are rejected for the MVP with `UNEXPECTED_FIELD` because neither `PROJECT_SPEC.md` nor `AGENTS.md` defines an allow-list extension policy for source article records.
+
+Exact-preservation protections:
+- Successful `ArticleRecord` values are returned unchanged.
+- DOI values are not normalized or replaced.
+- Titles and abstracts are not stripped, lowercased, rewritten, summarized, or supplemented.
+- Empty `Abstract` values are valid and preserved exactly.
+- Whitespace-only strings are preserved; non-empty checks do not trim source values.
+- `Year` must be a strict integer and Boolean values are rejected.
+
+Current full-data findings:
+- `data/articles.json` contains 468 source records.
+- 467 records have valid non-empty unique DOI values.
+- Exactly one record, index 467, is invalid because `DOI` is empty.
+- 14 records have empty `Abstract` values and otherwise load successfully.
+
+Error cases covered:
+- File not found
+- Invalid JSON
+- Top-level JSON value not a list
+- Article entry not an object
+- Missing `DOI`, `Title`, `Year`, or `Abstract`
+- Empty `DOI` and `Title`
+- Non-string `DOI`, `Title`, or `Abstract`
+- String, floating-point, and Boolean `Year`
+- Null field values
+- Duplicate DOI
+- Unexpected extra fields
+- Multiple invalid records reported in one aggregate result
+
+Verification:
+- `python -m ruff check .` passed: `All checks passed!`
+- `python -m ruff format --check .` passed: `11 files already formatted`
+- `python -m pytest` passed: `51 passed in 0.18s`
+
+Out of scope, not started:
+- Model prompts
+- Model-provider integration
+- Topic routing
+- Term routing
+- Variable-level classification
+- Persistence
+- Caching
+- Batch processing
+- UI and root `app.py`
+
+Source-file status:
+- `data/articles.json` unchanged by this milestone.
+- `data/gcmd_hierarchy.json` unchanged by this milestone.
+- `prototype/app_hf_poc.py` unchanged by this milestone.
+
+Deviations from approved plan:
+- `IMPLEMENTATION_PLAN.md` was revised to reflect the clarified Milestone 3 rule that empty `Abstract` values are valid.
+
+Remaining risks or assumptions:
+- The current source file remains partially invalid until the empty DOI at index 467 is corrected.
+- Whitespace-only DOI or Title values are non-empty and therefore preserved as valid under the explicit no-trimming rule; this can be tightened later only with an approved source-data rule.

src/gcmd_classifier/articles/__init__.py

@@ -0,0 +1,15 @@
+"""Article loading and validation for source journal records."""
+
+from gcmd_classifier.articles.loader import (
+    load_articles,
+    read_article_json,
+    validate_article_record,
+    validate_article_records,
+)
+
+__all__ = [
+    "load_articles",
+    "read_article_json",
+    "validate_article_record",
+    "validate_article_records",
+]

src/gcmd_classifier/articles/loader.py

@@ -0,0 +1,186 @@
+"""Deterministic loading and validation for source article records."""
+
+from __future__ import annotations
+
+import json
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+from pydantic import ValidationError
+
+from gcmd_classifier.errors import (
+    ArticleFileNotFoundError,
+    ArticleJSONDecodeError,
+    ArticleTopLevelTypeError,
+)
+from gcmd_classifier.models import ArticleLoadResult, ArticleRecord, ArticleValidationIssue
+
+REQUIRED_FIELDS = ("DOI", "Title", "Year", "Abstract")
+_STRING_FIELDS = ("DOI", "Title", "Abstract")
+_NON_EMPTY_STRING_FIELDS = ("DOI", "Title")
+
+
+def read_article_json(path: str | Path) -> Any:
+    """Read and decode an article JSON file without validating record contents."""
+    article_path = Path(path)
+    try:
+        raw_bytes = article_path.read_bytes()
+    except FileNotFoundError as exc:
+        raise ArticleFileNotFoundError(f"Article file not found: {article_path}") from exc
+    try:
+        return json.loads(raw_bytes)
+    except json.JSONDecodeError as exc:
+        raise ArticleJSONDecodeError(f"Invalid JSON in article file: {article_path}") from exc
+
+
+def validate_article_record(
+    raw_record: Any, index: int
+) -> tuple[ArticleRecord | None, tuple[ArticleValidationIssue, ...]]:
+    """Validate one article record, preserving source values exactly on success."""
+    issues: list[ArticleValidationIssue] = []
+    if not isinstance(raw_record, dict):
+        return None, (
+            ArticleValidationIssue(
+                index=index,
+                code="ARTICLE_NOT_OBJECT",
+                message="Article entry must be a JSON object.",
+            ),
+        )
+
+    for field in REQUIRED_FIELDS:
+        if field not in raw_record:
+            issues.append(
+                ArticleValidationIssue(
+                    index=index,
+                    code="MISSING_REQUIRED_FIELD",
+                    field=field,
+                    message=f"Article record is missing required field {field!r}.",
+                    DOI=_safe_doi(raw_record),
+                )
+            )
+
+    for field in _STRING_FIELDS:
+        if field in raw_record and not isinstance(raw_record[field], str):
+            issues.append(
+                ArticleValidationIssue(
+                    index=index,
+                    code="INVALID_FIELD_TYPE",
+                    field=field,
+                    message=f"Article field {field!r} must be a string.",
+                    DOI=_safe_doi(raw_record),
+                )
+            )
+
+    for field in _NON_EMPTY_STRING_FIELDS:
+        if field in raw_record and isinstance(raw_record[field], str) and not raw_record[field]:
+            issues.append(
+                ArticleValidationIssue(
+                    index=index,
+                    code="EMPTY_REQUIRED_STRING",
+                    field=field,
+                    message=f"Article field {field!r} must be a non-empty string.",
+                    DOI=_safe_doi(raw_record),
+                )
+            )
+
+    if "Year" in raw_record:
+        year = raw_record["Year"]
+        if isinstance(year, bool) or not isinstance(year, int):
+            issues.append(
+                ArticleValidationIssue(
+                    index=index,
+                    code="INVALID_FIELD_TYPE",
+                    field="Year",
+                    message="Article field 'Year' must be a strict integer and not a Boolean.",
+                    DOI=_safe_doi(raw_record),
+                )
+            )
+
+    unexpected_fields = tuple(field for field in raw_record if field not in REQUIRED_FIELDS)
+    for field in unexpected_fields:
+        issues.append(
+            ArticleValidationIssue(
+                index=index,
+                code="UNEXPECTED_FIELD",
+                field=field,
+                message=f"Unexpected article field {field!r} is not allowed in the MVP.",
+                DOI=_safe_doi(raw_record),
+            )
+        )
+
+    if issues:
+        return None, tuple(issues)
+
+    try:
+        return ArticleRecord.model_validate(raw_record), ()
+    except ValidationError as exc:
+        return None, tuple(_issues_from_pydantic_error(exc, raw_record=raw_record, index=index))
+
+
+def validate_article_records(raw_records: Any) -> ArticleLoadResult:
+    """Validate a top-level article list and collect all record-level validation errors."""
+    if not isinstance(raw_records, list):
+        raise ArticleTopLevelTypeError("Article source JSON must be a top-level list.")
+
+    valid_candidates: list[tuple[int, ArticleRecord]] = []
+    errors: list[ArticleValidationIssue] = []
+    for index, raw_record in enumerate(raw_records):
+        article, issues = validate_article_record(raw_record, index)
+        errors.extend(issues)
+        if article is not None:
+            valid_candidates.append((index, article))
+
+    doi_counts = Counter(article.DOI for _, article in valid_candidates)
+    duplicate_dois = {doi for doi, count in doi_counts.items() if count > 1}
+    if duplicate_dois:
+        unique_candidates: list[tuple[int, ArticleRecord]] = []
+        for index, article in valid_candidates:
+            if article.DOI in duplicate_dois:
+                errors.append(
+                    ArticleValidationIssue(
+                        index=index,
+                        code="DUPLICATE_DOI",
+                        field="DOI",
+                        message=f"Duplicate DOI value {article.DOI!r}.",
+                        DOI=article.DOI,
+                    )
+                )
+            else:
+                unique_candidates.append((index, article))
+        valid_candidates = unique_candidates
+
+    return ArticleLoadResult(
+        articles=tuple(article for _, article in valid_candidates),
+        errors=tuple(errors),
+        source_count=len(raw_records),
+    )
+
+
+def load_articles(path: str | Path) -> ArticleLoadResult:
+    """Read and validate article records from a JSON file."""
+    return validate_article_records(read_article_json(path))
+
+
+def _safe_doi(raw_record: dict[str, Any]) -> str | None:
+    value = raw_record.get("DOI")
+    return value if isinstance(value, str) else None
+
+
+def _issues_from_pydantic_error(
+    exc: ValidationError, *, raw_record: dict[str, Any], index: int
+) -> tuple[ArticleValidationIssue, ...]:
+    issues: list[ArticleValidationIssue] = []
+    for error in exc.errors():
+        location = error.get("loc", ())
+        field = str(location[0]) if location else None
+        issues.append(
+            ArticleValidationIssue(
+                index=index,
+                code="MODEL_VALIDATION_ERROR",
+                field=field,
+                message=str(error.get("msg", "Article model validation failed.")),
+                DOI=_safe_doi(raw_record),
+            )
+        )
+    return tuple(issues)

src/gcmd_classifier/errors.py

@@ -27,3 +27,19 @@ class InvalidHierarchyTransitionError(VocabularyError):
 
 class VocabularyLookupError(VocabularyError):
     """Raised when a requested vocabulary concept or relationship is missing."""
+
+
+class ArticleError(GCMDClassifierError):
+    """Base exception for article loading and validation errors."""
+
+
+class ArticleFileNotFoundError(ArticleError):
+    """Raised when an article source file does not exist."""
+
+
+class ArticleJSONDecodeError(ArticleError):
+    """Raised when an article source file contains invalid JSON."""
+
+
+class ArticleTopLevelTypeError(ArticleError):
+    """Raised when an article source file is not a top-level JSON list."""

src/gcmd_classifier/models.py

@@ -40,3 +40,46 @@ class CanonicalConceptRecord(BaseModel):
     assignable: bool
     definition: str | None = None
     vocabulary_version: str
+
+
+class ArticleRecord(BaseModel):
+    """Validated source article record with exact serialized field names."""
+
+    model_config = ConfigDict(extra="forbid", frozen=True, strict=True)
+
+    DOI: str
+    Title: str
+    Year: int
+    Abstract: str
+
+
+class ArticleValidationIssue(BaseModel):
+    """Structured validation issue for one article source record."""
+
+    model_config = ConfigDict(frozen=True)
+
+    index: int | None
+    code: str
+    message: str
+    field: str | None = None
+    DOI: str | None = None
+
+
+class ArticleLoadResult(BaseModel):
+    """Aggregate article loading result preserving valid records and all issues."""
+
+    model_config = ConfigDict(frozen=True)
+
+    articles: tuple[ArticleRecord, ...]
+    errors: tuple[ArticleValidationIssue, ...] = Field(default_factory=tuple)
+    source_count: int
+
+    @property
+    def valid_count(self) -> int:
+        """Number of records that passed validation and duplicate checks."""
+        return len(self.articles)
+
+    @property
+    def has_errors(self) -> bool:
+        """Whether any source records failed validation."""
+        return bool(self.errors)

tests/fixtures/articles_invalid.json

@@ -0,0 +1,102 @@
+[
+  {
+    "Title": "Missing DOI",
+    "Year": 2024,
+    "Abstract": "A record without DOI."
+  },
+  {
+    "DOI": "10.1000/MISSINGTITLE",
+    "Year": 2024,
+    "Abstract": "A record without title."
+  },
+  {
+    "DOI": "10.1000/MISSINGYEAR",
+    "Title": "Missing year",
+    "Abstract": "A record without year."
+  },
+  {
+    "DOI": "10.1000/MISSINGABSTRACT",
+    "Title": "Missing abstract",
+    "Year": 2024
+  },
+  {
+    "DOI": "",
+    "Title": "Empty DOI",
+    "Year": 2024,
+    "Abstract": "Empty DOI should fail."
+  },
+  {
+    "DOI": "10.1000/EMPTYTITLE",
+    "Title": "",
+    "Year": 2024,
+    "Abstract": "Empty title should fail."
+  },
+  {
+    "DOI": "10.1000/WHITESPACE",
+    "Title": "   ",
+    "Year": 2024,
+    "Abstract": "Whitespace title is preserved and currently valid."
+  },
+  {
+    "DOI": 123,
+    "Title": "Non-string DOI",
+    "Year": 2024,
+    "Abstract": "DOI must be a string."
+  },
+  {
+    "DOI": "10.1000/NONSTRINGTITLE",
+    "Title": ["bad"],
+    "Year": 2024,
+    "Abstract": "Title must be a string."
+  },
+  {
+    "DOI": "10.1000/NONSTRINGABSTRACT",
+    "Title": "Non-string abstract",
+    "Year": 2024,
+    "Abstract": 42
+  },
+  {
+    "DOI": "10.1000/STRINGYEAR",
+    "Title": "String year",
+    "Year": "2024",
+    "Abstract": "Year must not be a string."
+  },
+  {
+    "DOI": "10.1000/FLOATYEAR",
+    "Title": "Float year",
+    "Year": 2024.0,
+    "Abstract": "Year must not be a float."
+  },
+  {
+    "DOI": "10.1000/BOOLYEAR",
+    "Title": "Boolean year",
+    "Year": true,
+    "Abstract": "Year must not be Boolean."
+  },
+  {
+    "DOI": null,
+    "Title": "Null DOI",
+    "Year": 2024,
+    "Abstract": "Null DOI should fail."
+  },
+  "not an object",
+  {
+    "DOI": "10.1000/DUPLICATE",
+    "Title": "Duplicate one",
+    "Year": 2024,
+    "Abstract": "First duplicate."
+  },
+  {
+    "DOI": "10.1000/DUPLICATE",
+    "Title": "Duplicate two",
+    "Year": 2025,
+    "Abstract": "Second duplicate."
+  },
+  {
+    "DOI": "10.1000/EXTRA",
+    "Title": "Unexpected field",
+    "Year": 2025,
+    "Abstract": "Extra fields are rejected for MVP.",
+    "Journal": "Example"
+  }
+]

tests/fixtures/articles_valid.json

@@ -0,0 +1,20 @@
+[
+  {
+    "DOI": "10.0001/ATMOS.TEST",
+    "Title": "  High-Resolution Rainfall Trends over Mountain Basins: A 20-Year Study  ",
+    "Year": 2024,
+    "Abstract": "Precipitation intensity, snowpack, and runoff timing are evaluated using station and satellite observations."
+  },
+  {
+    "DOI": "10.0002/MULTI.TEST",
+    "Title": "Atmosphere-ocean coupling during coastal marine heatwaves",
+    "Year": 2025,
+    "Abstract": "The study links sea surface temperature anomalies, boundary-layer humidity, and plankton bloom timing."
+  },
+  {
+    "DOI": "10.0003/TITLEONLY.TEST",
+    "Title": "Editorial: New instrumentation for environmental archives?",
+    "Year": 2023,
+    "Abstract": ""
+  }
+]

tests/test_articles.py

@@ -0,0 +1,246 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from gcmd_classifier.articles import (
+    load_articles,
+    read_article_json,
+    validate_article_record,
+    validate_article_records,
+)
+from gcmd_classifier.errors import (
+    ArticleFileNotFoundError,
+    ArticleJSONDecodeError,
+    ArticleTopLevelTypeError,
+)
+from gcmd_classifier.models import ArticleRecord
+
+VALID_FIXTURE = Path("tests/fixtures/articles_valid.json")
+INVALID_FIXTURE = Path("tests/fixtures/articles_invalid.json")
+FULL_ARTICLES_PATH = Path("data/articles.json")
+
+
+def test_valid_article_loading() -> None:
+    result = load_articles(VALID_FIXTURE)
+
+    assert result.source_count == 3
+    assert result.valid_count == 3
+    assert result.errors == ()
+    assert all(isinstance(article, ArticleRecord) for article in result.articles)
+
+
+def test_preserves_source_order() -> None:
+    result = load_articles(VALID_FIXTURE)
+
+    assert [article.DOI for article in result.articles] == [
+        "10.0001/ATMOS.TEST",
+        "10.0002/MULTI.TEST",
+        "10.0003/TITLEONLY.TEST",
+    ]
+
+
+def test_exact_source_value_preservation() -> None:
+    raw = json.loads(VALID_FIXTURE.read_text())
+    result = load_articles(VALID_FIXTURE)
+    first = result.articles[0]
+
+    assert first.DOI == raw[0]["DOI"]
+    assert first.Title == raw[0]["Title"]
+    assert first.Year == raw[0]["Year"]
+    assert first.Abstract == raw[0]["Abstract"]
+    assert first.Title.startswith("  ")
+    assert first.Title.endswith("  ")
+
+
+def test_empty_abstract_is_valid_and_preserved() -> None:
+    result = load_articles(VALID_FIXTURE)
+    article = result.articles[2]
+
+    assert article.Abstract == ""
+    assert article.Title == "Editorial: New instrumentation for environmental archives?"
+
+
+def test_exact_serialization_field_names() -> None:
+    article = load_articles(VALID_FIXTURE).articles[0]
+
+    assert article.model_dump() == {
+        "DOI": "10.0001/ATMOS.TEST",
+        "Title": "  High-Resolution Rainfall Trends over Mountain Basins: A 20-Year Study  ",
+        "Year": 2024,
+        "Abstract": "Precipitation intensity, snowpack, and runoff timing are evaluated using "
+        "station and satellite observations.",
+    }
+
+
+def test_invalid_fixture_reports_multiple_record_errors() -> None:
+    result = load_articles(INVALID_FIXTURE)
+    codes = [error.code for error in result.errors]
+
+    assert result.source_count == 18
+    assert result.has_errors is True
+    assert "MISSING_REQUIRED_FIELD" in codes
+    assert "EMPTY_REQUIRED_STRING" in codes
+    assert "INVALID_FIELD_TYPE" in codes
+    assert "ARTICLE_NOT_OBJECT" in codes
+    assert "DUPLICATE_DOI" in codes
+    assert "UNEXPECTED_FIELD" in codes
+    assert len(result.errors) >= 17
+
+
+@pytest.mark.parametrize(
+    ("record", "field"),
+    [
+        ({"Title": "T", "Year": 2024, "Abstract": "A"}, "DOI"),
+        ({"DOI": "10.x", "Year": 2024, "Abstract": "A"}, "Title"),
+        ({"DOI": "10.x", "Title": "T", "Abstract": "A"}, "Year"),
+        ({"DOI": "10.x", "Title": "T", "Year": 2024}, "Abstract"),
+    ],
+)
+def test_missing_required_fields(record: dict, field: str) -> None:
+    article, errors = validate_article_record(record, index=7)
+
+    assert article is None
+    assert any(error.code == "MISSING_REQUIRED_FIELD" and error.field == field for error in errors)
+    assert all(error.index == 7 for error in errors)
+
+
+@pytest.mark.parametrize(
+    ("record", "field"),
+    [
+        ({"DOI": "", "Title": "T", "Year": 2024, "Abstract": "A"}, "DOI"),
+        ({"DOI": "10.x", "Title": "", "Year": 2024, "Abstract": "A"}, "Title"),
+    ],
+)
+def test_empty_required_strings(record: dict, field: str) -> None:
+    article, errors = validate_article_record(record, index=1)
+
+    assert article is None
+    assert any(error.code == "EMPTY_REQUIRED_STRING" and error.field == field for error in errors)
+
+
+def test_whitespace_only_title_is_preserved_not_normalized() -> None:
+    record = {"DOI": "10.x/space", "Title": "   ", "Year": 2024, "Abstract": "A"}
+
+    article, errors = validate_article_record(record, index=0)
+
+    assert errors == ()
+    assert article is not None
+    assert article.Title == "   "
+
+
+@pytest.mark.parametrize(
+    ("field", "value"),
+    [
+        ("DOI", 123),
+        ("Title", ["bad"]),
+        ("Abstract", 42),
+    ],
+)
+def test_non_string_fields_rejected(field: str, value: object) -> None:
+    record = {"DOI": "10.x", "Title": "T", "Year": 2024, "Abstract": "A"}
+    record[field] = value
+
+    article, errors = validate_article_record(record, index=2)
+
+    assert article is None
+    assert any(error.code == "INVALID_FIELD_TYPE" and error.field == field for error in errors)
+
+
+@pytest.mark.parametrize("year", ["2024", 2024.0, True, False])
+def test_invalid_year_types_rejected(year: object) -> None:
+    record = {"DOI": "10.x/year", "Title": "T", "Year": year, "Abstract": "A"}
+
+    article, errors = validate_article_record(record, index=3)
+
+    assert article is None
+    assert any(error.code == "INVALID_FIELD_TYPE" and error.field == "Year" for error in errors)
+
+
+def test_null_field_values_rejected() -> None:
+    record = {"DOI": None, "Title": None, "Year": None, "Abstract": None}
+
+    article, errors = validate_article_record(record, index=4)
+
+    assert article is None
+    assert len([error for error in errors if error.code == "INVALID_FIELD_TYPE"]) == 4
+
+
+def test_duplicate_doi_detection_excludes_duplicate_records_from_valid_records() -> None:
+    raw_records = [
+        {"DOI": "10.same", "Title": "One", "Year": 2024, "Abstract": "A"},
+        {"DOI": "10.same", "Title": "Two", "Year": 2025, "Abstract": "B"},
+        {"DOI": "10.unique", "Title": "Three", "Year": 2026, "Abstract": "C"},
+    ]
+
+    result = validate_article_records(raw_records)
+
+    assert [article.DOI for article in result.articles] == ["10.unique"]
+    assert [error.index for error in result.errors if error.code == "DUPLICATE_DOI"] == [0, 1]
+
+
+def test_non_object_article_entry() -> None:
+    result = validate_article_records(["bad"])
+
+    assert result.articles == ()
+    assert result.errors[0].code == "ARTICLE_NOT_OBJECT"
+    assert result.errors[0].index == 0
+
+
+def test_top_level_non_list_json() -> None:
+    with pytest.raises(ArticleTopLevelTypeError):
+        validate_article_records({"DOI": "10.x"})
+
+
+def test_invalid_json(tmp_path: Path) -> None:
+    path = tmp_path / "invalid.json"
+    path.write_text("[{bad json]")
+
+    with pytest.raises(ArticleJSONDecodeError):
+        read_article_json(path)
+
+
+def test_file_not_found_behavior(tmp_path: Path) -> None:
+    with pytest.raises(ArticleFileNotFoundError):
+        load_articles(tmp_path / "missing.json")
+
+
+def test_source_file_immutability() -> None:
+    before = FULL_ARTICLES_PATH.read_bytes()
+    load_articles(FULL_ARTICLES_PATH)
+    after = FULL_ARTICLES_PATH.read_bytes()
+
+    assert after == before
+
+
+def test_full_data_smoke_current_articles_file() -> None:
+    raw_records = json.loads(FULL_ARTICLES_PATH.read_text())
+    result = load_articles(FULL_ARTICLES_PATH)
+
+    assert result.source_count == 468
+    assert len(raw_records) == 468
+    assert result.valid_count == 467
+    assert len({article.DOI for article in result.articles}) == 467
+    assert result.has_errors is True
+    assert len(result.errors) == 1
+    error = result.errors[0]
+    assert error.index == 467
+    assert error.field == "DOI"
+    assert error.code == "EMPTY_REQUIRED_STRING"
+
+    empty_abstract_indices = [
+        index for index, record in enumerate(raw_records) if record.get("Abstract") == ""
+    ]
+    assert len(empty_abstract_indices) == 14
+    assert all(
+        raw_records[index]["DOI"] in {article.DOI for article in result.articles}
+        for index in empty_abstract_indices
+    )
+
+    valid_raw_records = [record for index, record in enumerate(raw_records) if index != 467]
+    assert [article.DOI for article in result.articles] == [
+        record["DOI"] for record in valid_raw_records
+    ]
+    assert result.articles[0].model_dump() == valid_raw_records[0]