Runqing93/Agent-Guided-Quantum-MLLM

Agent-Guided Quantum Replacement and Parameter-Efficient Adaptation for Multimodal Large Models

Agent-Guided Discovery of Quantum Replacements and Parameter-Efficient Adaptation Methods for Multimodal Large Language Models

简体中文 | English

The project uses a human-supervised Agent workflow to systematically explore parameterized quantum circuits and quantum-classical hybrid modules that can be inserted into multimodal large language models through source-code inspection, training-set example analysis, candidate specification, implementation evolution, and experimental review.

The current open-source release uses Qwen3.8-27B as a case study and comprises independent experimental units spanning candidate configurations, environment checks, frozen baselines, ablations, audits, training runs, and formal re-evaluations. The candidates cover visual residuals, question-conditioned routing, relation mixing, quantum-generated low-rank parameters, value-projection replacements, shared normalization, and complete FFN replacements.

This project is published and maintained by QuSpect Technology Co., Ltd. (量观知元), Beijing, China.

Parameter Archive Overview

Item	Details
Frozen base model	Qwen/Qwen3.8-27B
Fixed revision	1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Experimental units	46
Units with bundled parameters	39
Bundled parameter files	111 .pt files totaling 128,425,026 bytes
Units without scheme parameters	ENVIRONMENT, SIMULATOR, QWEN38_BASE, QH001_INJECTION, QH023_PROTO, QH029, FFN036_AUDIT
Primary runtime environment	Python 3.12, PyTorch 2.8, CUDA 12.8, NVIDIA CUDA GPUs
Quantum simulation method	FP32/complex64 exact-statevector on CUDA

None of the bundled checkpoints contains the complete 27B base-model weights. Users must separately obtain the revision listed above from the official Qwen repository and place it at models/Qwen3.8-27B/ inside the selected experimental unit.

Project Highlights

• One-to-one mapping between parameters and experiments: Each unit keeps its scheme parameters, implementation snapshot, execution entry points, experimental records, and evidence files in the same directory, reducing the risk of mixing versions.
• Independently distributed experimental units: Each directory retains its own dependencies, stage plan, parameter-verification utility, and data-preparation entry point and can be used independently in a new directory.
• Verifiable parameters: manifest.json records the relative path, byte count, and SHA-256 of each checkpoint; verify_parameters.py performs a consistency check before execution.
• Complete experimental trajectory: Successful runs, failed runs, no-entanglement ablations, deletion baselines, classical controls, and frozen-checkpoint re-evaluations are all retained according to their actual status.
• Auditable portable rewrites: The manifest records the SHA-256 values of both the original and distributed files and marks portable rewrites introduced to remove absolute paths.
• Fixed base model and data boundaries: The case-study experiments fix the Qwen3.8-27B revision; the uses of training-set mining, frozen evaluation, and public test sets are distinguished in the corresponding experimental records.

Research Scope

The quantum-classical hybrid structures investigated in this repository include:

1. quantum residuals, anchors, FiLM, and relation routing over visual tokens or visual representations;
2. low-rank parameters and orthogonal subspaces generated or modulated by quantum circuits;
3. low-parameter replacements for linear layers such as v_proj;
4. cross-layer shared bases, spectral modulation, and shared RMSNorm;
5. compressed replacements for complete SwiGLU FFNs and the evolution of their quantum residuals;
6. controls or causal audits involving no entanglement, disabled quantum branches, layer deletion, and classical modules.

The experiments use ScienceQA, TextVQA, CLEVR, EditCLEVR, WikiText, and C4. Each dataset is used only in the relevant experimental units. The exact revision, split, sample budget, and preprocessing procedure are specified in the corresponding directory's README.md, stage_plan.json, and records/EXPERIMENT_RECORD.md.

Repository Structure

.
├─ assets/                         # Project identity and other overview-page assets
├─ ENVIRONMENT/                    # Environment and infrastructure audit
├─ SIMULATOR/                      # GPU exact-statevector microbenchmarks
├─ QWEN38_BASE/                    # Frozen-base loading and inference baseline
├─ QH*/                            # Quantum-classical hybrid candidates, ablations, and re-evaluations
├─ FFN036_AUDIT/                   # Complete-FFN deletion sensitivity audit
├─ CC037_C4/                       # Independent C4 re-evaluation of a classical high-compression replacement
├─ LICENSE                         # MIT License for original project content
├─ THIRD_PARTY_NOTICES.md          # Third-party terms for the base model, datasets, and dependencies
├─ README_CN.md                    # Chinese model card
└─ README.md                       # English model card (Hugging Face default entry point)

Each experimental unit generally contains:

<EXPERIMENT>/
├─ models/<EXPERIMENT>/            # Scheme parameters bundled with this repository, if available
├─ code/src/quantum_qwen38/        # Snapshots of candidate modules and simulator implementations
├─ code/scripts/                   # Preprocessing, training, inference, and evaluation entry points
├─ artifacts/                      # Distributed lock files and experimental outputs, if available
├─ records/EXPERIMENT_RECORD.md    # Original experimental status, protocol, results, and limitations
├─ manifest.json                   # Hash inventory for parameters, source, entry points, and evidence
├─ stage_plan.json                 # Execution plan for the stages actually reached by this unit
├─ run_reached_stage.py            # Portable stage runner
├─ verify_parameters.py            # Parameter inventory and SHA-256 verification
├─ download_base_model.py          # Frozen-base download utility
├─ download_data.py                # Dataset download and preprocessing entry point
├─ requirements.txt                # Dependencies for the unit
└─ README.md                       # Unit-level reproduction instructions

The code snapshots retained in individual experimental units are intentional. Later candidates may modify implementations with the same filename; independent snapshots prevent newer implementations from replacing the versions used to reproduce earlier experiments.

Quick Start

The following example uses QH014. First download the complete repository from this repository page, then enter the selected experimental directory; do not copy only an individual .pt file.

1. Enter the Experimental Unit and Review Its Status

cd QH014

Before running anything, read README.md and records/EXPERIMENT_RECORD.md in the current directory. Different units reached different stages. Some support only static checks or audits and should not be treated as trained models.

2. Create a Python Environment

The historical experimental environment used 4×A800-SXM4-80GB GPUs, Python 3.12, PyTorch 2.8/CUDA 12.8, and the Transformers 5.16 development line. The quantum training and inference entry points require an NVIDIA CUDA GPU and reject silent fallback to the CPU.

Linux shell:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

3. Download the Frozen Base Model

The base-model weights are not redistributed with this repository. Every experimental unit provides a download script with the same interface:

python download_base_model.py --output ./models/Qwen3.8-27B

The registered upstream repository is Qwen/Qwen3.8-27B, fixed at the following revision:

1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

The equivalent Hugging Face CLI command is:

hf download Qwen/Qwen3.8-27B \
  --revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 \
  --local-dir ./models/Qwen3.8-27B

4. Verify the Bundled Scheme Parameters

python verify_parameters.py

The verification utility reads manifest.json in the current unit and checks the path, size, and SHA-256 of every registered parameter under models/<EXPERIMENT>/. For environment, baseline, static-check, or audit-only units without checkpoints, this step behaves according to that unit's instructions and does not fabricate scheme weights.

5. Prepare the Data and Run the Reached Stage

python download_data.py --root ./datasets
python run_reached_stage.py

run_reached_stage.py executes only the preprocessing, training, frozen-evaluation, or analysis steps registered in stage_plan.json. It does not imply that every unit supports free-form text generation, nor does it package failed candidates or candidates that completed only static checks as trained models.

6. Run a Base-Model Inference Self-Check

python base_inference.py \
  --model ./models/Qwen3.8-27B \
  --prompt "Explain the purpose of this model."

This command verifies only the base-model download and generation path; it does not represent the performance of the quantum scheme. Scheme performance should be determined from the real-data frozen inference or formal evaluation registered for the corresponding unit.

Experimental Unit Index

Infrastructure, Baselines, and Early Validation

Experimental Unit	Description
ENVIRONMENT	Environment and infrastructure checks
SIMULATOR	GPU exact-statevector simulator microbenchmarks
QWEN38_BASE	Qwen3.8-27B loading and frozen baseline
QH001_INJECTION	Quantum injection into the actual model and a one-step optimization check
QH001B_STAGE_A	Stage-A experiment on 128 randomly selected ScienceQA examples
QH008_SCIENCEQA	ScienceQA train-only hard-example mining
QH008_TEXTVQA	TextVQA OCR capability-domain check

Visual Adaptation, Quantum Residuals, and Relation Routing

Experimental Unit	Description
QH009	Correlation-observable gating
QH010	Cayley two-qubit adapter
QH011	Visual-prefill Cayley adapter
QH012	Four-qubit visual brickwork
QH013	Sparsely routed quantum-correlation gating
QH014	Question-conditioned single-image quantum anchor
QH015	Question-conditioned quantum FiLM
QH016	Question-routed spatial-relation module
QH017	Quantum-state-fidelity relation routing
QH018	Conservative quantum-residual routing
QH018_NOENT	Multi-seed no-entanglement product-state ablation for QH018
QH019	Coherent amplitude mixing over six relations
QH020	A 12-parameter mixer trained separately after a shared trunk
QH021	Causal audit of the 12-parameter mixer under an exactly shared trunk

Parameter Generation, Low-Rank Adaptation, and Layer Replacement

Experimental Unit	Description
QH022	Inference-resident quantum PEFT coefficient generator
QH023_PROTO	Static Pauli/Stiefel orthogonal-subspace prototype
QH023	Formal full-development-set evaluation of the Pauli/Stiefel scheme on EditCLEVR
QH024	True replacement of a complete v_proj
QH025	End-to-end final-hidden priming
QH026	Butterfly data-reupload true-layer replacement
QH029	Raw-Qwen MPO/disentangler feasibility check
QH031	Cross-layer shared-basis true replacement
QH032	Nonlinear Pauli-observable readout
QH033	Second stage of teacher-logit KL training
QH034	Quantum spectral modulation in shared singular coordinates
QH035	Shared RMSNorm with a runtime quantum residual

Complete FFN Replacement, Quantum-Residual Evolution, and Formal Re-evaluation

Experimental Unit	Description
FFN036_AUDIT	Complete-FFN deletion sensitivity audit across 64 layers
QH037	Complete FFN replacement with grouped-entanglement QVAF
QH038	Complete FFN replacement with shifted-ring cross-group FC-VQC
QH039	Switchable quantum residual on a frozen high-compression scaffold
QH040	Bounded trust-region quantum residual
QH041	Bounded quantum residual with signed-X readout
QH042	Quantum residual with signed-XZ nearest-neighbor correlation observables
QH043	Second-harmonic signed-X data re-uploading
CC037_C4	Independent C4 confirmation of a classical high-compression FFN replacement
QH044_TRAIN	Equal-budget continued training from the QH037/CC037 optimizer states
QH044_FORMAL	Formal causal evaluation of the frozen QH044 checkpoint on C4-128
QH045_TRAIN	Selective-entanglement training with 64 parameters from the same starting point
QH045_FORMAL	Formal evaluation of the frozen QH045 checkpoint on C4-64

How to Read the Experimental Results

The primary source for determining an experiment's status and the boundaries of its conclusions is records/EXPERIMENT_RECORD.md in the corresponding directory. When reading a record, check the following together:

1. status and purpose: distinguish complete experiments, partial experiments, static checks, failures, and stopped runs;
2. data locks and sample budgets: verify the boundaries among training-set mining, validation, and testing;
3. stage_plan.json: confirm the stages actually executed by the distributed entry point;
4. manifest.json: verify the SHA-256 values of parameters, source files, entry points, evidence, and portable rewrites;
5. artifacts/: inspect per-example outputs, statistical analyses, or lock files when distributed;
6. checkpoint inventory: confirm that the parameters used for a result exist and pass hash verification.

Sample sizes, seeds, training budgets, and statistical rules are not identical across experiments. Schemes therefore should not be ranked solely by individual point estimates taken from different directories. Formal comparisons should use aligned interfaces, frozen base models, data splits, optimization budgets, and evaluation protocols.

Usage Limitations and Reproducibility Boundaries

• A checkpoint applies only to the experimental implementation, fixed base-model revision, and directory layout registered in its manifest.json; it cannot be used independently of the base model.
• This repository is intended for research reproduction and auditing. It does not claim that these candidates meet the requirements of production deployment, general-purpose dialogue, or safety-critical applications.
• The presence of a checkpoint in a unit indicates only that historical parameters have been archived; it does not mean that the candidate met the predefined success criteria.
• The original and distributed scripts may differ only in portable rewrites that remove absolute paths; their hashes and rewrite markers are both recorded in manifest.json.
• Training and inference resource requirements vary by experiment. The 27B base model and exact-statevector paths generally require high-memory NVIDIA GPUs; see the README for the individual unit for details.

License and Third-Party Materials

Original code and documentation in this project, together with newly introduced parameters that the publisher has the right to license, are released under the MIT License. The MIT License permits academic and commercial use, but copies or substantial portions must retain the copyright and license notices.

The Qwen3.8-27B base model, datasets, software dependencies, and derived parameters that may involve third-party rights remain subject to their respective terms. This project's MIT declaration does not relicense third-party materials. See THIRD_PARTY_NOTICES.md for the complete statement.

Citation

If you use this code or build upon our work in your research, please cite the following preprint:

@misc{quspect2026agentguided,
  author       = {{QuSpect Technology Co., Ltd., Beijing, China}},
  title        = {{Agent-Guided Discovery of Quantum Replacements and Parameter-Efficient Adaptation Methods for Multimodal Large Language Models}},
  year         = {2026},
  month        = sep,
  howpublished = {Zenodo},
  doi          = {10.5281/zenodo.22301614},
  url          = {https://doi.org/10.5281/zenodo.22301614},
  note         = {Preprint, version v1}
}