cogni-x/AfriqueLlama-8B-Instruct-TaskVector

AfriqueLlama-8B-Instruct (task-vector merge, experimental)

Status: experimental — not yet validated at scale. Do not treat as a drop-in replacement for McGill-NLP/AfriqueLlama-8B without testing.

What this is

McGill-NLP/AfriqueLlama-8B + the instruction-tuning task vector from unsloth/Meta-Llama-3.1-8B → unsloth/Meta-Llama-3.1-8B-Instruct, applied via task-vector arithmetic (Ilharco et al., "Editing Models with Task Arithmetic"):

instruct_delta = Llama-3.1-8B-Instruct - Llama-3.1-8B
merged_weights = AfriqueLlama-8B + 1.0 * instruct_delta

All 291/291 parameter tensors matched cleanly across the three checkpoints (same architecture, no skipped/mismatched tensors). The tokenizer's chat_template and eos_token are corrected to the standard Llama-3.1-Instruct convention (<|eot_id|>), matching the fix in lora_finetuning/model/chat_template_fixes.py.

Why

McGill-NLP/AfriqueLlama-8B is a continued-pretraining-only checkpoint — per the AfriqueLLM paper (Yu et al.), Section 7 Limitations: "We focus on base model CPT without instruction tuning." It has never seen chat-turn structure and doesn't reliably predict its own stop token. The MOYO response-generation LoRA, fine-tuned directly on top of the raw checkpoint, inherited this: even after fixing model.generation_config.eos_token_id (a real, separate bug — see lora_finetuning/model/loader.py) and retraining, the fine-tuned model still failed to stop on <|eot_id|> in 5/5 sampled trials at temperature 0.3, derailing into repeated/looping text or leaked multilingual pretraining content (observed: Korean, Turkish, Cyrillic-looking fragments) once it ran past where it should have stopped.

Validation so far

A quick smoke test (cognixpert/scripts/test_retrained_model.py, no MOYO LoRA applied — this checkpoint alone) against the same conversation that triggered the original bug:

• 5/5 trials stopped naturally (37-49 generated tokens, well under the 256-token cap), 0/5 derailed.
• Responses were coherent and appropriately empathetic in tone despite zero MOYO-specific fine-tuning on top of this checkpoint.

This has not been run through the full AfroBench-Lite-style evaluation the base AfriqueLLM paper uses, so African-language task performance after the merge is unverified — the task vector could plausibly have partially overwritten some CPT-acquired language capability. That's the open question this checkpoint exists to let us test empirically (by training the MOYO LoRA on top of it and comparing) rather than resolve in advance.

How it was produced

See cognixpert/scripts/task_vector_merge.py in CogniX-LTD/MOYO-AI-Model.