models/README.md

---
language: en
license: mit
tags:
- multi-label-classification
- tfidf
- embeddings
- random-forest
- oversampling
- mlsmote
- software-engineering
datasets:
- NLBSE/SkillCompetition
model-index:
- name: random_forest_tfidf_gridsearch
  results:
  - status: success
    metrics:
      cv_best_f1_micro: 0.595038375202279
      test_precision_micro: 0.690371373744215
      test_recall_micro: 0.5287455692919513
      test_f1_micro: 0.5988446098110252
    params:
      estimator__max_depth: '10'
      estimator__min_samples_split: '2'
      estimator__n_estimators: '200'
      feature_type: embedding
      model_type: RandomForest + MultiOutput
      use_cleaned: 'True'
      oversampling: 'False'
    dvc:
      path: random_forest_tfidf_gridsearch.pkl
- name: random_forest_tfidf_gridsearch_smote
  results:
  - status: success
    metrics:
      cv_best_f1_micro: 0.59092598557871
      test_precision_micro: 0.6923300238053766
      test_recall_micro: 0.5154318319356791
      test_f1_micro: 0.59092598557871
    params:
      feature_type: tfidf
      oversampling: 'MLSMOTE (RandomOverSampler fallback)'
    dvc:
      path: random_forest_tfidf_gridsearch_smote.pkl
- name: random_forest_embedding_gridsearch
  results:
  - status: success
    metrics:
      cv_best_f1_micro: 0.6012826418169578
      test_precision_micro: 0.703060266254212
      test_recall_micro: 0.5252460640075934
      test_f1_micro: 0.6012826418169578
    params:
      feature_type: embedding
      oversampling: 'False'
    dvc:
      path: random_forest_embedding_gridsearch.pkl
- name: random_forest_embedding_gridsearch_smote
  results:
  - status: success
    metrics:
      cv_best_f1_micro: 0.5962084744755453
      test_precision_micro: 0.7031004709576139
      test_recall_micro: 0.5175288364319172
      test_f1_micro: 0.5962084744755453
    params:
      feature_type: embedding
      oversampling: 'MLSMOTE (RandomOverSampler fallback)'
    dvc:
      path: random_forest_embedding_gridsearch_smote.pkl
---

 
Model cards for committed models

Overview
- This file documents four trained model artifacts available in the repository: two TF‑IDF based Random Forest models (baseline and with oversampling) and two embedding‑based Random Forest models (baseline and with oversampling).
- For dataset provenance and preprocessing details see `data/README.md`.

1) random_forest_tfidf_gridsearch

Model details
- Name: `random_forest_tfidf_gridsearch`
- Organization: Hopcroft (se4ai2526-uniba)
- Model type: `RandomForestClassifier` wrapped in `MultiOutputClassifier` for multi-label outputs
- Branch: `Milestone-4`

Intended use
- Suitable for research and benchmarking on multi-label skill prediction for GitHub PRs/issues. Not intended for automated high‑stakes decisions or profiling individuals without further validation.

Training data and preprocessing
- Dataset: Processed SkillScope Dataset (NLBSE/SkillCompetition) as prepared for this project.
- Features: TF‑IDF (unigrams and bigrams), up to `MAX_TFIDF_FEATURES=5000`.
- Feature and label files are referenced via `get_feature_paths(feature_type='tfidf', use_cleaned=True)` in `config.py`.

Evaluation
- Reported metrics include micro‑precision, micro‑recall and micro‑F1 on a held‑out test split.
- Protocol: 80/20 multilabel‑stratified split; hyperparameters selected via 5‑fold cross‑validation optimizing `f1_micro`.
- MLflow run: `random_forest_tfidf_gridsearch` (see `hopcroft_skill_classification_tool_competition/config.py`).

Limitations and recommendations
- Trained on Java repositories; generalization to other languages is not ensured.
- Label imbalance affects rare labels; apply per‑label thresholds or further sampling strategies if required.

Usage
- Artifact path: `models/random_forest_tfidf_gridsearch.pkl`.
- Example:
  ```python
  import joblib
  model = joblib.load('models/random_forest_tfidf_gridsearch.pkl')
  y = model.predict(X_tfidf)
  ```

2) random_forest_tfidf_gridsearch_smote

Model details
- Name: `random_forest_tfidf_gridsearch_smote`
- Model type: `RandomForestClassifier` inside `MultiOutputClassifier` trained with multi‑label oversampling

Intended use
- Intended to improve recall for under‑represented labels by applying MLSMOTE (or RandomOverSampler fallback) during training.

Training and preprocessing
- Features: TF‑IDF (same configuration as the baseline).
- Oversampling: local MLSMOTE implementation when available; otherwise `RandomOverSampler`. Oversampling metadata (method and synthetic sample counts) are logged to MLflow.
- Training script: `hopcroft_skill_classification_tool_competition/modeling/train.py` (action `smote`).

Evaluation
- MLflow run: `random_forest_tfidf_gridsearch_smote`.

Limitations and recommendations
- Synthetic samples may introduce distributional artifacts; validate synthetic examples and per‑label metrics before deployment.

Usage
- Artifact path: `models/random_forest_tfidf_gridsearch_smote.pkl`.

3) random_forest_embedding_gridsearch

Model details
- Name: `random_forest_embedding_gridsearch`
- Features: sentence embeddings produced by `all-MiniLM-L6-v2` (see `config.EMBEDDING_MODEL_NAME`).

Intended use
- Uses semantic embeddings to capture contextual information from PR text; suitable for research and prototyping.

Training and preprocessing
- Embeddings generated and stored via `get_feature_paths(feature_type='embedding', use_cleaned=True)`.
- Training script: see `hopcroft_skill_classification_tool_competition/modeling/train.py`.

Evaluation
- MLflow run: `random_forest_embedding_gridsearch`.

Limitations and recommendations
- Embeddings encode dataset biases; verify performance when transferring to other repositories or languages.

Usage
- Artifact path: `models/random_forest_embedding_gridsearch.pkl`.
- Example:
  ```python
  model.predict(X_embeddings)
  ```

4) random_forest_embedding_gridsearch_smote

Model details
- Name: `random_forest_embedding_gridsearch_smote`
- Combines embedding features with multi‑label oversampling to address rare labels.

Training and evaluation
- Oversampling: MLSMOTE preferred; `RandomOverSampler` fallback if MLSMOTE is unavailable.
- MLflow run: `random_forest_embedding_gridsearch_smote`.

Limitations and recommendations
- Review synthetic examples and re‑evaluate on target data prior to deployment.

Usage
- Artifact path: `models/random_forest_embedding_gridsearch_smote.pkl`.

Publishing guidance for Hugging Face Hub
- The YAML front‑matter enables rendering on the Hugging Face Hub. Recommended repository contents for publishing:
  - `README.md` (this file)
  - model artifact(s) (`*.pkl`)
  - vectorizer(s) and label map (e.g. `tfidf_vectorizer.pkl`, `label_names.pkl`)
  - a minimal inference example or notebook

  Evaluation Data and Protocol
  - Evaluation split: an 80/20 multilabel‑stratified train/test split was used for final evaluation.
  - Cross-validation: hyperparameters were selected via 5‑fold cross‑validation optimizing `f1_micro`.
  - Test metrics reported: micro precision, micro recall, micro F1 (reported in the YAML `model-index` for each model).

  Quantitative Analyses
  - Reported unitary results: micro‑precision, micro‑recall and micro‑F1 on the held‑out test split for each model.
  - Where available, `cv_best_f1_micro` is the best cross‑validation f1_micro recorded during training; when a CV value was not present in tracking, the test F1 is used as a proxy and noted in the README.
  - Notes on comparability: TF‑IDF and embedding models are evaluated on the same held‑out splits (features differ); reported metrics are comparable for broad benchmarking but not for per‑label fairness analyses.

  How Metrics Were Computed
  - Metrics were computed using scikit‑learn's `precision_score`, `recall_score`, and `f1_score` with `average='micro'` and `zero_division=0` on the held‑out test labels and model predictions.
  - Test feature and label files used are available under `data/processed/tfidf/` and `data/processed/embedding/` (paths referenced from `hopcroft_skill_classification_tool_competition.config.get_feature_paths`).

  Ethical Considerations and Caveats
  - The dataset contains examples from Java repositories; model generalization to other languages or domains is not guaranteed.
  - Label imbalance is present; oversampling (MLSMOTE or RandomOverSampler fallback) was used in two variants to improve recall for rare labels — inspect per‑label metrics before deploying.
  - The models and README are intended for research and benchmarking. They are not validated for safety‑critical or high‑stakes automated decisioning.