| | --- |
| | license: apache-2.0 |
| | language: |
| | - en |
| | metrics: |
| | - accuracy |
| | library_name: transformers |
| | pipeline_tag: text-classification |
| | tags: |
| | - customer-service-tickets |
| | - github-issues |
| | - bart-large-mnli |
| | - zero-shot-classification |
| | - NLP |
| | widget: |
| | - text: "Sign up form is not working" |
| | example_title: "Example 1" |
| | - text: "json and yaml support" |
| | example_title: "Example 2" |
| | - text: "fullscreen and tabs media key don't do what they should" |
| | example_title: "Example 2" |
| | --- |
| | |
| | # GitHub issues classifier (using zero shot classification) |
| |
|
| | Predicts wether a statement is a feature request, issue/bug or question |
| |
|
| | This model was trained using the [**Zero-shot classifier distillation**](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) method |
| | with the [BART-large-mnli](https://huggingface.co/facebook/bart-large-mnli) model as teacher model, to train a classifier on Github issues from the [Github Issues Prediction dataset](https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction) |
| |
|
| | ## Labels |
| |
|
| | As per the dataset Kaggle competition, the classifier predicts wether an issue is a bug, feature or question. After playing around with different labels pre-training I've used a different mapping |
| | of labels that yielded better predictions (see notebook [here](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for details), labels being |
| |
|
| | * issue |
| | * feature request |
| | * question |
| |
|
| |
|
| | ## Training data |
| |
|
| | * 15k of Github issues titles ("unlabeled_titles_simple.txt") |
| | * Hypothesis used: "This request is a {}" |
| | * Teacher model used: valhalla/distilbart-mnli-12-1 |
| | * Studend model used: distilbert-base-uncased |
| |
|
| | ## Results |
| |
|
| | Agreement of student and teacher predictions: **94.82%** |
| |
|
| | See [this notebook](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for more info on feature engineering choice made |
| |
|
| |
|
| | ## How to train using your own dataset |
| | * Download training dataset from https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction |
| | * Modify and run convert.py, updating the paths to convert to a CSV |
| | * Run distill.py with the csv file (see [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) for more info) |
| |
|
| | ## Acknowledgements |
| |
|
| | * Joe Davison and his article on [Zero-Shot Learning in Modern NLP](https://joeddav.github.io/blog/2020/05/29/ZSL.html) |
| | * Jeremy Howard, fast.ai and his notebook [Iterate like a grandmaster](https://www.kaggle.com/code/antoinemacia/iterate-like-a-grandmaster) |
