| | --- |
| | language: |
| | - en |
| | - fr |
| | - ro |
| | - de |
| | license: apache-2.0 |
| | library_name: transformers |
| | datasets: |
| | - c4 |
| | --- |
| | |
| | # Model Card for EncT5 |
| |
|
| | EncT5 is a variant of T5 that utilizes mainly the encoder for non-autoregressive (ie. classification and regression) |
| | tasks. The model is from the paper [Fine-tuning T5 Encoder for Non-autoregressive Tasks](https://arxiv.org/abs/2110.08426) |
| | by Frederick Liu, Terry Huang, Shihang Lyu, Siamak Shakeri, Hongkun Yu, Jing Li |
| |
|
| |
|
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| |
|
| | EncT5 uses the same base weights at T5, but **must be fine-tuning before use**. There are several special features |
| | to EncT5: |
| |
|
| | 1. There are less decoder layers (a single decoder layer by default), and so has fewer parameters/resources than the |
| | standard T5. |
| | 3. There is a separate decoder word embedding, with the decoder input ids being predefined constants. During |
| | fine-tuning, the decoder embedding learns to use these constants as "prompts" to the encoder for the corresponding |
| | classification/regression tasks. |
| | 5. There is a classification head on top of the decoder output. |
| |
|
| | Research has shown that this model can be more efficient and usable over T5 and BERT for non-autoregressive |
| | tasks such as classification and regression. |
| |
|
| | - **Developed by:** Frederick Liu, Terry Huang, Shihang Lyu, Siamak Shakeri, Hongkun Yu, Jing Li. See the |
| | [associated paper](https://arxiv.org/abs/2110.08426). |
| | - **Model type:** Language Model |
| | - **Language(s) (NLP):** English, French, Romanian, German |
| | - **License:** Apache 2.0 |
| | - **Based on model:** [T5](https://huggingface.co/google-t5/t5-base) |
| | - **Repository:** [Github repro](https://github.com/hackyon/EncT5) |
| | - **Paper:** [Fine-tuning T5 Encoder for Non-autoregressive Tasks](https://arxiv.org/abs/2110.08426) |
| |
|
| | ## How to Get Started with the Model |
| |
|
| | Use the code below to get started with the model. |
| |
|
| | ```python |
| | model = AutoModelForSequenceClassification.from_pretrained("hackyon/enct5-base", trust_remote_code=True) |
| | # Fine-tune the model before use. |
| | ``` |
| |
|
| | See the [github repro](https://github.com/hackyon/EncT5) for a more comprehensive guide. |
| |
|
| | ## Training Details |
| |
|
| | ### Training Data |
| |
|
| | The weights of this model are directly copied from [t5-base](https://huggingface.co/google-t5/t5-base). |
| |
|
| | ### Training Procedure |
| |
|
| | This model **must be fine-tuned** before use. The decoder word embedding and classification head are both untrained. |
| |
|
| |
|
