Update README.md
Browse files
README.md
CHANGED
|
@@ -10,36 +10,18 @@ tags:
|
|
| 10 |
|
| 11 |
# Why? What? TL;DR?
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
# Testing Environment
|
| 17 |
-
|
| 18 |
-
- Frontend is staging version of Silly Tavern.
|
| 19 |
-
- Backend is the latest version of KoboldCPP for Windows using CUDA 12.
|
| 20 |
-
- Using **CuBLAS** but **not using QuantMatMul (mmq)**.
|
| 21 |
-
- Fixed Seed for all tests: **123**
|
| 22 |
-
- **7-10B Models:**
|
| 23 |
-
- All models are loaded in Q8_0 (GGUF)
|
| 24 |
-
- **Flash Attention** and **ContextShift** enabled.
|
| 25 |
-
- All models are extended to **16K context length** (auto rope from KCPP)
|
| 26 |
-
- Response size set to 1024 tokens max.
|
| 27 |
-
- **11-15B Models:**
|
| 28 |
-
- All models are loaded in Q4_KM or whatever is the highest/closest available (GGUF)
|
| 29 |
-
- **Flash Attention** and **8Bit cache compression** are enabled.
|
| 30 |
-
- All models are extended to **12K context length** (auto rope from KCPP)
|
| 31 |
-
- Response size set to 512 tokens max.
|
| 32 |
|
|
|
|
| 33 |
|
| 34 |
-
#
|
| 35 |
|
| 36 |
-
|
| 37 |
-
- All models are tested in whichever instruct format they are supposed to be comfortable with (as long as it's ChatML or L3 Instruct)
|
| 38 |
|
|
|
|
| 39 |
|
| 40 |
-
# Available Tests
|
| 41 |
|
| 42 |
-
##
|
| 43 |
|
| 44 |
The goal of this test, featuring a dog (Rex) and his owner (EsKa), is to determine if a model is good at obeying a system prompt and character card. The trick being that dogs can't talk, but LLM love to.
|
| 45 |
|
|
@@ -47,19 +29,7 @@ The goal of this test, featuring a dog (Rex) and his owner (EsKa), is to determi
|
|
| 47 |
- [Files, cards and settings can be found here](https://huggingface.co/SerialKicked/ModelTestingBed/tree/main/DoggoEval)
|
| 48 |
- TODO: Charts and screenshots
|
| 49 |
|
| 50 |
-
### MinotaurEval
|
| 51 |
-
|
| 52 |
-
TODO: The goal of this test is to check if a model is able of following a very specific prompting method and maintain situational awareness in the smallest labyrinth in the world.
|
| 53 |
-
|
| 54 |
-
- Discussions will be hosted here.
|
| 55 |
-
- Files and cards will be available soon (tm).
|
| 56 |
-
|
| 57 |
-
### TimeEval
|
| 58 |
-
|
| 59 |
-
TODO: The goal of this test is to see if the bot is able to behave in 16K context, recall and summarise "old" info accurately.
|
| 60 |
|
| 61 |
-
- Discussions will be hosted here.
|
| 62 |
-
- Files and cards will be available soon (tm).
|
| 63 |
|
| 64 |
|
| 65 |
# Limitations
|
|
|
|
| 10 |
|
| 11 |
# Why? What? TL;DR?
|
| 12 |
|
| 13 |
+
Various tests on, well, various LLM.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
+
# Available Tests
|
| 16 |
|
| 17 |
+
## LLM Drawing Test (2026 ongoing)
|
| 18 |
|
| 19 |
+
This test is meant to evaluate models in a difficult task requiring a competency in spatial awareness, image recognition, tools calls, and creativity. In a new empty chat, the model is asked to copy the provided image to the best of its abilities. Model is then evaluated on accuracy of function calls (fail rate) and the resemblance to the original drawing.
|
|
|
|
| 20 |
|
| 21 |
+
- [Results](https://huggingface.co/SerialKicked/ModelTestingBed/discussions/4)
|
| 22 |
|
|
|
|
| 23 |
|
| 24 |
+
## DoggoEval (Done)
|
| 25 |
|
| 26 |
The goal of this test, featuring a dog (Rex) and his owner (EsKa), is to determine if a model is good at obeying a system prompt and character card. The trick being that dogs can't talk, but LLM love to.
|
| 27 |
|
|
|
|
| 29 |
- [Files, cards and settings can be found here](https://huggingface.co/SerialKicked/ModelTestingBed/tree/main/DoggoEval)
|
| 30 |
- TODO: Charts and screenshots
|
| 31 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
|
|
|
|
|
|
| 33 |
|
| 34 |
|
| 35 |
# Limitations
|