| | --- |
| | base_model: katanemo/Arch-Router-1.5B |
| | base_model_relation: quantized |
| | quantized_by: ArtusDev |
| | --- |
| | <style> |
| | .container-dark { |
| | font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; |
| | line-height: 1.6; |
| | color: #d4d4d4; |
| | } |
| | a { |
| | color: #569cd6; |
| | text-decoration: none; |
| | font-weight: 600; |
| | } |
| | a:hover { |
| | text-decoration: underline; |
| | } |
| | .card-dark { |
| | background-color: #252526; |
| | border-radius: 12px; |
| | padding: 24px; |
| | margin-bottom: 20px; |
| | box-shadow: 0 4px 12px rgba(0,0,0,0.3); |
| | border: 1px solid #3c3c3c; |
| | } |
| | .card-dark h1 { |
| | font-size: 2.2em; |
| | color: #ffffff; |
| | text-align: center; |
| | margin-bottom: 10px; |
| | } |
| | .card-dark.card-dark-title h1 { |
| | font-size: 1.5em; |
| | } |
| | .card-dark .subtitle { |
| | text-align: center; |
| | font-size: 1.1em; |
| | color: #a0a0a0; |
| | } |
| | .card-dark h2 { |
| | font-size: 1.5em; |
| | margin-top: 0; |
| | padding-bottom: 10px; |
| | border-bottom: 1px solid #3c3c3c; |
| | color: #c586c0; |
| | } |
| | .card-dark h3 { |
| | font-size: 1.2em; |
| | color: #d4d4d4; |
| | } |
| | .styled-table { |
| | display: table; |
| | border: none; |
| | width: 100%; |
| | font-size: 0.95em; |
| | margin-bottom: 0px; |
| | } |
| | .styled-table thead th { |
| | background-color: #333333; |
| | color: #c586c0; |
| | text-align: left; |
| | } |
| | .styled-table th { |
| | padding: 12px 15px; |
| | } |
| | .styled-table td { |
| | padding: 0; |
| | } |
| | .styled-table table, .styled-table th, .styled-table td { |
| | border-left: none; |
| | border-right: none; |
| | border-bottom: none; |
| | } |
| | .styled-table td { |
| | border-bottom: 1px solid #3c3c3c; |
| | } |
| | .styled-table tbody tr { |
| | transition: background-color 0.1s ease; |
| | } |
| | .styled-table tbody tr:hover { |
| | background-color: #3a3a3a; |
| | } |
| | .styled-table tr:last-child td { |
| | border-bottom: none; |
| | } |
| | .styled-table td a { |
| | display: block; |
| | padding: 12px 15px; |
| | } |
| | .styled-table td a.fake-link { |
| | text-decoration:none; |
| | color:inherit; |
| | } |
| | details { |
| | margin-top: 20px; |
| | border: 1px solid #3c3c3c; |
| | border-radius: 8px; |
| | overflow: hidden; |
| | } |
| | summary { |
| | cursor: pointer; |
| | padding: 12px 18px; |
| | background-color: #6A5ACD; |
| | font-weight: 600; |
| | display: flex; |
| | align-items: center; |
| | gap: 10px; |
| | justify-content: space-between; |
| | list-style: none; |
| | } |
| | summary::-webkit-details-marker { |
| | display: none; |
| | } |
| | summary:hover { |
| | filter: brightness(1.1); |
| | } |
| | summary::after { |
| | content: ''; |
| | display: inline-block; |
| | width: 8px; |
| | height: 8px; |
| | border-bottom: 2px solid white; |
| | border-right: 2px solid white; |
| | transform: rotate(45deg); |
| | transition: transform 0.3s ease; |
| | } |
| | details[open] > summary::after { |
| | transform: rotate(225deg); |
| | } |
| | .details-content { |
| | padding: 18px; |
| | } |
| | .btn-purple { |
| | display: inline-block; |
| | background-color: #6A5ACD; |
| | color: white !important; |
| | padding: 12px 24px; |
| | border-radius: 8px; |
| | text-decoration: none; |
| | font-weight: 600; |
| | transition: background-color 0.3s ease, transform 0.2s ease; |
| | text-align: center; |
| | } |
| | .btn-purple:hover { |
| | background-color: #7B68EE; |
| | transform: translateY(-2px); |
| | } |
| | </style> |
| | |
| | <div class="container-dark"> |
| |
|
| | <div class="card-dark card-dark-title"> |
| | <h1>ArtusDev/katanemo_Arch-Router-1.5B-EXL3</h1> |
| | <p class="subtitle"> |
| | EXL3 quants of <a href="https://huggingface.co/katanemo/Arch-Router-1.5B" target="_blank">katanemo/Arch-Router-1.5B</a> using <a href="https://github.com/turboderp-org/exllamav3/" target="_blank">exllamav3</a> for quantization. |
| | </p> |
| | </div> |
| | |
| | <div class="card-dark"> |
| | <h2>Quants</h2> |
| | <table class="styled-table"> |
| | <thead> |
| | <tr> |
| | <th>Quant</th> |
| | <th>BPW</th> |
| | <th>Head Bits</th> |
| | <th>Size (GB)</th> |
| | </tr> |
| | </thead> |
| | <tbody> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/2.5bpw_H6" target="_blank">2.5_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/2.5bpw_H6" target="_blank">2.5</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/2.5bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/2.5bpw_H6" target="_blank">1.07</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.0bpw_H6" target="_blank">3.0_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.0bpw_H6" target="_blank">3.0</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.0bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.0bpw_H6" target="_blank">1.16</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.5bpw_H6" target="_blank">3.5_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.5bpw_H6" target="_blank">3.5</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.5bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/3.5bpw_H6" target="_blank">1.24</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.0bpw_H6" target="_blank">4.0_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.0bpw_H6" target="_blank">4.0</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.0bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.0bpw_H6" target="_blank">1.32</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.5bpw_H6" target="_blank">4.5_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.5bpw_H6" target="_blank">4.5</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.5bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/4.5bpw_H6" target="_blank">1.40</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/5.0bpw_H6" target="_blank">5.0_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/5.0bpw_H6" target="_blank">5.0</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/5.0bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/5.0bpw_H6" target="_blank">1.48</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/6.0bpw_H6" target="_blank">6.0_H6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/6.0bpw_H6" target="_blank">6.0</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/6.0bpw_H6" target="_blank">6</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/6.0bpw_H6" target="_blank">1.65</a></td> |
| | </tr> |
| | <tr> |
| | <td><a href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/8.0bpw_H8" target="_blank">8.0_H8</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/8.0bpw_H8" target="_blank">8.0</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/8.0bpw_H8" target="_blank">8</a></td> |
| | <td><a class="fake-link" href="https://huggingface.co/ArtusDev/katanemo_Arch-Router-1.5B-EXL3/tree/8.0bpw_H8" target="_blank">2.03</a></td> |
| | </tr> |
| | </tbody> |
| | </table> |
| | </div> |
| | |
| | <div class="card-dark"> |
| | <h2>How to Download and Use Quants</h2> |
| | <p>You can download quants by targeting specific size using the Hugging Face CLI.</p> |
| | <details> |
| | <summary>Click for download commands</summary> |
| | <div class="details-content"> |
| | <b>1. Install huggingface-cli:</b> |
| | <pre><code>pip install -U "huggingface_hub[cli]"</code></pre> |
| | <b>2. Download a specific quant:</b> |
| | <pre><code>huggingface-cli download ArtusDev/katanemo_Arch-Router-1.5B-EXL3 --revision "5.0bpw_H6" --local-dir ./</code></pre> |
| | </div> |
| | </details> |
| | <p>EXL3 quants can be run with any inference client that supports EXL3, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p> |
| | </div> |
| | |
| | <div class="card-dark"> |
| | <h2>Quant Requests</h2> |
| | <div style="text-align: center; margin-top: 25px;"> |
| | <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quants</a> |
| | </div> |
| | <p class="subtitle"> |
| | See <a href="https://huggingface.co/ArtusDev/requests-exl" target="_blank">EXL community hub</a> for request guidelines. |
| | </p> |
| | </div> |
| | |
| | <div class="card-dark"> |
| | <h2>Acknowledgements</h2> |
| | <p style="text-align: center; font-size: 1.1em; margin-top: 10px;"> |
| | Made possible with cloud compute from <a href="https://lium.io/" target="_blank"><b>lium.io</b></a> |
| | </p> |
| | </div> |
| | |
| | </div> |
| |
|
