Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Antrenarea unui nou tokenizer dintr-unul vechi","local":"training-a-new-tokenizer-from-an-old-one","sections":[{"title":"Asamblarea unui corpus","local":"assembling-a-corpus","sections":[],"depth":2},{"title":"Antrenarea unui nou tokenizer","local":"training-a-new-tokenizer","sections":[],"depth":2},{"title":"Salvarea tokenizerului","local":"saving-the-tokenizer","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/course/pr_1069/rum/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/entry/start.1de7c3d2.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/scheduler.37c15a92.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/singletons.e13b7dfd.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/index.18351ede.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/paths.e130b7b0.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/entry/app.1f82014c.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/index.2bf4358c.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/nodes/0.3c83e1ab.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/nodes/67.7c578c3d.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/Tip.363c041f.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/Youtube.1e50a667.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/CodeBlock.4e987730.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/CourseFloatingBanner.6add7356.js"> | |
| <link rel="modulepreload" href="/docs/course/pr_1069/rum/_app/immutable/chunks/getInferenceSnippets.24b50994.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Antrenarea unui nou tokenizer dintr-unul vechi","local":"training-a-new-tokenizer-from-an-old-one","sections":[{"title":"Asamblarea unui corpus","local":"assembling-a-corpus","sections":[],"depth":2},{"title":"Antrenarea unui nou tokenizer","local":"training-a-new-tokenizer","sections":[],"depth":2},{"title":"Salvarea tokenizerului","local":"saving-the-tokenizer","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="training-a-new-tokenizer-from-an-old-one" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#training-a-new-tokenizer-from-an-old-one"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Antrenarea unui nou tokenizer dintr-unul vechi</span></h1> <div class="flex space-x-1 absolute z-10 right-0 top-0"><a href="https://discuss.huggingface.co/t/chapter-6-questions" target="_blank"><img alt="Ask a Question" class="!m-0" src="https://img.shields.io/badge/Ask%20a%20question-ffcb4c.svg?logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgLTEgMTA0IDEwNiI+PGRlZnM+PHN0eWxlPi5jbHMtMXtmaWxsOiMyMzFmMjA7fS5jbHMtMntmaWxsOiNmZmY5YWU7fS5jbHMtM3tmaWxsOiMwMGFlZWY7fS5jbHMtNHtmaWxsOiMwMGE5NGY7fS5jbHMtNXtmaWxsOiNmMTVkMjI7fS5jbHMtNntmaWxsOiNlMzFiMjM7fTwvc3R5bGU+PC9kZWZzPjx0aXRsZT5EaXNjb3Vyc2VfbG9nbzwvdGl0bGU+PGcgaWQ9IkxheWVyXzIiPjxnIGlkPSJMYXllcl8zIj48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik01MS44NywwQzIzLjcxLDAsMCwyMi44MywwLDUxYzAsLjkxLDAsNTIuODEsMCw1Mi44MWw1MS44Ni0uMDVjMjguMTYsMCw1MS0yMy43MSw1MS01MS44N1M4MCwwLDUxLjg3LDBaIi8+PHBhdGggY2xhc3M9ImNscy0yIiBkPSJNNTIuMzcsMTkuNzRBMzEuNjIsMzEuNjIsMCwwLDAsMjQuNTgsNjYuNDFsLTUuNzIsMTguNEwzOS40LDgwLjE3YTMxLjYxLDMxLjYxLDAsMSwwLDEzLTYwLjQzWiIvPjxwYXRoIGNsYXNzPSJjbHMtMyIgZD0iTTc3LjQ1LDMyLjEyYTMxLjYsMzEuNiwwLDAsMS0zOC4wNSw0OEwxOC44Niw4NC44MmwyMC45MS0yLjQ3QTMxLjYsMzEuNiwwLDAsMCw3Ny40NSwzMi4xMloiLz48cGF0aCBjbGFzcz0iY2xzLTQiIGQ9Ik03MS42MywyNi4yOUEzMS42LDMxLjYsMCwwLDEsMzguOCw3OEwxOC44Niw4NC44MiwzOS40LDgwLjE3QTMxLjYsMzEuNiwwLDAsMCw3MS42MywyNi4yOVoiLz48cGF0aCBjbGFzcz0iY2xzLTUiIGQ9Ik0yNi40Nyw2Ny4xMWEzMS42MSwzMS42MSwwLDAsMSw1MS0zNUEzMS42MSwzMS42MSwwLDAsMCwyNC41OCw2Ni40MWwtNS43MiwxOC40WiIvPjxwYXRoIGNsYXNzPSJjbHMtNiIgZD0iTTI0LjU4LDY2LjQxQTMxLjYxLDMxLjYxLDAsMCwxLDcxLjYzLDI2LjI5YTMxLjYxLDMxLjYxLDAsMCwwLTQ5LDM5LjYzbC0zLjc2LDE4LjlaIi8+PC9nPjwvZz48L3N2Zz4="></a> <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter6/section2.ipynb" target="_blank"><img alt="Open In Colab" class="!m-0" src="https://colab.research.google.com/assets/colab-badge.svg"></a> <a href="https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter6/section2.ipynb" target="_blank"><img alt="Open In Studio Lab" class="!m-0" src="https://studiolab.sagemaker.aws/studiolab.svg"></a></div> <p data-svelte-h="svelte-854u19">Dacă un model de limbaj nu este disponibil în limba dorită sau dacă corpusul tău este foarte diferit de cel pe care modelul de limbaj a fost antrenat, este probabil că veți dori să antrenați modelul de la zero, folosind un tokenizer adaptat datelor tale. Acest lucru va necesita antrenarea unui nou tokenizer pe datasetul tău. Dar ce înseamnă exact asta? Când am examinat pentru prima dată tokenizatorii în <a href="/course/chapter2">Capitolul 2</a>, am văzut că majoritatea modelelor Transformer folosesc un <em>algoritm de subword tokenization</em>. Pentru a identifica care subcuvinte sunt de interes și apar cel mai frecvent în corpusul respectiv, tokenizerul trebuie să examineze cu atenție toate textele din corpus - un proces pe care îl numim <em>antrenare</em>. Regulile exacte care conduc această antrenare depind de tipul de tokenizer utilizat și vom prezenta cei trei algoritmi principali mai târziu în acest capitol.</p> <iframe class="w-full xl:w-4/6 h-80" src="https://www.youtube-nocookie.com/embed/DJimQynXZsQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-1fjtbr6">⚠️ Antrenarea unui tokenizer nu este același lucru ca antrenarea unui model! Antrenarea modelului folosește stochastic gradient descent pentru a face pierderea puțin mai mică pentru fiecare batch. Este randomizată prin natură (ceea ce înseamnă că trebuie să setați niște seeduri pentru a obține aceleași rezultate atunci când faceți aceeași antrenare de două ori). Antrenarea unui tokenizer este un proces statistic care încearcă să identifice care subcuvinte sunt cele mai bune pentru a fi selectate pentru un anumit corpus, și regulile exacte utilizate pentru a le selecta depind de algoritmul de tokenizare. Este determinist, ceea ce înseamnă că întotdeauna obțineți aceleași rezultate atunci când antrenați cu același algoritm pe același corpus.</p></div> <h2 class="relative group"><a id="assembling-a-corpus" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#assembling-a-corpus"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Asamblarea unui corpus</span></h2> <p data-svelte-h="svelte-b2mhww">Există o interfață API foarte simplă în 🤗 Transformers pe care o puteți utiliza pentru a antrena un nou tokenizer cu aceleași caracteristici ca unul existent: <code>AutoTokenizer.train_new_from_iterator()</code>. Pentru a vedea acest lucru în acțiune, să zicem că vrem să antrenăm GPT-2 de la zero, dar într-o altă limbă decât engleza. Prima noastră sarcină va fi să adunăm multe date în acea limbă într-un corpus de antrenare. Pentru a oferi exemple pe care toată lumea le poate înțelege, nu vom folosi o limbă ca rusă sau chineza aici, ci mai degrabă o limbă engleză specializată: codul Python.</p> <p data-svelte-h="svelte-zlu0hu">Biblioteca <a href="https://github.com/huggingface/datasets" rel="nofollow">🤗 Datasets</a> ne poate ajuta să asamblăm un corpus de cod sursă Python. Vom folosi funcția obișnuită <code>load_dataset()</code> pentru a descărca și a păstra în cache datasetul <a href="https://huggingface.co/datasets/code_search_net" rel="nofollow">CodeSearchNet</a>. Acest dataset a fost creat pentru <a href="https://wandb.ai/github/CodeSearchNet/benchmark" rel="nofollow">Provocarea CodeSearchNet</a> și conține milioane de funcții din biblioteci open-source de pe GitHub în mai multe limbaje de programare. Aici, vom încărca partea Python a acestui dataset:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-comment"># Acest lucru poate dura câteva minute pentru a încărca, așa că luați o pauză și beți o ceașcă de cafea sau ceai în timp ce așteptați!</span> | |
| raw_datasets = load_dataset(<span class="hljs-string">"code_search_net"</span>, <span class="hljs-string">"python"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-dypd3i">Putem să ne uităm la splitul de antrenare pentru a vedea la care coloane avem acces:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->raw_datasets[<span class="hljs-string">"train"</span>]<!-- HTML_TAG_END --></pre></div> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->Dataset({ | |
| features: [<span class="hljs-string">'repository_name'</span>, <span class="hljs-string">'func_path_in_repository'</span>, <span class="hljs-string">'func_name'</span>, <span class="hljs-string">'whole_func_string'</span>, <span class="hljs-string">'language'</span>, | |
| <span class="hljs-string">'func_code_string'</span>, <span class="hljs-string">'func_code_tokens'</span>, <span class="hljs-string">'func_documentation_string'</span>, <span class="hljs-string">'func_documentation_tokens'</span>, <span class="hljs-string">'split_name'</span>, | |
| <span class="hljs-string">'func_code_url'</span> | |
| ], | |
| num_rows: <span class="hljs-number">412178</span> | |
| })<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-6p6w6o">Putem vedea că dataset-ul separă docstringurile de cod și sugerează o tokenizare a ambelor. Aici, vom folosi doar coloana <code>whole_func_string</code> pentru a antrena tokenizerul nostru. Putem să ne uităm la un exemplu al unei astfel de funcții prin indexarea în splitul de antrenare:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-built_in">print</span>(raw_datasets[<span class="hljs-string">"train"</span>][<span class="hljs-number">123456</span>][<span class="hljs-string">"whole_func_string"</span>])<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-xo26nw">care ar trebui să printeze următorul lucru:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">handle_simple_responses</span>(<span class="hljs-params"> | |
| self, timeout_ms=<span class="hljs-literal">None</span>, info_cb=DEFAULT_MESSAGE_CALLBACK</span>): | |
| <span class="hljs-string">"""Accepts normal responses from the device. | |
| Args: | |
| timeout_ms: Timeout in milliseconds to wait for each response. | |
| info_cb: Optional callback for text sent from the bootloader. | |
| Returns: | |
| OKAY packet's message. | |
| """</span> | |
| <span class="hljs-keyword">return</span> self._accept_responses(<span class="hljs-string">'OKAY'</span>, info_cb, timeout_ms=timeout_ms)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1ky6max">Primul lucru pe care trebuie să-l facem este să transformăm setul de date într-un <em>iterator</em> de liste de texte - de exemplu, o listă de liste de texte. Utilizarea listelor de texte va permite tokenizerului nostru să funcționeze mai rapid (antrenându-se pe batch-uri de texte în loc de a procesa texte individuale unul câte unul), iar acesta ar trebui să fie un iterator dacă dorim să evităm să avem tot în memoria RAM deodată. Dacă corpusul tău este uriaș, veți dori să profitați de faptul că 🤗 Datasets nu încarcă totul în memoria RAM, ci stochează elementele datasetului pe disc.</p> <p data-svelte-h="svelte-1574z9b">Următoarea operație ar crea o listă de liste de 1.000 de texte fiecare, dar ar încărca totul în memorie:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Nu faceți uncomment următoarei linii dacă datasetul vostru este mare, ci doar dacă este mic!</span> | |
| <span class="hljs-comment"># training_corpus = [raw_datasets["train"][i: i + 1000]["whole_func_string"] for i in range(0, len(raw_datasets["train"]), 1000)]</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1fn9yfp">Utilizând un generator Python, putem evita ca Python să încarce orice în memorie până când este realmente necesar. Pentru a crea un astfel de generator, trebuie doar să înlocuiți parantezele pătrate cu paranteze rotunde:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->training_corpus = ( | |
| raw_datasets[<span class="hljs-string">"train"</span>][i : i + <span class="hljs-number">1000</span>][<span class="hljs-string">"whole_func_string"</span>] | |
| <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(raw_datasets[<span class="hljs-string">"train"</span>]), <span class="hljs-number">1000</span>) | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-uzz71g">Această linie de cod nu extrage niciun element al setului de date; doar creează un obiect pe care îl puteți utiliza într-un <code>for</code> loop din Python. Textele vor fi încărcate doar atunci când veți avea nevoie de ele(adică atunci când sunteți la pasul loopului <code>for</code> care le solicită), iar doar 1.000 de texte vor fi încărcate la un moment dat. Acest mod vă permite să nu epuizați memoria RAM chiar dacă prelucrați un set de date uriaș.</p> <p data-svelte-h="svelte-24rfb3">Problema cu un obiect generator este că poate fi utilizat doar o dată, așadar, în loc să ne ofere lista primelor 10 cifre de două ori:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->gen = (i <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">10</span>)) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-built_in">list</span>(gen)) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-built_in">list</span>(gen))<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1e38ip1">noi le primim o dată și apoi o listă goală:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->[<span class="hljs-number">0</span>, <span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>, <span class="hljs-number">6</span>, <span class="hljs-number">7</span>, <span class="hljs-number">8</span>, <span class="hljs-number">9</span>] | |
| []<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1r75idm">Din acest motive noi definim o funcție ce returnează în schimb un generator:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_training_corpus</span>(): | |
| <span class="hljs-keyword">return</span> ( | |
| raw_datasets[<span class="hljs-string">"train"</span>][i : i + <span class="hljs-number">1000</span>][<span class="hljs-string">"whole_func_string"</span>] | |
| <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(raw_datasets[<span class="hljs-string">"train"</span>]), <span class="hljs-number">1000</span>) | |
| ) | |
| training_corpus = get_training_corpus()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-17wr5xd">În același timp poți defini un generator înăuntrul unui <code>for</code> loop folosing statementul <code>yield</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_training_corpus</span>(): | |
| dataset = raw_datasets[<span class="hljs-string">"train"</span>] | |
| <span class="hljs-keyword">for</span> start_idx <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(dataset), <span class="hljs-number">1000</span>): | |
| samples = dataset[start_idx : start_idx + <span class="hljs-number">1000</span>] | |
| <span class="hljs-keyword">yield</span> samples[<span class="hljs-string">"whole_func_string"</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-u3ibn8">ceea ce va produce același generator ca înainte, dar îți va permite să folosești o logică mai complexă decât cea pe care ai putea să o folosești într-un list comprehension.</p> <h2 class="relative group"><a id="training-a-new-tokenizer" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#training-a-new-tokenizer"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Antrenarea unui nou tokenizer</span></h2> <p data-svelte-h="svelte-1bpcz72">Acum că avem corpusul nostru sub forma unui iterator de batch-uri de texte, suntem gata să antrenăm un nou tokenizer. Pentru a face acest lucru, trebuie mai întâi să încărcăm tokenizerul pe care dorim să-l asociem cu modelul nostru (aici, GPT-2):</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer | |
| old_tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"gpt2"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-14tyssm">Chiar dacă urmează să antrenăm un nou tokenizer, este o idee bună să facem acest lucru pentru a evita să începem să facem tot de la zero. Astfel, nu vom fi nevoiți să specificăm nimic despre algoritmul de tokenizare sau despre special tokens pe care îi vom utiliza; noul nostru tokenizer va fi exact la fel ca GPT-2, iar singur lucrul care se va schimba este vocabularul, care va fi determinat de antrenarea pe corpusul nostru.</p> <p data-svelte-h="svelte-mxnwdx">Mai întâi, hai să vedem cum ar interpreta acest tokenizer un exemplu de funcție:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->example = <span class="hljs-string">'''def add_numbers(a, b): | |
| """Add the two numbers `a` and `b`.""" | |
| return a + b'''</span> | |
| tokens = old_tokenizer.tokenize(example) | |
| tokens<!-- HTML_TAG_END --></pre></div> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->[<span class="hljs-string">'def'</span>, <span class="hljs-string">'Ġadd'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'n'</span>, <span class="hljs-string">'umbers'</span>, <span class="hljs-string">'('</span>, <span class="hljs-string">'a'</span>, <span class="hljs-string">','</span>, <span class="hljs-string">'Ġb'</span>, <span class="hljs-string">'):'</span>, <span class="hljs-string">'Ċ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġ"""'</span>, <span class="hljs-string">'Add'</span>, <span class="hljs-string">'Ġthe'</span>, <span class="hljs-string">'Ġtwo'</span>, | |
| <span class="hljs-string">'Ġnumbers'</span>, <span class="hljs-string">'Ġ`'</span>, <span class="hljs-string">'a'</span>, <span class="hljs-string">'`'</span>, <span class="hljs-string">'Ġand'</span>, <span class="hljs-string">'Ġ`'</span>, <span class="hljs-string">'b'</span>, <span class="hljs-string">'`'</span>, <span class="hljs-string">'."'</span>, <span class="hljs-string">'""'</span>, <span class="hljs-string">'Ċ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġ'</span>, <span class="hljs-string">'Ġreturn'</span>, <span class="hljs-string">'Ġa'</span>, <span class="hljs-string">'Ġ+'</span>, <span class="hljs-string">'Ġb'</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-17o6vop">Acest tokenizer are câteva simboluri speciale, cum ar fi <code>Ġ</code> și <code>Ċ</code>, care denotă spații și noi linii, respectiv. Așa cum se poate vedea, acest lucru nu este prea eficient: tokenizerul returnează tokenuri individuale pentru fiecare spațiu, când ar putea grupa împreună nivelurile de indentare (deoarece având seturi de patru sau opt spații va fi foarte comun în cod). De asemenea, a divizat numele funcției într-un mod ciudat, nefiind obișnuit să vadă cuvinte care conțin caracterele <code>_</code>.</p> <p data-svelte-h="svelte-16r8h8v">Acum hai să antrenăm un nou tokenizer și să vedem dacă rezolvă aceste probleme. Pentru aceasta, vom utiliza metoda <code>train_new_from_iterator()</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokenizer = old_tokenizer.train_new_from_iterator(training_corpus, <span class="hljs-number">52000</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-q16oh5">Această comandă ar putea dura puțin timp dacă corpusul este foarte mare, dar pentru acest dataset de 1,6 GB de texte este extrem de rapid (1 minut și 16 secunde pe un procesor AMD Ryzen 9 3900X cu 12 nuclee).</p> <p data-svelte-h="svelte-t4gk7m">Reține că <code>AutoTokenizer.train_new_from_iterator()</code> funcționează doar dacă tokenizerul pe care îl utilizați este un tokenizer “rapid”. Așa cum veți vedea în următoarea secțiune, biblioteca 🤗 Transformers conține două tipuri de tokenizeri: unii sunt scriși în pur Python, iar alții (cei rapizi) sunt susținuți de biblioteca 🤗 Tokenizers, care este scrisă în limbajul de programare Rust. Python este limbajul cel mai frecvent utilizat pentru aplicații de data science și deep learning, dar atunci când orice trebuie să fie paralelizat pentru a fi rapid, trebuie să fie scris într-un alt limbaj de programare. De exemplu, multiplicările matricelor care sunt la baza calculelor modelului sunt scrise în CUDA, o bibliotecă C optimizată pentru GPU-uri.</p> <p data-svelte-h="svelte-1tsuh93">Antrenarea unui tokenizer nou în pur Python ar fi extrem de lent, de aceea am dezvoltat biblioteca 🤗 Tokenizers. Reține că, la fel cum nu a trebuit să învățați limbajul CUDA pentru a putea executa modelul pe un batch de inputuri pe un GPU, nu veți avea nevoie să învățați Rust pentru a utiliza un tokenizer rapid. Biblioteca 🤗 Tokenizers oferă legături Python pentru multe metode care apelează intern unele bucăți de cod în Rust; de exemplu, pentru a paraleliza antrenarea noului tokenizer sau, așa cum am văzut în <a href="/course/chapter3">Capitolul 3</a>, tokenizarea unui batch de inputuri.</p> <p data-svelte-h="svelte-1l022ld">Majoritatea modelelor Transformer au un tokenizer rapid disponibil (există unele excepții pe care le puteți verifica <a href="https://huggingface.co/transformers/#supported-frameworks" rel="nofollow">aici</a>), iar API-ul <code>AutoTokenizer</code> selectează întotdeauna tokenizerul rapid pentru tine dacă este disponibil. În următoarea secțiune, vom examina unele dintre celelalte caracteristici speciale ale tokenizerilor rapizi, care vor fi foarte utile pentru sarcini precum clasificarea tokenilor și răspunderea la întrebări. Înainte de a face acest lucru, totuși, să încercăm noul nostru tokenizer pe exemplul anterior:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokens = tokenizer.tokenize(example) | |
| tokens<!-- HTML_TAG_END --></pre></div> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->[<span class="hljs-string">'def'</span>, <span class="hljs-string">'Ġadd'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'numbers'</span>, <span class="hljs-string">'('</span>, <span class="hljs-string">'a'</span>, <span class="hljs-string">','</span>, <span class="hljs-string">'Ġb'</span>, <span class="hljs-string">'):'</span>, <span class="hljs-string">'ĊĠĠĠ'</span>, <span class="hljs-string">'Ġ"""'</span>, <span class="hljs-string">'Add'</span>, <span class="hljs-string">'Ġthe'</span>, <span class="hljs-string">'Ġtwo'</span>, <span class="hljs-string">'Ġnumbers'</span>, <span class="hljs-string">'Ġ`'</span>, | |
| <span class="hljs-string">'a'</span>, <span class="hljs-string">'`'</span>, <span class="hljs-string">'Ġand'</span>, <span class="hljs-string">'Ġ`'</span>, <span class="hljs-string">'b'</span>, <span class="hljs-string">'`."""'</span>, <span class="hljs-string">'ĊĠĠĠ'</span>, <span class="hljs-string">'Ġreturn'</span>, <span class="hljs-string">'Ġa'</span>, <span class="hljs-string">'Ġ+'</span>, <span class="hljs-string">'Ġb'</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ekl199">Aici din nou vedem simboluri speciale ca <code>Ġ</code> sau <code>Ċ</code> care denotă spații sau linii noi, dar în același timp putem vedea că tokenizerul nostru a învățat câțiva tokens care sunt foarte specifici la corpusul de funcții Python: de exemplu, tokenul <code>ĊĠĠĠ</code> care reprezintă indentarea, sau tokenul <code>Ġ"""</code> care reprezintă cele trei ghilimele cu care se începe un docstring. Tokenizerul, de asemenea face split corect numelui funției pe <code>_</code>. Aceasta chiar este o reprezentare compactă: comparativ, utilizând limba tokenizerului englez pe același exemplu ne va da o propoziție mai lungă:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-built_in">print</span>(<span class="hljs-built_in">len</span>(tokens)) | |
| <span class="hljs-built_in">print</span>(<span class="hljs-built_in">len</span>(old_tokenizer.tokenize(example)))<!-- HTML_TAG_END --></pre></div> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-number">27</span> | |
| <span class="hljs-number">36</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-bk9upm">Hai să ne uităm la un alt exemplu:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->example = <span class="hljs-string">"""class LinearLayer(): | |
| def __init__(self, input_size, output_size): | |
| self.weight = torch.randn(input_size, output_size) | |
| self.bias = torch.zeros(output_size) | |
| def __call__(self, x): | |
| return x @ self.weights + self.bias | |
| """</span> | |
| tokenizer.tokenize(example)<!-- HTML_TAG_END --></pre></div> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->[<span class="hljs-string">'class'</span>, <span class="hljs-string">'ĠLinear'</span>, <span class="hljs-string">'Layer'</span>, <span class="hljs-string">'():'</span>, <span class="hljs-string">'ĊĠĠĠ'</span>, <span class="hljs-string">'Ġdef'</span>, <span class="hljs-string">'Ġ__'</span>, <span class="hljs-string">'init'</span>, <span class="hljs-string">'__('</span>, <span class="hljs-string">'self'</span>, <span class="hljs-string">','</span>, <span class="hljs-string">'Ġinput'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'size'</span>, <span class="hljs-string">','</span>, | |
| <span class="hljs-string">'Ġoutput'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'size'</span>, <span class="hljs-string">'):'</span>, <span class="hljs-string">'ĊĠĠĠĠĠĠĠ'</span>, <span class="hljs-string">'Ġself'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'weight'</span>, <span class="hljs-string">'Ġ='</span>, <span class="hljs-string">'Ġtorch'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'randn'</span>, <span class="hljs-string">'('</span>, <span class="hljs-string">'input'</span>, <span class="hljs-string">'_'</span>, | |
| <span class="hljs-string">'size'</span>, <span class="hljs-string">','</span>, <span class="hljs-string">'Ġoutput'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'size'</span>, <span class="hljs-string">')'</span>, <span class="hljs-string">'ĊĠĠĠĠĠĠĠ'</span>, <span class="hljs-string">'Ġself'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'bias'</span>, <span class="hljs-string">'Ġ='</span>, <span class="hljs-string">'Ġtorch'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'zeros'</span>, <span class="hljs-string">'('</span>, | |
| <span class="hljs-string">'output'</span>, <span class="hljs-string">'_'</span>, <span class="hljs-string">'size'</span>, <span class="hljs-string">')'</span>, <span class="hljs-string">'ĊĊĠĠĠ'</span>, <span class="hljs-string">'Ġdef'</span>, <span class="hljs-string">'Ġ__'</span>, <span class="hljs-string">'call'</span>, <span class="hljs-string">'__('</span>, <span class="hljs-string">'self'</span>, <span class="hljs-string">','</span>, <span class="hljs-string">'Ġx'</span>, <span class="hljs-string">'):'</span>, <span class="hljs-string">'ĊĠĠĠĠĠĠĠ'</span>, | |
| <span class="hljs-string">'Ġreturn'</span>, <span class="hljs-string">'Ġx'</span>, <span class="hljs-string">'Ġ@'</span>, <span class="hljs-string">'Ġself'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'weights'</span>, <span class="hljs-string">'Ġ+'</span>, <span class="hljs-string">'Ġself'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'bias'</span>, <span class="hljs-string">'ĊĠĠĠĠ'</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1pp2ax4">În plus față de tokenul corespunzător unei indentări, aici putem vedea și un token pentru o indentare dublă: <code>ĊĠĠĠĠĠĠĠ</code>. Cuvintele speciale din Python, cum ar fi <code>class</code>, <code>init</code>, <code>call</code>, <code>self</code> și <code>return</code>, sunt tokenizate fiecare ca un singur token, și putem vedea că, pe lângă divizarea la <code>_</code> și <code>.</code>, tokenizerul divizează corect chiar și numele scrise în stil camel-case: <code>LinearLayer</code> este tokenizeat ca <code>["ĠLinear", "Layer"]</code>.</p> <h2 class="relative group"><a id="saving-the-tokenizer" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#saving-the-tokenizer"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Salvarea tokenizerului</span></h2> <p data-svelte-h="svelte-uc88s5">Pentru a ne asigura că îl putem utiliza mai târziu, trebuie să salvăm noul nostru tokenizer. Asemănător salvării unui model, acest lucru se realizează cu metoda <code>save_pretrained()</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokenizer.save_pretrained(<span class="hljs-string">"code-search-net-tokenizer"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-11wk93f">Acest lucru va crea un nou folder numit <em>code-search-net-tokenizer</em>, care va conține toate fișierele necesare pentru a reîncărca tokenizerul. Dacă doriți să partajați acest tokenizer cu colegii și prietenii tăi, îl puteți încărca pe Hub prin conectarea la contul tău. Dacă lucrați într-un notebook, există un convenience function pentru a vă ajuta cu acest lucru:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> huggingface_hub <span class="hljs-keyword">import</span> notebook_login | |
| notebook_login()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-12gid1z">Acest lucru va afișa un widget în care puteți introduce credențialele de conectare Hugging Face. Dacă nu lucrați într-un notebook, introduceți simplu următoarea linie în terminal:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->huggingface-cli login<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-64lsmi">După ce v-ați conectat, puteți face push tokenizerului prin executarea următoarei comenzi:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->tokenizer.push_to_hub(<span class="hljs-string">"code-search-net-tokenizer"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-xvvlpi">Acest lucru va crea un nou repositoriu în namespace-ul tău cu numele <code>code-search-net-tokenizer</code>, care va conține fișierul tokenizerului. După aceea, puteți încărca tokenizerul de oriunde cu metoda <code>from_pretrained()</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Înlocuiți "huggingface-course" mai jos cu namespace-ul tău pentru a utiliza propriul tokenizer</span> | |
| tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"huggingface-course/code-search-net-tokenizer"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1uwvvjx">Acum sunteți gata să antrenați un model de limbaj de la zero și să îl ajustați pentru sarcina voastră! Vom face acest lucru în <a href="/course/chapter7">Capitolul 7</a>, dar mai întâi, în continuarea acestui capitol, vom arunca o privire mai atentă asupra tokenizerilor rapizi și vom explora în detaliu ce se întâmplă atunci când apelați metoda <code>train_new_from_iterator()</code>.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/course/blob/main/chapters/rum/chapter6/2.mdx" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1ftlxhy = { | |
| assets: "/docs/course/pr_1069/rum", | |
| base: "/docs/course/pr_1069/rum", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/course/pr_1069/rum/_app/immutable/entry/start.1de7c3d2.js"), | |
| import("/docs/course/pr_1069/rum/_app/immutable/entry/app.1f82014c.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 67], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 70.5 kB
- Xet hash:
- 2f931f147183b8a4f23a8184fd64de11dc98624fbbc99cc1bad7d02e0255c832
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.