Hugging Face's logo

LocalWisdom
/
PurpleFastText

Feature Extraction
Safetensors
English
model_hub_mixin
pytorch_model_hub_mixin
Model card Files
xet
 Community
hanzceo commited on
Commit
3f2478c
·
verified ·
1 Parent(s): afb9be2

Upload train.ipynb

Browse files
Files changed (1) hide show
  1. train.ipynb +792 -0
train.ipynb ADDED
@@ -0,0 +1,792 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "code",
5
+ "metadata": {
6
+ "collapsed": false,
7
+ "scrolled": true
8
+ },
9
+ "source": [
10
+ "%uv pip install tokenizers datasets huggingface_hub"
11
+ ],
12
+ "execution_count": 1,
13
+ "outputs": [
14
+ {
15
+ "output_type": "stream",
16
+ "name": "stdout",
17
+ "text": [
18
+ "\u001b[2mUsing Python 3.12.6 environment at: /usr/local\u001b[0m\r\n",
19
+ "\u001b[37m\u280b\u001b[0m \u001b[2mResolving dependencies... \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mResolving dependencies... \u001b[0m\r\u001b[2K\u001b[37m\u280b\u001b[0m \u001b[2mResolving dependencies... \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mResolving dependencies... \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mtokenizers==0.22.0 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mdatasets==5.0.0 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mhuggingface-hub==0.34.4 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mfilelock==3.13.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mnumpy==2.1.2 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mpyarrow==25.0.0 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mdill==0.4.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mpandas==2.3.2 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mnumpy==2.1.2 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mrequests==2.32.5 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mhttpx==0.28.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mtqdm==4.67.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mxxhash==3.8.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mmultiprocess==0.70.19 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mfsspec==2024.6.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mfsspec==2024.6.1 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mpackaging==25.0 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mpyyaml==6.0.2 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mtyping-extensions==4.12.2 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mhf-xet==1.1.9 \u001b[0m\r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mhf-xet==1.1.9 \u001b[0m\r\u001b[2K\u001b[2mResolved \u001b[1m37 packages\u001b[0m \u001b[2min 197ms\u001b[0m\u001b[0m\r\n",
20
+ "\u001b[37m\u280b\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/0) \r\u001b[2K\u001b[37m\u280b\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5) \r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5) \r\u001b[2K\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
21
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/146.76 KiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
22
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 16.00 KiB/146.76 KiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
23
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 16.00 KiB/146.76 KiB\r\n",
24
+ "\u001b[2mdatasets \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
25
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 16.00 KiB/146.76 KiB\r\n",
26
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
27
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 16.00 KiB/146.76 KiB\r\n",
28
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 30.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
29
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 32.00 KiB/146.76 KiB\r\n",
30
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 30.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
31
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
32
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 30.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
33
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
34
+ "\u001b[2mdatasets \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 46.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
35
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
36
+ "\u001b[2mdatasets \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 62.87 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
37
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
38
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
39
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
40
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
41
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/47.77 MiB \u001b[3A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[3A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
42
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
43
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
44
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[3A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[3A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
45
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
46
+ "\u001b[2mxxhash \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/215.30 KiB\r\n",
47
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
48
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[4A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[4A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
49
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
50
+ "\u001b[2mxxhash \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 14.85 KiB/215.30 KiB\r\n",
51
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
52
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[4A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[4A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
53
+ "\u001b[2mdill \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/117.21 KiB\r\n",
54
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
55
+ "\u001b[2mxxhash \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 14.85 KiB/215.30 KiB\r\n",
56
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
57
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[5A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[5A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
58
+ "\u001b[2mdill \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 14.84 KiB/117.21 KiB\r\n",
59
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
60
+ "\u001b[2mxxhash \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 14.85 KiB/215.30 KiB\r\n",
61
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 76.23 KiB/542.07 KiB\r\n",
62
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[5A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[5A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
63
+ "\u001b[2mdill \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 14.84 KiB/117.21 KiB\r\n",
64
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 48.00 KiB/146.76 KiB\r\n",
65
+ "\u001b[2mxxhash \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 14.85 KiB/215.30 KiB\r\n",
66
+ "\u001b[2mdatasets \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 92.23 KiB/542.07 KiB\r\n",
67
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 14.81 KiB/47.77 MiB \u001b[5A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[5A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
68
+ "\u001b[2mdill \u001b[0m \u001b[32m-----------------\u001b[2m-------------\u001b[0m\u001b[0m 62.84 KiB/117.21 KiB\r\n",
69
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m----------------\u001b[2m--------------\u001b[0m\u001b[0m 76.96 KiB/146.76 KiB\r\n",
70
+ "\u001b[2mdatasets \u001b[0m \u001b[32m---------\u001b[2m---------------------\u001b[0m\u001b[0m 156.23 KiB/542.07 KiB\r\n",
71
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 735.89 KiB/47.77 MiB \u001b[4A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[4A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
72
+ "\u001b[2mdill \u001b[0m \u001b[32m---------------------\u001b[2m---------\u001b[0m\u001b[0m 78.84 KiB/117.21 KiB\r\n",
73
+ "\u001b[2mmultiprocess\u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 92.96 KiB/146.76 KiB\r\n",
74
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 236.12 KiB/542.07 KiB\r\n",
75
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 1.07 MiB/47.77 MiB \u001b[4A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[4A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
76
+ "\u001b[2mdill \u001b[0m \u001b[32m-----------------------------\u001b[2m-\u001b[0m\u001b[0m 110.84 KiB/117.21 KiB\r\n",
77
+ "\u001b[2mdatasets \u001b[0m \u001b[32m------------------\u001b[2m------------\u001b[0m\u001b[0m 316.23 KiB/542.07 KiB\r\n",
78
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 2.12 MiB/47.77 MiB \u001b[3A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[3A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
79
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-------------------\u001b[2m-----------\u001b[0m\u001b[0m 332.23 KiB/542.07 KiB\r\n",
80
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 2.62 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
81
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 348.23 KiB/542.07 KiB\r\n",
82
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 2.80 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
83
+ "\u001b[2mdatasets \u001b[0m \u001b[32m---------------------\u001b[2m---------\u001b[0m\u001b[0m 364.23 KiB/542.07 KiB\r\n",
84
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 6.25 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2819\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/5)\r\n",
85
+ "\u001b[2mdatasets \u001b[0m \u001b[32m-----------------------\u001b[2m-------\u001b[0m\u001b[0m 412.23 KiB/542.07 KiB\r\n",
86
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 8.42 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2839\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
87
+ "\u001b[2mdatasets \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 428.23 KiB/542.07 KiB\r\n",
88
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--------\u001b[2m----------------------\u001b[0m\u001b[0m 11.58 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2839\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
89
+ "\u001b[2mdatasets \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 428.23 KiB/542.07 KiB\r\n",
90
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 14.45 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2839\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
91
+ "\u001b[2mdatasets \u001b[0m \u001b[32m--------------------------\u001b[2m----\u001b[0m\u001b[0m 460.23 KiB/542.07 KiB\r\n",
92
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 16.89 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2839\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
93
+ "\u001b[2mdatasets \u001b[0m \u001b[32m------------------------------\u001b[2m\u001b[0m\u001b[0m 524.23 KiB/542.07 KiB\r\n",
94
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 19.98 MiB/47.77 MiB \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2A\u001b[37m\u2838\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
95
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 20.87 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2838\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
96
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m---------------\u001b[2m---------------\u001b[0m\u001b[0m 22.70 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2838\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
97
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m----------------\u001b[2m--------------\u001b[0m\u001b[0m 25.08 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2838\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
98
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m------------------\u001b[2m------------\u001b[0m\u001b[0m 27.73 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2838\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/5)\r\n",
99
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 30.47 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u283c\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
100
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 33.67 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u283c\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
101
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------\u001b[2m-------\u001b[0m\u001b[0m 36.00 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u283c\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
102
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-------------------------\u001b[2m-----\u001b[0m\u001b[0m 38.79 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2834\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
103
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m---------------------------\u001b[2m---\u001b[0m\u001b[0m 41.76 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2834\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
104
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 44.22 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2834\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
105
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------------\u001b[2m-\u001b[0m\u001b[0m 45.86 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2834\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
106
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m------------------------------\u001b[2m\u001b[0m\u001b[0m 46.45 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2834\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5)\r\n",
107
+ "\u001b[2mpyarrow \u001b[0m \u001b[32m------------------------------\u001b[2m\u001b[0m\u001b[0m 47.33 MiB/47.77 MiB \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A\u001b[37m\u2826\u001b[0m \u001b[2mPreparing packages...\u001b[0m (4/5) \r\u001b[2K\u001b[2mPrepared \u001b[1m5 packages\u001b[0m \u001b[2min 1.02s\u001b[0m\u001b[0m\r\n",
108
+ "\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [0/0] \u001b[2mInstalling wheels... \u001b[0m\r\u001b[2K\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [0/5] \u001b[2mInstalling wheels... \u001b[0m\r\u001b[2K\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [0/5] \u001b[2mxxhash==3.8.1 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [1/5] \u001b[2mxxhash==3.8.1 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [1/5] \u001b[2mmultiprocess==0.70.19 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [2/5] \u001b[2mmultiprocess==0.70.19 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [2/5] \u001b[2mdill==0.4.1 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [3/5] \u001b[2mdill==0.4.1 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591\u2591\u2591\u2591\u2591 [3/5] \u001b[2mdatasets==5.0.0 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591 [4/5] \u001b[2mdatasets==5.0.0 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2591\u2591\u2591\u2591 [4/5] \u001b[2mpyarrow==25.0.0 \u001b[0m\r\u001b[2K\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 [5/5] \u001b[2mpyarrow==25.0.0 \u001b[0m\r\u001b[2K\u001b[2mInstalled \u001b[1m5 packages\u001b[0m \u001b[2min 83ms\u001b[0m\u001b[0m\r\n",
109
+ " \u001b[32m+\u001b[39m \u001b[1mdatasets\u001b[0m\u001b[2m==5.0.0\u001b[0m\r\n",
110
+ " \u001b[32m+\u001b[39m \u001b[1mdill\u001b[0m\u001b[2m==0.4.1\u001b[0m\r\n",
111
+ " \u001b[32m+\u001b[39m \u001b[1mmultiprocess\u001b[0m\u001b[2m==0.70.19\u001b[0m\r\n",
112
+ " \u001b[32m+\u001b[39m \u001b[1mpyarrow\u001b[0m\u001b[2m==25.0.0\u001b[0m\r\n",
113
+ " \u001b[32m+\u001b[39m \u001b[1mxxhash\u001b[0m\u001b[2m==3.8.1\u001b[0m\r\n",
114
+ "Note: you may need to restart the kernel to use updated packages.\n"
115
+ ]
116
+ }
117
+ ]
118
+ },
119
+ {
120
+ "cell_type": "code",
121
+ "metadata": {
122
+ "collapsed": false,
123
+ "scrolled": true
124
+ },
125
+ "source": [
126
+ "import torch\n",
127
+ "import torch.nn as nn\n",
128
+ "import torch.optim as optim\n",
129
+ "from datasets import Dataset, load_dataset, concatenate_datasets\n",
130
+ "from tokenizers import Tokenizer, Regex\n",
131
+ "from tokenizers.models import BPE\n",
132
+ "from tokenizers.pre_tokenizers import Sequence, Whitespace, Punctuation, Split\n",
133
+ "from tokenizers.trainers import BpeTrainer\n",
134
+ "from huggingface_hub import PyTorchModelHubMixin, notebook_login, HfApi"
135
+ ],
136
+ "execution_count": 18,
137
+ "outputs": []
138
+ },
139
+ {
140
+ "cell_type": "code",
141
+ "metadata": {
142
+ "collapsed": false,
143
+ "scrolled": true
144
+ },
145
+ "source": [
146
+ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
147
+ "default_config = dict(\n",
148
+ " context_window=2,\n",
149
+ " vocab_size=0,\n",
150
+ " embedding_dim=200,\n",
151
+ "\n",
152
+ " batch_size=8192,\n",
153
+ " num_negatives=5\n",
154
+ ")"
155
+ ],
156
+ "execution_count": 3,
157
+ "outputs": []
158
+ },
159
+ {
160
+ "cell_type": "code",
161
+ "metadata": {
162
+ "collapsed": false,
163
+ "scrolled": true
164
+ },
165
+ "source": [
166
+ "ds_train = load_dataset(\"bobox/OpenbookQA-4ST\", data_dir=\"filtered\", split=\"train+validation\", revision=\"refs/convert/parquet\")\n",
167
+ "ds_test = load_dataset(\"bobox/OpenbookQA-4ST\", data_dir=\"filtered\", split=\"test\", revision=\"refs/convert/parquet\")\n",
168
+ "ds = concatenate_datasets([ds_train, ds_test])\n",
169
+ "ds_len = len(ds_train)+len(ds_test)"
170
+ ],
171
+ "execution_count": 4,
172
+ "outputs": [
173
+ {
174
+ "output_type": "display_data",
175
+ "data": {
176
+ "application/vnd.jupyter.widget-view+json": {
177
+ "model_id": "c84e223fda7b40dface4157869fb589b",
178
+ "version_minor": 0.0,
179
+ "version_major": 2.0
180
+ },
181
+ "text/plain": "filtered/train/0000.parquet: 0%| | 0.00/328k [00:00<?, ?B/s]"
182
+ },
183
+ "metadata": {}
184
+ },
185
+ {
186
+ "output_type": "display_data",
187
+ "data": {
188
+ "application/vnd.jupyter.widget-view+json": {
189
+ "model_id": "06420636fba64f92a6870b33da17e20e",
190
+ "version_minor": 0.0,
191
+ "version_major": 2.0
192
+ },
193
+ "text/plain": "filtered/validation/0000.parquet: 0%| | 0.00/41.8k [00:00<?, ?B/s]"
194
+ },
195
+ "metadata": {}
196
+ },
197
+ {
198
+ "output_type": "display_data",
199
+ "data": {
200
+ "application/vnd.jupyter.widget-view+json": {
201
+ "model_id": "027daf714a8240f98189eded8f4edec6",
202
+ "version_minor": 0.0,
203
+ "version_major": 2.0
204
+ },
205
+ "text/plain": "filtered/test/0000.parquet: 0%| | 0.00/48.2k [00:00<?, ?B/s]"
206
+ },
207
+ "metadata": {}
208
+ },
209
+ {
210
+ "output_type": "display_data",
211
+ "data": {
212
+ "application/vnd.jupyter.widget-view+json": {
213
+ "model_id": "7f0903bfd5d24caea5033dda18c0e77a",
214
+ "version_minor": 0.0,
215
+ "version_major": 2.0
216
+ },
217
+ "text/plain": "Generating train split: 0 examples [00:00, ? examples/s]"
218
+ },
219
+ "metadata": {}
220
+ },
221
+ {
222
+ "output_type": "display_data",
223
+ "data": {
224
+ "application/vnd.jupyter.widget-view+json": {
225
+ "model_id": "fefa92bf3e604b35949fbde0b24ce40c",
226
+ "version_minor": 0.0,
227
+ "version_major": 2.0
228
+ },
229
+ "text/plain": "Generating validation split: 0 examples [00:00, ? examples/s]"
230
+ },
231
+ "metadata": {}
232
+ },
233
+ {
234
+ "output_type": "display_data",
235
+ "data": {
236
+ "application/vnd.jupyter.widget-view+json": {
237
+ "model_id": "cc77d154ff814d7bba05e723f43331f3",
238
+ "version_minor": 0.0,
239
+ "version_major": 2.0
240
+ },
241
+ "text/plain": "Generating test split: 0 examples [00:00, ? examples/s]"
242
+ },
243
+ "metadata": {}
244
+ }
245
+ ]
246
+ },
247
+ {
248
+ "cell_type": "code",
249
+ "metadata": {
250
+ "collapsed": false,
251
+ "scrolled": true
252
+ },
253
+ "source": [
254
+ "tokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\n",
255
+ "tokenizer.pre_tokenizer = Sequence([\n",
256
+ " Whitespace(),\n",
257
+ " Punctuation(behavior=\"removed\"),\n",
258
+ " Split(Regex(r\"\\d+\"), behavior=\"removed\")\n",
259
+ "])\n",
260
+ "\n",
261
+ "tokenizer_trainer = BpeTrainer(special_tokens=[\"[UNK]\"], max_token_length=4)\n",
262
+ "\n",
263
+ "def batcher(n_batch=1000):\n",
264
+ " for i in range(0, ds_len, n_batch):\n",
265
+ " yield ds[\"fact\"][i : i + n_batch]\n",
266
+ "\n",
267
+ "tokenizer.train_from_iterator(batcher(), trainer=tokenizer_trainer, length=ds_len)\n",
268
+ "\n",
269
+ "tokenizer.save(\"tokenizer.json\")"
270
+ ],
271
+ "execution_count": 5,
272
+ "outputs": [
273
+ {
274
+ "output_type": "stream",
275
+ "name": "stdout",
276
+ "text": [
277
+ "\n",
278
+ "\n",
279
+ "\n"
280
+ ]
281
+ }
282
+ ]
283
+ },
284
+ {
285
+ "cell_type": "code",
286
+ "metadata": {
287
+ "collapsed": false,
288
+ "scrolled": true
289
+ },
290
+ "source": [
291
+ "class FastText(nn.Module, PyTorchModelHubMixin):\n",
292
+ " def __init__(self, config):\n",
293
+ " nn.Module.__init__(self)\n",
294
+ "\n",
295
+ " self.context_window = config[\"context_window\"]\n",
296
+ " self.vocab_size = config[\"vocab_size\"]\n",
297
+ " self.embedding_dim = config[\"embedding_dim\"]\n",
298
+ "\n",
299
+ " self.embedding = nn.EmbeddingBag(self.vocab_size, self.embedding_dim, mode=\"sum\", sparse=True)\n",
300
+ " self.context_embedding = nn.Embedding(self.vocab_size, self.embedding_dim, sparse=True)\n",
301
+ "\n",
302
+ " def forward(self, input_ids, offsets, context_ids, negative_ids):\n",
303
+ " u_wt = self.embedding(input_ids, offsets)\n",
304
+ " v_wc = self.context_embedding(context_ids)\n",
305
+ "\n",
306
+ " pos_score = (u_wt * v_wc).sum(dim=1)\n",
307
+ "\n",
308
+ " v_neg = self.context_embedding(negative_ids)\n",
309
+ " neg_score = torch.bmm(v_neg, u_wt.unsqueeze(2)).squeeze(2)\n",
310
+ "\n",
311
+ " log1p_exp = lambda x: torch.log1p(torch.exp(-x.abs())) + torch.relu(x)\n",
312
+ " loss = log1p_exp(-pos_score).mean() + log1p_exp(neg_score).sum(dim=1).mean()\n",
313
+ " return loss"
314
+ ],
315
+ "execution_count": 6,
316
+ "outputs": []
317
+ },
318
+ {
319
+ "cell_type": "code",
320
+ "metadata": {
321
+ "collapsed": false,
322
+ "scrolled": true
323
+ },
324
+ "source": [
325
+ "default_config[\"vocab_size\"] = tokenizer.get_vocab_size()\n",
326
+ "PurpleFastText = FastText(config=default_config).to(device)"
327
+ ],
328
+ "execution_count": 7,
329
+ "outputs": []
330
+ },
331
+ {
332
+ "cell_type": "code",
333
+ "metadata": {
334
+ "collapsed": false,
335
+ "scrolled": true
336
+ },
337
+ "source": [
338
+ "def format_data(batch):\n",
339
+ " input_ids = []\n",
340
+ " offsets = []\n",
341
+ " context_ids = []\n",
342
+ " negative_ids = []\n",
343
+ "\n",
344
+ " for sentence in batch[\"fact\"]:\n",
345
+ " tokens = tokenizer.encode(sentence)\n",
346
+ " toks = tokens.ids\n",
347
+ "\n",
348
+ " for i in range(len(toks)):\n",
349
+ " start = max(0, i - default_config[\"context_window\"])\n",
350
+ " end = min(len(toks), i + default_config[\"context_window\"] + 1)\n",
351
+ "\n",
352
+ " for j in range(start, end):\n",
353
+ " if i != j:\n",
354
+ " input_ids.append(toks[i])\n",
355
+ " context_ids.append(toks[j])\n",
356
+ "\n",
357
+ " negative_ids = (\n",
358
+ " torch.randint(\n",
359
+ " low=0,\n",
360
+ " high=default_config[\"vocab_size\"],\n",
361
+ " size=(len(input_ids), default_config[\"num_negatives\"]),\n",
362
+ " dtype=torch.long\n",
363
+ " ).tolist()\n",
364
+ " )\n",
365
+ "\n",
366
+ " return dict(\n",
367
+ " input_ids=input_ids,\n",
368
+ " context_ids=context_ids,\n",
369
+ " negative_ids=negative_ids\n",
370
+ " )\n",
371
+ "\n",
372
+ "def collator(batch):\n",
373
+ " batch_dict = torch.utils.data.default_collate(batch)\n",
374
+ "\n",
375
+ " input_ids = batch_dict[\"input_ids\"]\n",
376
+ " context_ids = batch_dict[\"context_ids\"]\n",
377
+ " negative_ids = batch_dict[\"negative_ids\"]\n",
378
+ "\n",
379
+ " offsets = torch.arange(len(input_ids), dtype=torch.long)\n",
380
+ "\n",
381
+ " return input_ids, offsets, context_ids, negative_ids\n",
382
+ "\n",
383
+ "dl = torch.utils.data.DataLoader(\n",
384
+ " ds_train.map(format_data, batched=True, remove_columns=ds_train.column_names).with_format(\"torch\"), # should've used IterableDataset from torch\n",
385
+ " batch_size=default_config[\"batch_size\"], shuffle=True,\n",
386
+ " collate_fn=collator,\n",
387
+ " num_workers=8, persistent_workers=True, prefetch_factor=2,\n",
388
+ " pin_memory=True if device.type == \"cuda\" else False\n",
389
+ ")"
390
+ ],
391
+ "execution_count": 12,
392
+ "outputs": []
393
+ },
394
+ {
395
+ "cell_type": "code",
396
+ "metadata": {
397
+ "collapsed": false,
398
+ "scrolled": true
399
+ },
400
+ "source": [
401
+ "optimizer = optim.SparseAdam(PurpleFastText.parameters(), lr=1e-4)\n",
402
+ "\n",
403
+ "for epoch in range(200):\n",
404
+ " for (input_ids, offsets, context_ids, negative_ids) in dl:\n",
405
+ " optimizer.zero_grad()\n",
406
+ " loss = PurpleFastText(input_ids.to(device), offsets.to(device), context_ids.to(device), negative_ids.to(device))\n",
407
+ " loss.backward()\n",
408
+ " optimizer.step()\n",
409
+ "\n",
410
+ " loss = loss.item()\n",
411
+ " print(f\"{epoch = } | {loss = }\")"
412
+ ],
413
+ "execution_count": 13,
414
+ "outputs": [
415
+ {
416
+ "output_type": "stream",
417
+ "name": "stderr",
418
+ "text": [
419
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
420
+ "To disable this warning, you can either:\n",
421
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
422
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
423
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
424
+ "To disable this warning, you can either:\n",
425
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
426
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
427
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
428
+ "To disable this warning, you can either:\n",
429
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
430
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
431
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
432
+ "To disable this warning, you can either:\n",
433
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
434
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
435
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
436
+ "To disable this warning, you can either:\n",
437
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
438
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
439
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
440
+ "To disable this warning, you can either:\n",
441
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
442
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
443
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
444
+ "To disable this warning, you can either:\n",
445
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
446
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
447
+ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
448
+ "To disable this warning, you can either:\n",
449
+ "\t- Avoid using `tokenizers` before the fork if possible\n",
450
+ "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
451
+ ]
452
+ },
453
+ {
454
+ "output_type": "stream",
455
+ "name": "stdout",
456
+ "text": [
457
+ "epoch = 0 | loss = 33.62425231933594\n",
458
+ "epoch = 1 | loss = 33.472225189208984\n",
459
+ "epoch = 2 | loss = 33.398258209228516\n",
460
+ "epoch = 3 | loss = 33.84187316894531\n",
461
+ "epoch = 4 | loss = 32.96810531616211\n",
462
+ "epoch = 5 | loss = 32.768104553222656\n",
463
+ "epoch = 6 | loss = 32.597679138183594\n",
464
+ "epoch = 7 | loss = 32.50499725341797\n",
465
+ "epoch = 8 | loss = 32.46271514892578\n",
466
+ "epoch = 9 | loss = 31.799774169921875\n",
467
+ "epoch = 10 | loss = 31.96149253845215\n",
468
+ "epoch = 11 | loss = 31.556041717529297\n",
469
+ "epoch = 12 | loss = 31.90681266784668\n",
470
+ "epoch = 13 | loss = 31.102127075195312\n",
471
+ "epoch = 14 | loss = 31.5416316986084\n",
472
+ "epoch = 15 | loss = 31.22119140625\n",
473
+ "epoch = 16 | loss = 31.240070343017578\n",
474
+ "epoch = 17 | loss = 30.6594181060791\n",
475
+ "epoch = 18 | loss = 30.212684631347656\n",
476
+ "epoch = 19 | loss = 30.307228088378906\n",
477
+ "epoch = 20 | loss = 30.078458786010742\n",
478
+ "epoch = 21 | loss = 29.99860382080078\n",
479
+ "epoch = 22 | loss = 29.850513458251953\n",
480
+ "epoch = 23 | loss = 29.601530075073242\n",
481
+ "epoch = 24 | loss = 29.231857299804688\n",
482
+ "epoch = 25 | loss = 29.260231018066406\n",
483
+ "epoch = 26 | loss = 29.491256713867188\n",
484
+ "epoch = 27 | loss = 28.56077003479004\n",
485
+ "epoch = 28 | loss = 29.046772003173828\n",
486
+ "epoch = 29 | loss = 28.38245964050293\n",
487
+ "epoch = 30 | loss = 28.465112686157227\n",
488
+ "epoch = 31 | loss = 28.316547393798828\n",
489
+ "epoch = 32 | loss = 28.164907455444336\n",
490
+ "epoch = 33 | loss = 27.77496910095215\n",
491
+ "epoch = 34 | loss = 27.681516647338867\n",
492
+ "epoch = 35 | loss = 27.743240356445312\n",
493
+ "epoch = 36 | loss = 27.441387176513672\n",
494
+ "epoch = 37 | loss = 27.788480758666992\n",
495
+ "epoch = 38 | loss = 27.45903778076172\n",
496
+ "epoch = 39 | loss = 27.11681365966797\n",
497
+ "epoch = 40 | loss = 27.122961044311523\n",
498
+ "epoch = 41 | loss = 26.90481948852539\n",
499
+ "epoch = 42 | loss = 26.61617088317871\n",
500
+ "epoch = 43 | loss = 26.463600158691406\n",
501
+ "epoch = 44 | loss = 26.532682418823242\n",
502
+ "epoch = 45 | loss = 26.35540008544922\n",
503
+ "epoch = 46 | loss = 26.15418243408203\n",
504
+ "epoch = 47 | loss = 26.277740478515625\n",
505
+ "epoch = 48 | loss = 26.20408058166504\n",
506
+ "epoch = 49 | loss = 25.71695327758789\n",
507
+ "epoch = 50 | loss = 25.684810638427734\n",
508
+ "epoch = 51 | loss = 25.397275924682617\n",
509
+ "epoch = 52 | loss = 25.11101531982422\n",
510
+ "epoch = 53 | loss = 25.443233489990234\n",
511
+ "epoch = 54 | loss = 24.976665496826172\n",
512
+ "epoch = 55 | loss = 24.98261070251465\n",
513
+ "epoch = 56 | loss = 24.903249740600586\n",
514
+ "epoch = 57 | loss = 24.44039535522461\n",
515
+ "epoch = 58 | loss = 24.10711669921875\n",
516
+ "epoch = 59 | loss = 24.02779769897461\n",
517
+ "epoch = 60 | loss = 24.355669021606445\n",
518
+ "epoch = 61 | loss = 23.761302947998047\n",
519
+ "epoch = 62 | loss = 24.263946533203125\n",
520
+ "epoch = 63 | loss = 23.884416580200195\n",
521
+ "epoch = 64 | loss = 23.76875877380371\n",
522
+ "epoch = 65 | loss = 23.63066864013672\n",
523
+ "epoch = 66 | loss = 23.35961151123047\n",
524
+ "epoch = 67 | loss = 23.55689239501953\n",
525
+ "epoch = 68 | loss = 23.246044158935547\n",
526
+ "epoch = 69 | loss = 22.76032257080078\n",
527
+ "epoch = 70 | loss = 23.042325973510742\n",
528
+ "epoch = 71 | loss = 22.743942260742188\n",
529
+ "epoch = 72 | loss = 22.37074089050293\n",
530
+ "epoch = 73 | loss = 22.6041259765625\n",
531
+ "epoch = 74 | loss = 22.25589370727539\n",
532
+ "epoch = 75 | loss = 22.096574783325195\n",
533
+ "epoch = 76 | loss = 22.071853637695312\n",
534
+ "epoch = 77 | loss = 22.31040382385254\n",
535
+ "epoch = 78 | loss = 22.118852615356445\n",
536
+ "epoch = 79 | loss = 21.45325469970703\n",
537
+ "epoch = 80 | loss = 21.61875343322754\n",
538
+ "epoch = 81 | loss = 21.439592361450195\n",
539
+ "epoch = 82 | loss = 21.3583927154541\n",
540
+ "epoch = 83 | loss = 21.06571388244629\n",
541
+ "epoch = 84 | loss = 20.578998565673828\n",
542
+ "epoch = 85 | loss = 20.79303550720215\n",
543
+ "epoch = 86 | loss = 20.762752532958984\n",
544
+ "epoch = 87 | loss = 20.799985885620117\n",
545
+ "epoch = 88 | loss = 20.517189025878906\n",
546
+ "epoch = 89 | loss = 20.281986236572266\n",
547
+ "epoch = 90 | loss = 20.166234970092773\n",
548
+ "epoch = 91 | loss = 20.12177848815918\n",
549
+ "epoch = 92 | loss = 20.381624221801758\n",
550
+ "epoch = 93 | loss = 19.79176902770996\n",
551
+ "epoch = 94 | loss = 19.528615951538086\n",
552
+ "epoch = 95 | loss = 19.95457649230957\n",
553
+ "epoch = 96 | loss = 19.54012680053711\n",
554
+ "epoch = 97 | loss = 19.215112686157227\n",
555
+ "epoch = 98 | loss = 19.4979305267334\n",
556
+ "epoch = 99 | loss = 19.16718864440918\n",
557
+ "epoch = 100 | loss = 18.99817657470703\n",
558
+ "epoch = 101 | loss = 18.84443473815918\n",
559
+ "epoch = 102 | loss = 18.61274528503418\n",
560
+ "epoch = 103 | loss = 18.64920425415039\n",
561
+ "epoch = 104 | loss = 18.58812141418457\n",
562
+ "epoch = 105 | loss = 18.30907440185547\n",
563
+ "epoch = 106 | loss = 18.271347045898438\n",
564
+ "epoch = 107 | loss = 18.45148468017578\n",
565
+ "epoch = 108 | loss = 18.378337860107422\n",
566
+ "epoch = 109 | loss = 18.269376754760742\n",
567
+ "epoch = 110 | loss = 17.706981658935547\n",
568
+ "epoch = 111 | loss = 17.62091064453125\n",
569
+ "epoch = 112 | loss = 17.225004196166992\n",
570
+ "epoch = 113 | loss = 17.723031997680664\n",
571
+ "epoch = 114 | loss = 17.18803596496582\n",
572
+ "epoch = 115 | loss = 17.075786590576172\n",
573
+ "epoch = 116 | loss = 16.95713233947754\n",
574
+ "epoch = 117 | loss = 16.833356857299805\n",
575
+ "epoch = 118 | loss = 16.814830780029297\n",
576
+ "epoch = 119 | loss = 17.00945472717285\n",
577
+ "epoch = 120 | loss = 16.623493194580078\n",
578
+ "epoch = 121 | loss = 16.313987731933594\n",
579
+ "epoch = 122 | loss = 16.316606521606445\n",
580
+ "epoch = 123 | loss = 16.264816284179688\n",
581
+ "epoch = 124 | loss = 15.747238159179688\n",
582
+ "epoch = 125 | loss = 15.969701766967773\n",
583
+ "epoch = 126 | loss = 16.052602767944336\n",
584
+ "epoch = 127 | loss = 16.00865936279297\n",
585
+ "epoch = 128 | loss = 15.772383689880371\n",
586
+ "epoch = 129 | loss = 15.535850524902344\n",
587
+ "epoch = 130 | loss = 15.065999031066895\n",
588
+ "epoch = 131 | loss = 15.313947677612305\n",
589
+ "epoch = 132 | loss = 15.149383544921875\n",
590
+ "epoch = 133 | loss = 15.007120132446289\n",
591
+ "epoch = 134 | loss = 15.279787063598633\n",
592
+ "epoch = 135 | loss = 14.991166114807129\n",
593
+ "epoch = 136 | loss = 14.836761474609375\n",
594
+ "epoch = 137 | loss = 14.972814559936523\n",
595
+ "epoch = 138 | loss = 14.291553497314453\n",
596
+ "epoch = 139 | loss = 14.598798751831055\n",
597
+ "epoch = 140 | loss = 14.328827857971191\n",
598
+ "epoch = 141 | loss = 14.343247413635254\n",
599
+ "epoch = 142 | loss = 14.173320770263672\n",
600
+ "epoch = 143 | loss = 14.070396423339844\n",
601
+ "epoch = 144 | loss = 13.911269187927246\n",
602
+ "epoch = 145 | loss = 13.662429809570312\n",
603
+ "epoch = 146 | loss = 13.669416427612305\n",
604
+ "epoch = 147 | loss = 13.736848831176758\n",
605
+ "epoch = 148 | loss = 13.631596565246582\n",
606
+ "epoch = 149 | loss = 13.368789672851562\n",
607
+ "epoch = 150 | loss = 13.092583656311035\n",
608
+ "epoch = 151 | loss = 13.113190650939941\n",
609
+ "epoch = 152 | loss = 13.169113159179688\n",
610
+ "epoch = 153 | loss = 13.064599990844727\n",
611
+ "epoch = 154 | loss = 12.857305526733398\n",
612
+ "epoch = 155 | loss = 12.743917465209961\n",
613
+ "epoch = 156 | loss = 12.533823013305664\n",
614
+ "epoch = 157 | loss = 12.30309772491455\n",
615
+ "epoch = 158 | loss = 12.425459861755371\n",
616
+ "epoch = 159 | loss = 12.448675155639648\n",
617
+ "epoch = 160 | loss = 12.417452812194824\n",
618
+ "epoch = 161 | loss = 12.414138793945312\n",
619
+ "epoch = 162 | loss = 12.402454376220703\n",
620
+ "epoch = 163 | loss = 12.141651153564453\n",
621
+ "epoch = 164 | loss = 12.028185844421387\n",
622
+ "epoch = 165 | loss = 11.94282341003418\n",
623
+ "epoch = 166 | loss = 11.632282257080078\n",
624
+ "epoch = 167 | loss = 11.308302879333496\n",
625
+ "epoch = 168 | loss = 11.397911071777344\n",
626
+ "epoch = 169 | loss = 11.540604591369629\n",
627
+ "epoch = 170 | loss = 11.422112464904785\n",
628
+ "epoch = 171 | loss = 11.166105270385742\n",
629
+ "epoch = 172 | loss = 11.410856246948242\n",
630
+ "epoch = 173 | loss = 11.141489028930664\n",
631
+ "epoch = 174 | loss = 11.055648803710938\n",
632
+ "epoch = 175 | loss = 11.138644218444824\n",
633
+ "epoch = 176 | loss = 10.659797668457031\n",
634
+ "epoch = 177 | loss = 10.853754043579102\n",
635
+ "epoch = 178 | loss = 10.80173110961914\n",
636
+ "epoch = 179 | loss = 10.684595108032227\n",
637
+ "epoch = 180 | loss = 10.418027877807617\n",
638
+ "epoch = 181 | loss = 10.73669719696045\n",
639
+ "epoch = 182 | loss = 10.225930213928223\n",
640
+ "epoch = 183 | loss = 10.185312271118164\n",
641
+ "epoch = 184 | loss = 10.047532081604004\n",
642
+ "epoch = 185 | loss = 10.030939102172852\n",
643
+ "epoch = 186 | loss = 10.077144622802734\n",
644
+ "epoch = 187 | loss = 9.813972473144531\n",
645
+ "epoch = 188 | loss = 9.702218055725098\n",
646
+ "epoch = 189 | loss = 9.979422569274902\n",
647
+ "epoch = 190 | loss = 9.73726749420166\n",
648
+ "epoch = 191 | loss = 9.609981536865234\n",
649
+ "epoch = 192 | loss = 9.520135879516602\n",
650
+ "epoch = 193 | loss = 9.530278205871582\n",
651
+ "epoch = 194 | loss = 9.442426681518555\n",
652
+ "epoch = 195 | loss = 9.331952095031738\n",
653
+ "epoch = 196 | loss = 9.179746627807617\n",
654
+ "epoch = 197 | loss = 9.125386238098145\n",
655
+ "epoch = 198 | loss = 9.122198104858398\n",
656
+ "epoch = 199 | loss = 9.065939903259277\n"
657
+ ]
658
+ }
659
+ ]
660
+ },
661
+ {
662
+ "cell_type": "code",
663
+ "metadata": {
664
+ "collapsed": false,
665
+ "scrolled": true
666
+ },
667
+ "source": [
668
+ "notebook_login()"
669
+ ],
670
+ "execution_count": 14,
671
+ "outputs": [
672
+ {
673
+ "output_type": "display_data",
674
+ "data": {
675
+ "application/vnd.jupyter.widget-view+json": {
676
+ "model_id": "dc1ce00109ba423a9a47d08ef1c453d7",
677
+ "version_minor": 0.0,
678
+ "version_major": 2.0
679
+ },
680
+ "text/plain": "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv\u2026"
681
+ },
682
+ "metadata": {}
683
+ }
684
+ ]
685
+ },
686
+ {
687
+ "cell_type": "code",
688
+ "metadata": {
689
+ "collapsed": false,
690
+ "scrolled": true
691
+ },
692
+ "source": [
693
+ "PurpleFastText.push_to_hub(\"LocalWisdom/PurpleFastText\")"
694
+ ],
695
+ "execution_count": 15,
696
+ "outputs": [
697
+ {
698
+ "output_type": "display_data",
699
+ "data": {
700
+ "application/vnd.jupyter.widget-view+json": {
701
+ "model_id": "cce21a8db91c44bca9595f6e81a7db96",
702
+ "version_minor": 0.0,
703
+ "version_major": 2.0
704
+ },
705
+ "text/plain": "Processing Files (0 / 0) : | | 0.00B / 0.00B "
706
+ },
707
+ "metadata": {}
708
+ },
709
+ {
710
+ "output_type": "display_data",
711
+ "data": {
712
+ "application/vnd.jupyter.widget-view+json": {
713
+ "model_id": "ba50891c5c514f208daf6febc9f34d6f",
714
+ "version_minor": 0.0,
715
+ "version_major": 2.0
716
+ },
717
+ "text/plain": "New Data Upload : | | 0.00B / 0.00B "
718
+ },
719
+ "metadata": {}
720
+ },
721
+ {
722
+ "output_type": "display_data",
723
+ "data": {
724
+ "application/vnd.jupyter.widget-view+json": {
725
+ "model_id": "b016ed071f6c451dbb9ca978a6663975",
726
+ "version_minor": 0.0,
727
+ "version_major": 2.0
728
+ },
729
+ "text/plain": " ...om/PurpleFastText/model.safetensors: 100%|##########| 1.43MB / 1.43MB "
730
+ },
731
+ "metadata": {}
732
+ },
733
+ {
734
+ "output_type": "execute_result",
735
+ "execution_count": 15,
736
+ "data": {
737
+ "text/plain": "CommitInfo(commit_url='https://huggingface.co/LocalWisdom/PurpleFastText/commit/6b71cc46afb2660032e44f86fd8ced34b032c3a1', commit_message='Push model using huggingface_hub.', commit_description='', oid='6b71cc46afb2660032e44f86fd8ced34b032c3a1', pr_url=None, repo_url=RepoUrl('https://huggingface.co/LocalWisdom/PurpleFastText', endpoint='https://huggingface.co', repo_type='model', repo_id='LocalWisdom/PurpleFastText'), pr_revision=None, pr_num=None)"
738
+ },
739
+ "metadata": {}
740
+ }
741
+ ]
742
+ },
743
+ {
744
+ "cell_type": "code",
745
+ "metadata": {
746
+ "collapsed": false,
747
+ "scrolled": true
748
+ },
749
+ "source": [
750
+ "api = HfApi()\n",
751
+ "\n",
752
+ "api.upload_file(\n",
753
+ " path_or_fileobj=\"tokenizer.json\",\n",
754
+ " path_in_repo=\"tokenizer.json\",\n",
755
+ " repo_id=\"LocalWisdom/PurpleFastText\",\n",
756
+ " repo_type=\"model\"\n",
757
+ ")"
758
+ ],
759
+ "execution_count": 19,
760
+ "outputs": [
761
+ {
762
+ "output_type": "execute_result",
763
+ "execution_count": 19,
764
+ "data": {
765
+ "text/plain": "CommitInfo(commit_url='https://huggingface.co/LocalWisdom/PurpleFastText/commit/afb9be250708a9d86719ae4a1e74330776abd9f0', commit_message='Upload tokenizer.json with huggingface_hub', commit_description='', oid='afb9be250708a9d86719ae4a1e74330776abd9f0', pr_url=None, repo_url=RepoUrl('https://huggingface.co/LocalWisdom/PurpleFastText', endpoint='https://huggingface.co', repo_type='model', repo_id='LocalWisdom/PurpleFastText'), pr_revision=None, pr_num=None)"
766
+ },
767
+ "metadata": {}
768
+ }
769
+ ]
770
+ }
771
+ ],
772
+ "metadata": {
773
+ "kernelspec": {
774
+ "display_name": "Python",
775
+ "language": "python",
776
+ "name": "python3"
777
+ },
778
+ "language_info": {
779
+ "codemirror_mode": {
780
+ "name": "ipython",
781
+ "version": 3
782
+ },
783
+ "file_extension": ".py",
784
+ "mimetype": "text/x-python",
785
+ "name": "python",
786
+ "nbconvert_exporter": "python",
787
+ "pygments_lexer": "ipython3"
788
+ }
789
+ },
790
+ "nbformat": 4,
791
+ "nbformat_minor": 5
792
+ }