ASomeoneWhoInterestedWithAI commited on
Commit
cceed08
·
verified ·
1 Parent(s): dbae999

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -0
README.md CHANGED
@@ -31,6 +31,8 @@ tags:
31
  - Test accuracy: 35.46%
32
  - Test Top-5 Accuracy: 61.87%
33
  - Code in second and third notebook
 
 
34
 
35
  # Explanation
36
  I was courious, what if a token look at other tokens, but without QKV? Instead, it's like make transformation for two tokens (current token and another token), then divide them. Let's say current token is token A and another token is token B. It's divide like "transformA(A) / transformB(B)" which the transform is a linear NN. With tanh for normalizing (to make it don't explode). And, the reverse ("transformA(B) / transformB(A)"). Then, the result of "transformA(A) / transformB(B)" multiply with A, and the reverse multiply with B. Then add them, then divide by 2. That's the new number for that interaction. Add to temp variable. Loop again for another token interaction (but for the code it's vectorized). Then, that variable averaged. That's the new A.
 
31
  - Test accuracy: 35.46%
32
  - Test Top-5 Accuracy: 61.87%
33
  - Code in second and third notebook
34
+ # STL-10 (labeled only)
35
+ - Detail in https://huggingface.co/ASomeoneWhoInterestedWithAI/LookThem_STL-10
36
 
37
  # Explanation
38
  I was courious, what if a token look at other tokens, but without QKV? Instead, it's like make transformation for two tokens (current token and another token), then divide them. Let's say current token is token A and another token is token B. It's divide like "transformA(A) / transformB(B)" which the transform is a linear NN. With tanh for normalizing (to make it don't explode). And, the reverse ("transformA(B) / transformB(A)"). Then, the result of "transformA(A) / transformB(B)" multiply with A, and the reverse multiply with B. Then add them, then divide by 2. That's the new number for that interaction. Add to temp variable. Loop again for another token interaction (but for the code it's vectorized). Then, that variable averaged. That's the new A.