Push model using huggingface_hub.

README.md

@@ -4,12 +4,42 @@ tags:
 - sentence-transformers
 - text-classification
 - generated_from_setfit_trainer
-widget: []
+widget:
+- text: "Analyze this clean_code: perms = models.Permission.objects.filter(\n    \
+    \        codename__in=(\"add_customuser\", \"change_customuser\")\n        )\n\
+    \        user.user_permissions.add(*perms)\n        request = self.factory.get(\"\
+    /rand\")\n        request.user = user"
+- text: 'Analyze this hardcoded_secret: SECRET_KEY = "super-secret-jwt-key-do-not-share"  #
+    Django production secret'
+- text: "Analyze this vulnerable_pattern: var name_input = document.getElementById\
+    \ ('name');\n\n\t\t\tif (user_json.name == '') {\n\t\t\t\tuser_info.innerHTML\
+    \ = 'User details: unknown user';\n\t\t\t\tname_input.value = 'unknown';\n\t\t\
+    \t} else {\n\t\t\t\tvar level = 'unknown';"
+- text: 'Analyze this hardcoded_secret: const db = new Pool({ password: ''Pr0duct10n#2024''
+    });'
+- text: "Analyze this hardcoded_secret: name:        \"Blake2b with 'Hello, World!'\"\
+    ,\n\t\t\thasher:      NewBlake2B(),\n\t\t\tinput:       []byte(\"Hello, World!\"\
+    ),\n\t\t\texpectedHex: \"511bc81dde11180838c562c82bb35f3223f46061ebde4a955c27b3f489cf1e03\"\
+    ,\n\t\t},\n\t\t{\n\t\t\tname:        \"Blake2b input at max size\","
 metrics:
 - accuracy
 pipeline_tag: text-classification
 library_name: setfit
 inference: true
+model-index:
+- name: SetFit
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: Unknown
+      type: unknown
+      split: test
+    metrics:
+    - type: accuracy
+      value: 0.9808612440191388
+      name: Accuracy
 ---
 
 # SetFit
@@ -39,6 +69,21 @@ The model has been trained using an efficient few-shot learning technique that i
 - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
 - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
 
+### Model Labels
+| Label            | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+|:-----------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| SAFE_CODE        | <ul><li>'Analyze this clean_code: },\n                        "summary": "Read Items",\n                        "operationId": "read_items_items__get",\n                        "security": [{"OAuth2PasswordBearer": []}],'</li><li>'Analyze this clean_code: :license: Apache 2.0, see LICENSE for more details.\n"""\n\nfrom __future__ import annotations\n\nimport warnings'</li><li>'Analyze this clean_code: it \'sets rel to stylesheet\' do\n      get \'/style\'\n      expect(headers[\'Link\']).to include(\'rel="stylesheet"\')\n    end\n\n    it \'returns html tag\' do'</li></ul>                                                                                                                                                                                                                                   |
+| VULNERABLE_LOGIC | <ul><li>'try {\n        res = await exec(`${ezu} apikey-update --apikey user --attributes doi,is_oa`);'</li><li>'cls()\n        os.system("git clone https://github.com/mrwn007/M3M0")'</li><li>'count = cursor.fetchone()[0]\n\tcursor.execute("SELECT text FROM posts OFFSET random()*" + str(count) + " LIMIT 1;")'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| REAL_SECRET      | <ul><li>'Analyze this hardcoded_secret: OPENAI_API_KEY=sk-proj-Tz9mK2nP7qR4sT6uW8vE5fG'</li><li>'Analyze this hardcoded_secret: $this->data = array (\n\t\t\t1 => new User (1, "tony", 0, \'1c8bfe8f801d79745c4631d09fff36c82aa37fc4cce4fc946683d7b336b63032\'),\n\t\t\t2 => new User (2, "morph", 1, \'e5326ba4359f77c2623244acb04f6ac35c4dfca330ebcccdf9b734e5b1df90a8\'),\n\t\t\t3 => new User (3, "chas", 1, \'a89237fc1f9dd8d424d8b8b98b890dbc4a817bfde59af17c39debcc4a14c21de\'),\n\t\t);\n\t\t$this->requestMethod = $requestMethod;\n\t\t$this->userId = $userId;'</li><li>"openai_key: ['Use sk-abcdefghijklmnopqrstuvwxyz0123456789 for calls', 'sk-abcdefghijklmnopqrstuvwxyz0123456789'],\n  github_token: ['Push with ghp_abcdefghijklmnopqrstuvwxyz0123456789', 'ghp_abcdefghijklmnopqrstuvwxyz0123456789'],"</li></ul> |
+| TEST_MOCK        | <ul><li>'Analyze this test_fixture: expected2 = "y=b\'\\\\xe2\\\\xe1\', x=b\'\\\\xe1\\\\xe2\'"\n        self.assertIn(txt, (expected1, expected2))\n\n    def test_text_kwargs(self):\n        txt = str(utils.log_pprint(kwargs={\'x\': \'ŧêßŧ\', \'y\': \'ŧßêŧ\'}))\n        expected1 = "x=\'ŧêßŧ\', y=\'ŧßêŧ\'"\n        expected2 = "y=\'ŧßêŧ\', x=\'ŧêßŧ\'"'</li><li>'const ACCESS_TOKEN = "AKIAIOSFODNN7EXAMPLE"'</li><li>"Analyze this test_fixture: os.pardir,                                      # /path/to/fboy\n)\n\nMEDIA_ROOT = os.path.join(FACTORY_ROOT, 'tmp_test')\n\nDATABASES = {\n    'default': {"</li></ul>                                                                                                                                                                                                  |
+
+## Evaluation
+
+### Metrics
+| Label   | Accuracy |
+|:--------|:---------|
+| **all** | 0.9809   |
+
 ## Uses
 
 ### Direct Use for Inference
@@ -57,7 +102,7 @@ from setfit import SetFitModel
 # Download from the 🤗 Hub
 model = SetFitModel.from_pretrained("setfit_model_id")
 # Run inference
-preds = model("I loved the spiderman movie!")
+preds = model("Analyze this hardcoded_secret: const db = new Pool({ password: 'Pr0duct10n#2024' });")
 ```
 
 <!--
@@ -86,6 +131,53 @@ preds = model("I loved the spiderman movie!")
 
 ## Training Details
 
+### Training Set Metrics
+| Training set | Min | Median  | Max |
+|:-------------|:----|:--------|:----|
+| Word count   | 1   | 22.5510 | 224 |
+
+| Label            | Training Sample Count |
+|:-----------------|:----------------------|
+| REAL_SECRET      | 113                   |
+| VULNERABLE_LOGIC | 240                   |
+| TEST_MOCK        | 240                   |
+| SAFE_CODE        | 240                   |
+
+### Training Hyperparameters
+- batch_size: (16, 16)
+- num_epochs: (1, 1)
+- max_steps: -1
+- sampling_strategy: oversampling
+- num_iterations: 5
+- body_learning_rate: (2e-05, 1e-05)
+- head_learning_rate: 0.01
+- loss: CosineSimilarityLoss
+- distance_metric: cosine_distance
+- margin: 0.25
+- end_to_end: False
+- use_amp: False
+- warmup_proportion: 0.1
+- l2_weight: 0.01
+- seed: 42
+- eval_max_steps: -1
+- load_best_model_at_end: True
+
+### Training Results
+| Epoch  | Step | Training Loss | Validation Loss |
+|:------:|:----:|:-------------:|:---------------:|
+| 0.0019 | 1    | 0.0013        | -               |
+| 0.0960 | 50   | 0.0125        | -               |
+| 0.1919 | 100  | 0.0052        | -               |
+| 0.2879 | 150  | 0.0097        | -               |
+| 0.3839 | 200  | 0.0039        | -               |
+| 0.4798 | 250  | 0.0032        | -               |
+| 0.5758 | 300  | 0.0015        | -               |
+| 0.6718 | 350  | 0.0016        | -               |
+| 0.7678 | 400  | 0.0011        | -               |
+| 0.8637 | 450  | 0.0013        | -               |
+| 0.9597 | 500  | 0.0022        | -               |
+| 1.0    | 521  | -             | 0.0106          |
+
 ### Framework Versions
 - Python: 3.12.12
 - SetFit: 1.1.3

config_setfit.json

@@ -1,9 +1,9 @@
 {
+  "normalize_embeddings": false,
   "labels": [
     "REAL_SECRET",
     "VULNERABLE_LOGIC",
     "TEST_MOCK",
     "SAFE_CODE"
-  ],
-  "normalize_embeddings": false
+  ]
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:90f75ef471b407684d35375cad34efdf0962e4ead6b1472c3a9d6eef4040c5f7
+oid sha256:d429423c6c6443c185e9b247e0baffd8c275b7203b7805fa8b752a627ba39327
 size 90864192

model_head.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f474dc11bfae94ec29f21ea6cee07d08b605b7b0b55f732d4cb62b0ed937dac
+oid sha256:912b00c8359808d2fe28ed9dd33f89b3cd6566839568ffa4510044c632dc92cb
 size 13191