End of training

72b1c49 verified 4 months ago

55.6 kB

	---
	tags:
	- ColBERT
	- PyLate
	- sentence-transformers
	- sentence-similarity
	- feature-extraction
	- generated_from_trainer
	- dataset_size:1188486
	- loss:Contrastive
	base_model: jhu-clsp/ettin-encoder-17m
	datasets:
	- benjamintli/code-retrieval-combined-v2-llm-negatives
	pipeline_tag: sentence-similarity
	library_name: PyLate
	---

	# PyLate model based on jhu-clsp/ettin-encoder-17m

	This is a [PyLate](https://github.com/lightonai/pylate) model finetuned from [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) on the [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) dataset. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator.

	## Model Details

	### Model Description
	- Model Type: PyLate model
	- Base model: [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) <!-- at revision 987607455c61e7a5bbc85f7758e0512ea6d0ae4c -->
	- Document Length: 180 tokens
	- Query Length: 32 tokens
	- Output Dimensionality: 128 tokens
	- Similarity Function: MaxSim
	- Training Dataset:
	- [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives)
	<!-- - Language: Unknown -->
	<!-- - License: Unknown -->

	### Model Sources

	- Documentation: [PyLate Documentation](https://lightonai.github.io/pylate/)
	- Repository: [PyLate on GitHub](https://github.com/lightonai/pylate)
	- Hugging Face: [PyLate models on Hugging Face](https://huggingface.co/models?library=PyLate)

	### Full Model Architecture

	```
	ColBERT(
	(0): Transformer({'max_seq_length': 31, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
	(1): Dense({'in_features': 256, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'use_residual': False})
	)
	```

	## Usage
	First install the PyLate library:

	```bash
	pip install -U pylate
	```

	### Retrieval

	Use this model with PyLate to index and retrieve documents. The index uses [FastPLAID](https://github.com/lightonai/fast-plaid) for efficient similarity search.

	#### Indexing documents

	Load the ColBERT model and initialize the PLAID index, then encode and index your documents:

	```python
	from pylate import indexes, models, retrieve

	# Step 1: Load the ColBERT model
	model = models.ColBERT(
	model_name_or_path="colbert-code-17m",
	)

	# Step 2: Initialize the PLAID index
	index = indexes.PLAID(
	index_folder="pylate-index",
	index_name="index",
	override=True, # This overwrites the existing index if any
	)

	# Step 3: Encode the documents
	documents_ids = ["1", "2", "3"]
	documents = ["document 1 text", "document 2 text", "document 3 text"]

	documents_embeddings = model.encode(
	documents,
	batch_size=32,
	is_query=False, # Ensure that it is set to False to indicate that these are documents, not queries
	show_progress_bar=True,
	)

	# Step 4: Add document embeddings to the index by providing embeddings and corresponding ids
	index.add_documents(
	documents_ids=documents_ids,
	documents_embeddings=documents_embeddings,
	)
	```

	Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it:

	```python
	# To load an index, simply instantiate it with the correct folder/name and without overriding it
	index = indexes.PLAID(
	index_folder="pylate-index",
	index_name="index",
	)
	```

	#### Retrieving top-k documents for queries

	Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries.
	To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores:

	```python
	# Step 1: Initialize the ColBERT retriever
	retriever = retrieve.ColBERT(index=index)

	# Step 2: Encode the queries
	queries_embeddings = model.encode(
	["query for document 3", "query for document 1"],
	batch_size=32,
	is_query=True, # # Ensure that it is set to False to indicate that these are queries
	show_progress_bar=True,
	)

	# Step 3: Retrieve top-k documents
	scores = retriever.retrieve(
	queries_embeddings=queries_embeddings,
	k=10, # Retrieve the top 10 matches for each query
	)
	```

	### Reranking
	If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank:

	```python
	from pylate import rank, models

	queries = [
	"query A",
	"query B",
	]

	documents = [
	["document A", "document B"],
	["document 1", "document C", "document B"],
	]

	documents_ids = [
	[1, 2],
	[1, 3, 2],
	]

	model = models.ColBERT(
	model_name_or_path="colbert-code-17m",
	)

	queries_embeddings = model.encode(
	queries,
	is_query=True,
	)

	documents_embeddings = model.encode(
	documents,
	is_query=False,
	)

	reranked_documents = rank.rerank(
	documents_ids=documents_ids,
	queries_embeddings=queries_embeddings,
	documents_embeddings=documents_embeddings,
	)
	```

	<!--
	### Direct Usage (Transformers)

	<details><summary>Click to see the direct usage in Transformers</summary>

	</details>
	-->

	<!--
	### Downstream Usage (Sentence Transformers)

	You can finetune this model on your own dataset.

	<details><summary>Click to expand</summary>

	</details>
	-->

	<!--
	### Out-of-Scope Use

	List how the model may foreseeably be misused and address what users ought not to do with the model.
	-->

	<!--
	## Bias, Risks and Limitations

	What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.
	-->

	<!--
	### Recommendations

	What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.
	-->

	## Training Details

	### Training Dataset

	#### code-retrieval-combined-v2-llm-negatives

	* Dataset: [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) at [1917069](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives/tree/19170694c88339259e15b3dcc7f99db2136bd00a)
	* Size: 1,188,486 training samples
	* Columns: <code>query</code>, <code>positive</code>, <code>source</code>, <code>hard_negatives</code>, and <code>negatives</code>
	* Approximate statistics based on the first 1000 samples:
	\| \| query \| positive \| source \| hard_negatives \| negatives \|
	\|:--------\|:----------------------------------------------------------------------------------\|:-----------------------------------------------------------------------------------\|:---------------------------------------------------------------------------------\|:-----------------------------------\|:-----------------------------------------------------------------------------------\|
	\| type \| string \| string \| string \| list \| string \|
	\| details \| <ul><li>min: 6 tokens</li><li>mean: 23.61 tokens</li><li>max: 32 tokens</li></ul> \| <ul><li>min: 14 tokens</li><li>mean: 31.43 tokens</li><li>max: 32 tokens</li></ul> \| <ul><li>min: 4 tokens</li><li>mean: 7.46 tokens</li><li>max: 10 tokens</li></ul> \| <ul><li>size: 1 elements</li></ul> \| <ul><li>min: 16 tokens</li><li>mean: 31.15 tokens</li><li>max: 32 tokens</li></ul> \|
	* Samples:
	\| query \| positive \| source \| hard_negatives \| negatives \|
	\|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:---------------------------\|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|
	\| <code>wait for AWS PCA CSR propagation before issuing certificate</code> \| <code>func (c ACMPCA) WaitUntilCertificateAuthorityCSRCreated(input GetCertificateAuthorityCsrInput) error {<br> return c.WaitUntilCertificateAuthorityCSRCreatedWithContext(aws.BackgroundContext(), input)<br>}</code> \| <code>csn_syntethic</code> \| <code>['func (c ACMPCA) WaitUntilAuditReportCreated(input DescribeCertificateAuthorityAuditReportInput) error {\n\treturn c.WaitUntilAuditReportCreatedWithContext(aws.BackgroundContext(), input)\n}']</code> \| <code>func (c ACMPCA) WaitUntilAuditReportCreated(input DescribeCertificateAuthorityAuditReportInput) error {<br> return c.WaitUntilAuditReportCreatedWithContext(aws.BackgroundContext(), input)<br>}</code> \|
	\| <code>func (f Filter) Gather() ([]dto.MetricFamily, error) {<br> mfs, err := f.Gatherer.Gather()<br> if err != nil {<br></code> \| <code> return nil, err<br> }<br> return f.Matcher.Match(mfs), nil<br>}</code> \| <code>csn_ccr</code> \| <code>['\n\tf.err = err\n\tif err != nil {\n\t\tfor _, node := range f.cells {\n\t\t\tnode.PropagateWatchError(err)\n\t\t}\n\t}\n}']</code> \| <code><br> f.err = err<br> if err != nil {<br> for _, node := range f.cells {<br> node.PropagateWatchError(err)<br> }<br> }<br>}</code> \|
	\| <code>def latent_to_dist(name, x, hparams, output_channels=None):<br> """Map latent to the mean and log-scale of a Gaussian.<br><br> Args:<br> name: variable scope.<br> x: 4-D Tensor of shape (NHWC)<br> hparams: HParams.<br> latent_architecture - can be "single_conv", "glow_nn" or "glow_resnet",<br> default = single_conv<br> latent_encoder_depth - int, depth of architecture, valid if<br> latent_architecture is "glow_nn" or "glow_resnet".<br> latent_pre_output_channels - 512, valid only when latent_architecture<br> is "glow_nn".<br> latent_encoder_width - 512, maximum width of the network<br> output_channels: int, number of output channels of the mean (and std).<br> if not provided, set it to be the output channels of x.<br> Returns:<br> dist: instance of tfp.distributions.Normal<br> Raises:<br> ValueError: If architecture not in ["single_conv", "glow_nn"]<br> """<br> architecture = hparams.get("latent_a...</code> \| <code> mid_channels=mid_channels)<br> mean_log_scale = conv("glow_nn_zeros", mean_log_scale,<br> filter_size=[3, 3], stride=[1, 1],<br> output_channels=2output_channels,<br> apply_actnorm=False, conv_init="zeros")<br> elif architecture == "glow_resnet":<br> h = x<br> for layer in range(depth):<br> h3 = conv_stack("latent_resnet_%d" % layer, h,<br> mid_channels=width, output_channels=x_shape[-1],<br> dropout=hparams.coupling_dropout)<br> h += h3<br> mean_log_scale = conv("glow_res_final", h, conv_init="zeros",<br> output_channels=2output_channels,<br> apply_actnorm=False)<br> else:<br> raise ValueError("expected architecture to be single_conv or glow_nn "<br> "got %s" % architecture)<br><br> mean = mean_log_scale[:, :, :, 0::2]<br> log_scale = mean_log_scale[:, :, :, 1::2]<br> return tfp.distribu...</code> \| <code>csn_ccr</code> \| <code>[' first_relu=False,\n padding="SAME",\n strides=(2, 2),\n force2d=True,\n name="conv0")\n x = common_layers.conv_block(\n x, 64, [((1, 1), (3, 3))], padding="SAME", force2d=True, name="conv1")\n x = xnet_resblock(x, min(128, hidden_dim), True, "block0")\n x = xnet_resblock(x, min(256, hidden_dim), False, "block1")\n return xnet_resblock(x, hidden_dim, False, "block2")']</code> \| <code> first_relu=False,<br> padding="SAME",<br> strides=(2, 2),<br> force2d=True,<br> name="conv0")<br> x = common_layers.conv_block(<br> x, 64, [((1, 1), (3, 3))], padding="SAME", force2d=True, name="conv1")<br> x = xnet_resblock(x, min(128, hidden_dim), True, "block0")<br> x = xnet_resblock(x, min(256, hidden_dim), False, "block1")<br> return xnet_resblock(x, hidden_dim, False, "block2")</code> \|
	* Loss: <code>pylate.losses.contrastive.Contrastive</code>

	### Evaluation Dataset

	#### code-retrieval-combined-v2-llm-negatives

	* Dataset: [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) at [1917069](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives/tree/19170694c88339259e15b3dcc7f99db2136bd00a)
	* Size: 12,005 evaluation samples
	* Columns: <code>query</code>, <code>positive</code>, <code>source</code>, <code>hard_negatives</code>, and <code>negatives</code>
	* Approximate statistics based on the first 1000 samples:
	\| \| query \| positive \| source \| hard_negatives \| negatives \|
	\|:--------\|:----------------------------------------------------------------------------------\|:-----------------------------------------------------------------------------------\|:---------------------------------------------------------------------------------\|:-----------------------------------\|:-----------------------------------------------------------------------------------\|
	\| type \| string \| string \| string \| list \| string \|
	\| details \| <ul><li>min: 6 tokens</li><li>mean: 23.78 tokens</li><li>max: 32 tokens</li></ul> \| <ul><li>min: 14 tokens</li><li>mean: 31.45 tokens</li><li>max: 32 tokens</li></ul> \| <ul><li>min: 4 tokens</li><li>mean: 7.55 tokens</li><li>max: 10 tokens</li></ul> \| <ul><li>size: 1 elements</li></ul> \| <ul><li>min: 14 tokens</li><li>mean: 31.15 tokens</li><li>max: 32 tokens</li></ul> \|
	* Samples:
	\| query \| positive \| source \| hard_negatives \| negatives \|
	\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:---------------------------\|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|
	\| <code>public static function list_templates($competencyid, $onlyvisible) {<br> global $DB;<br><br> $sql = 'SELECT tpl.*<br> FROM {' . template::TABLE . '} tpl<br> JOIN {' . self::TABLE . '} tplcomp<br> ON tplcomp.templateid = tpl.id<br> WHERE tplcomp.competencyid = ? ';<br> $params = array($competencyid);<br><br> if ($onlyvisible) {<br> $sql .= ' AND tpl.visible = ?';<br> $params[] = 1;<br> }<br><br> </code> \| <code> $sql .= ' ORDER BY tpl.id ASC';<br><br> $results = $DB->get_records_sql($sql, $params);<br><br> $instances = array();<br> foreach ($results as $result) {<br> array_push($instances, new template(0, $result));<br> }<br><br> return $instances;<br> }</code> \| <code>csn_ccr</code> \| <code>[" FROM {assign_grades} g\n JOIN(' . $esql . ') e ON e.id = g.userid\n WHERE g.assignment = :assignid';\n\n return $DB->count_records_sql($sql, $params);\n }"]</code> \| <code> FROM {assign_grades} g<br> JOIN(' . $esql . ') e ON e.id = g.userid<br> WHERE g.assignment = :assignid';<br><br> return $DB->count_records_sql($sql, $params);<br> }</code> \|
	\| <code>python multiprocessing sandboxed process isolation file system network</code> \| <code>def start(self):<br> '''Create a process in which the isolated code will be run.'''<br> assert self._client is None<br><br> logger.debug('IsolationContext[%d] starting', id(self))<br><br> # Create the queues<br> request_queue = multiprocessing.Queue()<br> response_queue = multiprocessing.Queue()<br><br> # Launch the server process<br> server = Server(request_queue, response_queue) # Do not keep a reference to this object!<br> server_process = multiprocessing.Process(target=server.loop)<br> server_process.start()<br><br> # Create a client to talk to the server<br> self._client = Client(server_process, request_queue, response_queue)</code> \| <code>csn_syntethic</code> \| <code>['def start(cls, _init_logging=True):\n """\n Arrange for the subprocess to be started, if it is not already running.\n\n The parent process picks a UNIX socket path the child will use prior to\n fork, creates a socketpair used essentially as a semaphore, then blocks\n waiting for the child to indicate the UNIX socket is ready for use.\n\n :param bool _init_logging:\n For testing, if :data:`False`, don\'t initialize logging.\n """\n if cls.worker_sock is not None:\n return\n\n if faulthandler is not None:\n faulthandler.enable()\n\n mitogen.utils.setup_gil()\n cls.unix_listener_path = mitogen.unix.make_socket_path()\n cls.worker_sock, cls.child_sock = socket.socketpair()\n atexit.register(lambda: clean_shutdown(cls.worker_sock))\n mitogen.core.set_cloexec(cls.worker_sock.fileno())\n mitogen.core.set_cloexec(cls.child_sock.fileno())\n\n cls.profiling = os.environ.get(\'MITOGEN_PROFILING\') is not None\n if cls.profiling:\n mitogen.core.enable_profiling()\n if _init_logging:\n ansible_mitogen.logging.setup()\n\n cls.original_env = dict(os.environ)\n cls.child_pid = os.fork()\n if cls.child_pid:\n save_pid(\'controller\')\n ansible_mitogen.logging.set_process_name(\'top\')\n ansible_mitogen.affinity.policy.assign_controller()\n cls.child_sock.close()\n cls.child_sock = None\n mitogen.core.io_op(cls.worker_sock.recv, 1)\n else:\n save_pid(\'mux\')\n ansible_mitogen.logging.set_process_name(\'mux\')\n ansible_mitogen.affinity.policy.assign_muxprocess()\n cls.worker_sock.close()\n cls.worker_sock = None\n self = cls()\n self.worker_main()']</code> \| <code>def start(cls, _init_logging=True):<br> """<br> Arrange for the subprocess to be started, if it is not already running.<br><br> The parent process picks a UNIX socket path the child will use prior to<br> fork, creates a socketpair used essentially as a semaphore, then blocks<br> waiting for the child to indicate the UNIX socket is ready for use.<br><br> :param bool _init_logging:<br> For testing, if :data:`False`, don't initialize logging.<br> """<br> if cls.worker_sock is not None:<br> return<br><br> if faulthandler is not None:<br> faulthandler.enable()<br><br> mitogen.utils.setup_gil()<br> cls.unix_listener_path = mitogen.unix.make_socket_path()<br> cls.worker_sock, cls.child_sock = socket.socketpair()<br> atexit.register(lambda: clean_shutdown(cls.worker_sock))<br> mitogen.core.set_cloexec(cls.worker_sock.fileno())<br> mitogen.core.set_cloexec(cls.child_sock.fileno())<br><br> cls.profiling = os.environ.get('MI...</code> \|
	\| <code>updates the current cookies with a new set<br><br>@param array $cookies new cookies with which to update current ones<br><br>@return boolean always return true<br>@access private</code> \| <code>function UpdateCookies($cookies) <br> { <br> if (sizeof($this->cookies) == 0) { <br> // no existing cookies: take whatever is new <br> if (sizeof($cookies) > 0) { <br> $this->debug('Setting new cookie(s)'); <br> $this->cookies = $cookies; <br> } <br> <br> return TRUE; <br> } <br> if (sizeof($cookies) == 0) { <br> // no new cookies: keep what we've got <br> return TRUE; <br> } <br> // merge <br> foreach ($cookies as $newCookie) { <br> if (!is_array($newCookie)) { <br> continue; <br> } <br> if ((!isset($newCookie['name'])) \\|\\| (!isset($newCookie['value']))) { <br> continue; <br> } <br> $newName = $newCookie['name']; <br> <br> $found = FALSE; <br> for ($i = 0; $i < count($this->cookies); $i++) { <br> $cookie = $this->cookies[$i]; <br> if (!is_array($cookie)) { <br> continue; <br> ...</code> \| <code>csn</code> \| <code>["public function setCookies(array $cookies) : Request\n {\n $query = http_build_query($cookies);\n parse_str($query, $this->cookies);\n\n if ($cookies) {\n $cookie = str_replace('&', '; ', $query);\n $this->setHeader('Cookie', $cookie);\n } else {\n $this->removeHeader('Cookie');\n }\n\n return $this;\n }"]</code> \| <code>public function setCookies(array $cookies) : Request<br> {<br> $query = http_build_query($cookies);<br> parse_str($query, $this->cookies);<br><br> if ($cookies) {<br> $cookie = str_replace('&', '; ', $query);<br> $this->setHeader('Cookie', $cookie);<br> } else {<br> $this->removeHeader('Cookie');<br> }<br><br> return $this;<br> }</code> \|
	* Loss: <code>pylate.losses.contrastive.Contrastive</code>

	### Training Hyperparameters
	#### Non-Default Hyperparameters

	- `eval_strategy`: steps
	- `per_device_train_batch_size`: 256
	- `per_device_eval_batch_size`: 256
	- `learning_rate`: 3e-06
	- `num_train_epochs`: 1
	- `fp16`: True
	- `hub_model_id`: colbert-code-17m

	#### All Hyperparameters
	<details><summary>Click to expand</summary>

	- `overwrite_output_dir`: False
	- `do_predict`: False
	- `eval_strategy`: steps
	- `prediction_loss_only`: True
	- `per_device_train_batch_size`: 256
	- `per_device_eval_batch_size`: 256
	- `per_gpu_train_batch_size`: None
	- `per_gpu_eval_batch_size`: None
	- `gradient_accumulation_steps`: 1
	- `eval_accumulation_steps`: None
	- `torch_empty_cache_steps`: None
	- `learning_rate`: 3e-06
	- `weight_decay`: 0.0
	- `adam_beta1`: 0.9
	- `adam_beta2`: 0.999
	- `adam_epsilon`: 1e-08
	- `max_grad_norm`: 1.0
	- `num_train_epochs`: 1
	- `max_steps`: -1
	- `lr_scheduler_type`: linear
	- `lr_scheduler_kwargs`: {}
	- `warmup_ratio`: 0.0
	- `warmup_steps`: 0
	- `log_level`: passive
	- `log_level_replica`: warning
	- `log_on_each_node`: True
	- `logging_nan_inf_filter`: True
	- `save_safetensors`: True
	- `save_on_each_node`: False
	- `save_only_model`: False
	- `restore_callback_states_from_checkpoint`: False
	- `no_cuda`: False
	- `use_cpu`: False
	- `use_mps_device`: False
	- `seed`: 42
	- `data_seed`: None
	- `jit_mode_eval`: False
	- `use_ipex`: False
	- `bf16`: False
	- `fp16`: True
	- `fp16_opt_level`: O1
	- `half_precision_backend`: auto
	- `bf16_full_eval`: False
	- `fp16_full_eval`: False
	- `tf32`: None
	- `local_rank`: 0
	- `ddp_backend`: None
	- `tpu_num_cores`: None
	- `tpu_metrics_debug`: False
	- `debug`: []
	- `dataloader_drop_last`: False
	- `dataloader_num_workers`: 0
	- `dataloader_prefetch_factor`: None
	- `past_index`: -1
	- `disable_tqdm`: False
	- `remove_unused_columns`: True
	- `label_names`: None
	- `load_best_model_at_end`: False
	- `ignore_data_skip`: False
	- `fsdp`: []
	- `fsdp_min_num_params`: 0
	- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
	- `fsdp_transformer_layer_cls_to_wrap`: None
	- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
	- `parallelism_config`: None
	- `deepspeed`: None
	- `label_smoothing_factor`: 0.0
	- `optim`: adamw_torch_fused
	- `optim_args`: None
	- `adafactor`: False
	- `group_by_length`: False
	- `length_column_name`: length
	- `ddp_find_unused_parameters`: None
	- `ddp_bucket_cap_mb`: None
	- `ddp_broadcast_buffers`: False
	- `dataloader_pin_memory`: True
	- `dataloader_persistent_workers`: False
	- `skip_memory_metrics`: True
	- `use_legacy_prediction_loop`: False
	- `push_to_hub`: False
	- `resume_from_checkpoint`: None
	- `hub_model_id`: colbert-code-17m
	- `hub_strategy`: every_save
	- `hub_private_repo`: None
	- `hub_always_push`: False
	- `hub_revision`: None
	- `gradient_checkpointing`: False
	- `gradient_checkpointing_kwargs`: None
	- `include_inputs_for_metrics`: False
	- `include_for_metrics`: []
	- `eval_do_concat_batches`: True
	- `fp16_backend`: auto
	- `push_to_hub_model_id`: None
	- `push_to_hub_organization`: None
	- `mp_parameters`:
	- `auto_find_batch_size`: False
	- `full_determinism`: False
	- `torchdynamo`: None
	- `ray_scope`: last
	- `ddp_timeout`: 1800
	- `torch_compile`: False
	- `torch_compile_backend`: None
	- `torch_compile_mode`: None
	- `include_tokens_per_second`: False
	- `include_num_input_tokens_seen`: False
	- `neftune_noise_alpha`: None
	- `optim_target_modules`: None
	- `batch_eval_metrics`: False
	- `eval_on_start`: False
	- `use_liger_kernel`: False
	- `liger_kernel_config`: None
	- `eval_use_gather_object`: False
	- `average_tokens_across_devices`: False
	- `prompts`: None
	- `batch_sampler`: batch_sampler
	- `multi_dataset_batch_sampler`: proportional
	- `router_mapping`: {}
	- `learning_rate_mapping`: {}

	</details>

	### Framework Versions
	- Python: 3.12.3
	- Sentence Transformers: 5.1.1
	- PyLate: 1.4.0
	- Transformers: 4.56.2
	- PyTorch: 2.9.0+cu128
	- Accelerate: 1.13.0
	- Datasets: 4.8.4
	- Tokenizers: 0.22.2


	## Citation

	### BibTeX

	#### Sentence Transformers
	```bibtex
	@inproceedings{reimers-2019-sentence-bert,
	title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
	author = "Reimers, Nils and Gurevych, Iryna",
	booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
	month = "11",
	year = "2019",
	publisher = "Association for Computational Linguistics",
	url = "https://arxiv.org/abs/1908.10084"
	}
	```

	#### PyLate
	```bibtex
	@inproceedings{DBLP:conf/cikm/ChaffinS25,
	author = {Antoine Chaffin and
	Rapha{"{e}}l Sourty},
	editor = {Meeyoung Cha and
	Chanyoung Park and
	Noseong Park and
	Carl Yang and
	Senjuti Basu Roy and
	Jessie Li and
	Jaap Kamps and
	Kijung Shin and
	Bryan Hooi and
	Lifang He},
	title = {PyLate: Flexible Training and Retrieval for Late Interaction Models},
	booktitle = {Proceedings of the 34th {ACM} International Conference on Information
	and Knowledge Management, {CIKM} 2025, Seoul, Republic of Korea, November
	10-14, 2025},
	pages = {6334--6339},
	publisher = {{ACM}},
	year = {2025},
	url = {https://github.com/lightonai/pylate},
	doi = {10.1145/3746252.3761608},
	}
	```

	<!--
	## Glossary

	Clearly define terms in order to be accessible across audiences.
	-->

	<!--
	## Model Card Authors

	Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.
	-->

	<!--
	## Model Card Contact

	Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.
	-->