TomasFAV/LiLTInvoiceCzechV01

LiLTInvoiceCzech (V1 – Synthetic + Random Layout)

This model is a fine-tuned version of SCUT-DLVCLab/lilt-roberta-en-base for structured information extraction from Czech invoices.

It achieves the following results on the evaluation set:

• Loss: 0.1907
• Precision: 0.6326
• Recall: 0.7491
• F1: 0.6859
• Accuracy: 0.9660

---

Model description

LiLTInvoiceCzech (V1) extends the baseline layout-aware model by introducing layout variability into the training data.

The model performs token-level classification using both textual and spatial (bounding box) information to extract structured invoice fields:

• supplier
• customer
• invoice number
• bank details
• totals
• dates

Compared to V0, this version is trained on synthetically generated invoices with randomized layouts, improving robustness to spatial variations.

---

Training data

The dataset consists of:

• synthetically generated invoices based on templates
• augmented variants with randomized layout structures
• corresponding bounding box annotations

Key properties:

• variable positioning of fields
• layout perturbations (shifts, spacing, ordering)
• preserved label consistency
• fully synthetic data

This dataset introduces layout diversity, which is especially important for layout-aware models.

---

Role in the pipeline

This model corresponds to:

V1 – Synthetic templates + randomized layouts

It is used to:

• evaluate the effect of layout variability on LiLT
• compare against:
  • V0 (fixed layouts)
  • later stages with hybrid and real data (V2, V3)
• analyze how layout-aware models benefit from synthetic augmentation

---

Intended uses

• Research in layout-aware document understanding
• Evaluation of spatial robustness in NLP models
• Benchmarking LiLT against text-only models (BERT)
• Czech invoice information extraction

---

Limitations

• Still trained only on synthetic data
• Layout variability is artificial
• No real-world noise (OCR errors, distortions)
• May not fully generalize to real invoice distributions

---

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 2
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 10
• mixed_precision_training: Native AMP

---

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
No log	1.0	38	0.1676	0.5917	0.6826	0.6339	0.9639
No log	2.0	76	0.1810	0.6123	0.6604	0.6355	0.9643
No log	3.0	114	0.1906	0.6317	0.7491	0.6854	0.9660
No log	4.0	152	0.1764	0.6380	0.6587	0.6482	0.9659
No log	5.0	190	0.1737	0.6544	0.6689	0.6616	0.9696
No log	6.0	228	0.1752	0.6728	0.6911	0.6818	0.9695
No log	7.0	266	0.1951	0.6083	0.6758	0.6403	0.9658
No log	8.0	304	0.1962	0.6162	0.6741	0.6438	0.9656
No log	9.0	342	0.1939	0.6700	0.6962	0.6828	0.9701
No log	10.0	380	0.1931	0.6645	0.6928	0.6784	0.9696

---

Framework versions

• Transformers 5.0.0
• PyTorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2