Update README.md

README.md

@@ -13,7 +13,7 @@ ASMTransformers is a project to train and use a machine learning model to compar
 to aid in the process of reverse engineering.
 
 # Status
-?
+Not actively maintained after publishing
 
 # Relevant links
 * [Github repo ASMtransformers](https://github.com/NetherlandsForensicInstitute/asmtransformers)
@@ -35,7 +35,7 @@ Don't forget to describe that during pretraining, the --mlm-prob parameter has b
 
 Parameters used: epochs=19, eval_steps=10000, batch_size=512, gradient_accumulation_steps=1, mlm_prob=0.4, bf16=True, tf32=True
 
-add estimated time it has cost to train:
+Estimated time it has cost to train:
 pretraining: 33 hours on TODO! 
 finetuning: 4 hours on 1 NVIDIA H200
 
@@ -86,11 +86,27 @@ Amount of functions per architecture per optimization
 ----------------------------
 
 # Preprocessing
-Several preprocessing steps have been taken, that differ slightly between architectures. 
-TODO: update to same as architecture.md in repo
+Several preprocessing steps have been taken: CFGs are processed to become a flat token list suitable for a tokenizer or 
+vocabulary builder. This procedure differs slightly between architectures, the processes can be inspected
+[here](https://github.com/NetherlandsForensicInstitute/asmtransformers/tree/main/asmtransformers/asmtransformers/preprocessors).
+
+Furthermore, some operands have been normalized, 
+to reduce token explosion caused by raw numeric values. These numerical values are expressed as powers of two, such that
+4096 would be 2 ** 12, so #0x2^c, so would 6000, but 9000 would be #0x2^d. As a consequence, small numbers are distinguishable
+in the vocabulary list, but bigger numbers get one batched representation.
+
+Finally, jump tokens are normalised to be relative `JUMP_ADDR_*` tokens (* being the line number of the jump adress in the given 
+function, rather than in the full CFG) CHECK KLOPT DIT?
+
+
 
 # Performance
-Performance was measured in two ways: Mean Reciprocal Rank (MRR) and Accuracy@1
+Performance was measured in two ways: Mean Reciprocal Rank (MRR) and Accuracy@1. These metrics are useful for this problem 
+since we compare a given embedded function to a database of embedded functions, and then rank the database by cosine similarity.
+Mean Reciprocal rank shows the mean rank (1 the positive example in the database being ranked first, 0.5 when the positive example
+is ranked second, 0.25 when the positive example is ranked fourth, etc). I.e. an MRR of 0.66 shows that the positive example was
+usually ranked first or second. Accuracy@1 shows how many times the positive example ranked first. It is a useful metric, 
+but it's also very strict, which is why we used it combined with MRR.
 
 |model|ARM64 mrr|ARM64 acc|AMD64 mrr|AMD64 acc|RISC-V mrr|RISC-V acc|I386 mrr|I386 acc|crosslingual mrr|crosslingual acc|
 |-----|---------|---------|---------|---------|----------|----------|--------|--------|----------------|----------------|