Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered
Paper • 2608.29464 • Published • 15
AI & ML interests
A one-year long research workshop on large language models: the Summer of Language Models 21 🌸
Recent Activity
View all activity
pminervini
authored a
paper 12 days ago
pminervini
submitted a
paper to Daily Papers 12 days ago
Muennighoff
submitted a
paper to Daily Papers 17 days ago
Muennighoff
submitted a
paper to Daily Papers 23 days ago
Security test - please ignore
#294 opened 27 days ago
by 
hackeronetestt
BramVanroy
posted an update about 1 month ago
Post
2734
**I benchmarked HF buckets against https access for Common Crawl.**
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Fixed CO2 metadata in README.md
1
#293 opened about 2 months ago
by 
Croc-Prog-HF
Post
5350
who's working on an NVFP4 version of Kimi-K3?
- 4 replies
pminervini
authored 11
papers 2 months ago
Is Complex Query Answering Really Complex?
Paper • 2410.12537 • Published
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
Paper • 2605.27243 • Published
Neurosymbolic Reasoning Shortcuts under the Independence Assumption
Paper • 2507.11357 • Published
Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
Paper • 2605.14787 • Published
Convolutional 2D Knowledge Graph Embeddings
Paper • 1707.01476 • Published
Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
Paper • 1808.08609 • Published
Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions
Paper • 2106.01798 • Published
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language
Paper • 1906.06187 • Published
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Paper • 1910.03065 • Published
Learning Reasoning Strategies in End-to-End Differentiable Proving
Paper • 2007.06477 • Published
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Paper • 2607.01002 • Published • 19
pminervini
submitted a
paper to Daily Papers 2 months ago