Pre-release review fixes: code robustness, card corrections

CREDITS_BOOKS.tsv

@@ -1,8 +1,10 @@
-# Source-1: per-work credits for the books in its training and validation splits that are not public domain or
-# CC0 (2,846 works). Each work is credited to the authors and publishers named in it, under the licence recorded
-# for it by the platform it came from (for three works, the IGO licence their own text states; the licence cell
-# says so). Modified: used to train a classifier. No text of these works is distributed with Source-1. This file
-# is part of NOTICE (see NOTICE, part 3).
+# Source-1: per-work credits for the books in the open-books part of its training and validation splits that are
+# not public domain or CC0 (2,846 works). Books and book chapters that came through Common Pile v0.1 (DOAB,
+# Pressbooks, LibreTexts, OER Commons) are credited at collection level in NOTICE 3.5. Each work is credited to
+# the authors and publishers named in it, under the licence recorded for it by the platform it came from (for
+# three works, the IGO licence their own text states; the licence cell says so). Modified: used to train a
+# classifier. No text of these works is distributed with Source-1. This file is part of NOTICE (see NOTICE,
+# part 3).
 # citation_or_note: the citation or attribution that the work's own text asks for, as the work gives it (line
 # breaks joined, PDF spacing repaired), for FAO books, OpenStax textbooks, Eurydice, JRC and other EU reports, and
 # some other books and reports (university-press books, Frontiers ebooks, research and project reports); or a
@@ -107,7 +109,7 @@ DOAB	Prácticas lingüísticas heterogéneas: Nuevas perspectivas para el estudi
 DOAB	Questioni di donne: Diplomazia informale e reti femminili alla corte dei Savoia-Carignano (XVII secolo)	Lurgo, Elisabetta	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://directory.doabooks.org/handle/20.500.12854/134564	
 DOAB	Religion og etikk i skole og barnehage	Afset, Bente; Redse, Arne	no	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://directory.doabooks.org/handle/20.500.12854/38379	
 DOAB	Réinventer l’art sacré: Le Groupe de Saint-Luc (1919-1945)	Noverraz, Camille	fr	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0)	https://directory.doabooks.org/handle/20.500.12854/152369	
-DOAB	Sjezd českých právníků 2022		cs	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://directory.doabooks.org/handle/20.500.12854/98695	
+DOAB	Sjezd českých právníků 2022	Jednota českých právníků (collective of authors, no editors recorded)	cs	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://directory.doabooks.org/handle/20.500.12854/98695	
 DOAB	Slovanský literární svět: kontexty a konfrontace III: Motiv domova ve slovanských literaturách	Bujnáková, Jana; Cepková Feješová, Zuzana; Derková, Vladimíra; Eniko, Mateja; Heinigová, Lenka; Hrancová, Hana	cs	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	https://directory.doabooks.org/handle/20.500.12854/80840	
 DOAB	Somatopedické simulační techniky a intervence: Metodické texty k projektu MUNI 4.0. Pedagogická fakulta, studijní program Logopedie (Bc.)	Opatřilová, Dagmar	cs	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	https://directory.doabooks.org/handle/20.500.12854/80845	
 DOAB	Speciálněpedagogická diagnostika somatopedická: Metodické texty k projektu MUNI 4.0. Pedagogická fakulta, studijní program Logopedie (Bc.)	Opatřilová, Dagmar	cs	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	https://directory.doabooks.org/handle/20.500.12854/80846	
@@ -767,7 +769,7 @@ FAO	Combattre la criminalité liée aux forêts en Afrique de l'Ouest	Chasi, R.M
 FAO	Commission générale des pêches pour la Méditerranée: Rapport de la quarante-huitième session	FAO;	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd9408fr	Citation requested in the work: FAO. 2026. Commission générale des pêches pour la Méditerranée: Rapport de la quarante-huitième session – Malaga, Espagne, 4-9 novembre 2025. Commission générale des pêches pour la Méditerranée (CGPM) – Rapports de session, n°48. Rome. https://doi.org/10.4060/cd9408fr
 FAO	Compendium of decisions made by the Parties to the FAO Agreement on Port State Measures	FAO;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0286en	Citation requested in the work: FAO. 2026. Compendium of decisions made by the Parties to the FAO Agreement on Port State Measures. Second edition. Rome. https://doi.org/10.4060/ce0286en
 FAO	Comptes rendus du deuxième Sommet sur la pêche artisanale	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd3997fr	Citation requested in the work: FAO. 2025. Comptes rendus du deuxième Sommet sur la pêche artisanale, 5-7 juillet 2024, Rome. FAO Comptes rendus des pêches et de l’aquaculture, n° 70. Rome. https://doi.org/10.4060/cd3997fr
-FAO	Cостояние мирового рыболовства и аквакультуры – 2026	​ФАО;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd8357ru	Citation requested in the work: "ФАО. 2026. Cостояние мирового рыболовства и аквакультуры – 2026. ""Голубая трансформация"": от замысла к практическим результатам. Рим. https://doi.org/10.4060/cd8357ru"
+FAO	Cостояние мирового рыболовства и аквакультуры – 2026	ФАО;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd8357ru	Citation requested in the work: "ФАО. 2026. Cостояние мирового рыболовства и аквакультуры – 2026. ""Голубая трансформация"": от замысла к практическим результатам. Рим. https://doi.org/10.4060/cd8357ru"
 FAO	Design of a climate-proof fish buying station	Josupeit, H.; Moretti, S.; Sciortino, J.A.; Urbani, R.; van Anrooy, R.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce1081en	Citation requested in the work: Josupeit, H., Moretti, S., Sciortino, J.A., Urbani, R. & Van Anrooy, R. 2026. Design of a climate-proof fish buying station. FAO Fisheries and Aquaculture Technical Paper, No. 691. Rome, FAO. https://doi.org/10.4060/ce1081en
 FAO	Developing holistic nutrition guidelines and standards for school meals	FAO; WFP;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce1121en	Citation requested in the work: FAO and WFP. 2026. Developing holistic nutrition guidelines and standards for school meals – A global methodology. Rome. https://doi.org/10.4060/ce1121en
 FAO	Developing nutrition-sensitive value chains	Andrianarimanana, M.; Galante, A.; Liu, B.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0611en	Citation requested in the work: Andrianarimanana, M., Galante, A. & Liu, B. 2026. Developing nutrition‑sensitive value chains – Guidelines for practitioners. Rome, FAO. https://doi.org/10.4060/ce0611en
@@ -818,7 +820,7 @@ FAO	Roles and values of camelids and their products	FAO;	en	CC BY 4.0	https://op
 FAO	Scaling up community-base fisheries management in the Pacific	Govan, H.; Tuxson, T.; Tauati, M.; Lalavanua, W.; Schwarz, A.; Kinch, J.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce1087en	Citation requested in the work: Govan, H., Tuxson,T., Tauati, M., Lalavanua, W., Schwarz, A. & Kinch, J. 2026. Scaling up community-based fisheries management in the Pacific – Outlook and prospects for securing sustainable coastal fisheries, livelihoods and ecosystems. Rome, FAO; Noumea, Pacific Community. https://doi.org/10.4060/ce1087en
 FAO	Seed to sip: Arabica coffee production guide for Saudi Arabia	Gichimu, B.M.; Ghosh, K.; Alfaifi, B.H.; Alfaifi, K.A.; Al Mutlaq, A.M.; Bustamante Adum, D.; Lubabali, H.A.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0725en	Citation requested in the work: Gichimu, B.M, Ghosh, K., Alfaifi, B.H., Alfaifi, K.A., Al Mutlaq, A.M., Bustamante Adum, D. & Lubabali, H.A. 2026. Seed to sip: Arabica coffee production guide for Saudi Arabia. Riyadh, FAO. https://doi.org/10.4060/ce0725en
 FAO	Shaping agrifood systems legislation	Rosenbaum, K.L.; Vidar, M.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce1406en	Citation requested in the work: Rosenbaum, K.L and Vidar, M. 2026. Shaping agrifood systems legislation – Good practices for legal advisors in drafting laws and supporting related processes. Legal Guide No. 5. Rome, FAO. https://doi.org/10.4060/ce1406en
-FAO	Soil health and fertilizer	Jafari, A.; Le Cotty, T.; Tefft, J.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd9916en	Citation requested in the work: Jafari, A., Le Cotty, T. & Tefft, J. 2026. Soil health and fertilizer – Policy and investment prospects in sub-Saharan Africa. Directions in Investment no. 19. Rome, Paris and Brussels, FAO, Agrinatura and the European Union. https://doi. org/10.4060/cd9916en
+FAO	Soil health and fertilizer	Jafari, A.; Le Cotty, T.; Tefft, J.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd9916en	Citation requested in the work: Jafari, A., Le Cotty, T. & Tefft, J. 2026. Soil health and fertilizer – Policy and investment prospects in sub-Saharan Africa. Directions in Investment no. 19. Rome, Paris and Brussels, FAO, Agrinatura and the European Union. https://doi.org/10.4060/cd9916en
 FAO	Status of the world's soil resources 2026	FAO; ITPS;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0792en	Citation requested in the work: FAO and ITPS. 2026. Status of the world's soil resources 2026. Rome, FAO. https://doi.org/10.4060/ce0792en
 FAO	Strengthening small and medium agroenterprise finance in the Near East and North Africa region	Aldredge, H.; Priebe, J.; Zook, D.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0863en	Citation requested in the work: Aldredge, H., Priebe, J. & Zook, D. 2026. Strengthening small and medium agroenterprise finance in the Near East and North Africa region: Bridging the finance gap. Rome, FAO. https://doi.org/10.4060/ce0863en
 FAO	Sổ tay hỏi đáp	Phùng, T.V.; Tôn, V.Đ.; Sơn, T.H; Phục, N.N.;	vi	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0930vi	Citation requested in the work: Phùng, T.V., Tôn, V.Đ, Sơn, T.H. and Phục, N.N. 2026. Sổ tay hỏi đáp về th c h nh t t an to n sinh học v xử lý chất thải trong chăn nuôi lợn quy mô vừa v nhỏ. Hà Nội, FAO.
@@ -832,14 +834,14 @@ FAO	Vodič za odgovornu i racionalnu primenu antibiotika kod goveda	FAO;	sr	CC B
 FAO	Vodič za odgovornu i racionalnu primenu antibiotika kod ovaca i koza	Krnjaić, D.; Savić, B.; Trailović, S.;	sr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0604sr	Citation requested in the work: Krnjaić, D, Savić, B, Trailović, S. 2026. Vodič za odgovornu i racionalnu primenu antibiotika kod ovaca i koza. Beograd, FAO.
 FAO	Vodič za odgovornu i racionalnu primenu antibiotika kod pasa i mačaka	Krnjaić, D.; Resanović, R,;	sr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0991sr	Citation requested in the work: Krnjaić, D, Resanović, R, Trailović, S. 2026. Vodič za odgovornu i racionalnu primenu antibiotika kod pasa i mačaka . Beograd, FAO.
 FAO	Voices of cooperators: A collective journey through the agrifood systems in the Near East and North Africa	FAO;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0708en	Citation requested in the work: FAO. 2026. Voices of cooperators: A collective journey through the agrifood systems in the Near East and North Africa region. Cairo. https://doi.org/10.4060/ce0708en
-FAO	Wood products in the bioeconomy	Reck, B.K.; Johnston, C.; Foong, A.; Gupta, A.; Karpov, A.; Holsten, A.; Misselwitz, P.; Keenan, R.J.; Formenton Cardoso, N.; Walter, S.; Bull, L.; Steel, E.A.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0315en	Citation requested in the work: Reck, B.K., Johnston, C., Foong, A., Gupta, A., Karpov, A., Holsten, A., Misselwitz, P., Keenan, R.J., Formenton Cardoso, N., Walter, S., Bull, L., and Steel, E.A. 2026. Wood products in the bioeconomy: Scenario-based assessment of the potential for engineered wood products in climate change mitigation. Rome, FAO. DOI https:// doi.org/10.4060/ce0315en
+FAO	Wood products in the bioeconomy	Reck, B.K.; Johnston, C.; Foong, A.; Gupta, A.; Karpov, A.; Holsten, A.; Misselwitz, P.; Keenan, R.J.; Formenton Cardoso, N.; Walter, S.; Bull, L.; Steel, E.A.;	en	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/ce0315en	Citation requested in the work: Reck, B.K., Johnston, C., Foong, A., Gupta, A., Karpov, A., Holsten, A., Misselwitz, P., Keenan, R.J., Formenton Cardoso, N., Walter, S., Bull, L., and Steel, E.A. 2026. Wood products in the bioeconomy: Scenario-based assessment of the potential for engineered wood products in climate change mitigation. Rome, FAO. DOI https://doi.org/10.4060/ce0315en
 FAO	Évaluation du projet «Consolidation de la paix dans la zone frontalière du nord-est de la Côte d’Ivoire par le biais d’une gestion participative et planifiée des ressources naturelles»	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd3911fr	Citation requested in the work: FAO. 2025. Évaluation du projet «Consolidation de la paix dans la zone frontalière du nord-est de la Côte d’Ivoire, par le biais d’une gestion participative et planifiée des ressources naturelles» – Code du projet: UNJP/IVC/037/PBF. Série évaluation de projet, 01/2025. Rome. https://doi.org/10.4060/cd3911fr
 FAO	Évaluation du projet «Promouvoir une production de cacao sans déforestation pour réduire les émissions en Côte d’Ivoire»	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd3686fr	Citation requested in the work: FAO. 2025. Évaluation du projet «Promouvoir une production de cacao sans déforestation pour réduire les émissions en Côte d’Ivoire» Rapport de mi-parcous, code du projet: GCP/IVC/609/GCF. Série évaluation de projet, n° 50/2024. Rome. https://doi.org/10.4060/cd3686fr
 FAO	Évaluation du projet «Soutien à l’auto-emploi de la jeunesse rurale, vecteur de paix et de cohésion sociale au Mali»	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd7607fr	Citation requested in the work: FAO. 2025. Évaluation du projet «Soutien à l’auto-emploi de la jeunesse rurale, vecteur de paix et de cohésion sociale au Mali» Code du projet: UNJP/MLI/068/PBF. Série évaluation de projet, n. 26/2025. Rome. https://doi.org/10.4060/cd7607fr
 FAO	Évaluation finale du projet «Revitaliser les agroécosystèmes oasiens à travers une approche durable, intégrée et paysagère dans la région de Drâa Tafilalet»	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd5845fr	Citation requested in the work: FAO. 2025. Évaluation finale du projet «Revitaliser les agroécosystèmes oasiens à travers une approche durable, intégrée et paysagère dans la région de Drâa Tafilalet» - Code du projet: GCP/MOR/046/GFF. Série évaluation de projet, N.° 15/2025. Rome. https://doi.org/10.4060/cd5845fr
 FAO	Évaluation finale du projet «Élimination des polluants organiques persistants et pesticides obsolètes et renforcement de la gestion rationnelle des pesticides au Cameroun»	FAO	fr	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd5148fr	Citation requested in the work: FAO. 2025. Évaluation finale du projet «Élimination des polluants organiques persistants et pesticides obsolètes et renforcement de la gestion rationnelle des pesticides au Cameroun» – Code du projet: GCP/CMR/031/GFF, Identifiant FEM: 4641. Série Évaluation de projet, 12/2025 Rome. https://doi.org/10.4060/cd5148fr
-FAO	Анализ кооперативного законодательства Российской Федерации и других стран СНГ	Клименко, О.И.; Кондракова, И.А.; Мадыгина, О.А.; Горячковская, Ю.М.; Яковлев, В.И.; Касулина, В.В.;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd5542ru	Citation requested in the work: Клименко О.И., Кондракова И. А., Мадыгина О. А., Горячковская Ю. М., Яковлев В.И., Касулина В.В. 2025. Анализ кооперативного законодательства Российской Федерации и других стран СНГ. ФАО Законодательное исследование № 119. Рим, ФАО. https://doi. org/10.4060/cd5542ru
-FAO	Комиссия Кодекс Алиментариус. Руководство по процедуре	​ФАО; ВОЗ;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd7978ru	Citation requested in the work: ФАО и ВОЗ. 2026. Комиссия Кодекс Алиментариус. Руководство по процедуре. Тридцать первое издание. Рим. https://doi.org/10.4060/cd7978ru
+FAO	Анализ кооперативного законодательства Российской Федерации и других стран СНГ	Клименко, О.И.; Кондракова, И.А.; Мадыгина, О.А.; Горячковская, Ю.М.; Яковлев, В.И.; Касулина, В.В.;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd5542ru	Citation requested in the work: Клименко О.И., Кондракова И. А., Мадыгина О. А., Горячковская Ю. М., Яковлев В.И., Касулина В.В. 2025. Анализ кооперативного законодательства Российской Федерации и других стран СНГ. ФАО Законодательное исследование № 119. Рим, ФАО. https://doi.org/10.4060/cd5542ru
+FAO	Комиссия Кодекс Алиментариус. Руководство по процедуре	ФАО; ВОЗ;	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd7978ru	Citation requested in the work: ФАО и ВОЗ. 2026. Комиссия Кодекс Алиментариус. Руководство по процедуре. Тридцать первое издание. Рим. https://doi.org/10.4060/cd7978ru
 FAO	Положение дел в области продовольствия и сельского хозяйства 2024	ФАО	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd2616ru	Citation requested in the work: ФАО. 2024. Положение дел в области продовольствия и сельского хозяйства – 2024. Преобразование агропродовольственных систем с ориентацией на ценностные параметры. Рим. https://doi.org/10.4060/cd2616ru
 FAO	Положение дел в области продовольствия и сельского хозяйства 2025	FAO	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd7067ru	Citation requested in the work: ФАО. 2025. Положение дел в области продовольствия и сельского хозяйства – 2025. Решение проблемы деградации почв с учетом масштабов землевладений. Рим. https://doi.org/10.4060/cd7067ru
 FAO	Положение дел на рынках сельскохозяйственной продукции – 2024	FAO	ru	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd2144ru	Citation requested in the work: ФАО. 2024. Положение дел на рынках сельскохозяйственной продукции – 2024. Торговля и питание: согласованность политики в интересах обеспечения здорового рациона. Рим. https://doi.org/10.4060/cd2144ru
@@ -848,12 +850,12 @@ FAO	Состояние мировых земельных и водных рес
 FAO	全球黑土现状	FAO	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc3124zh	Citation requested in the work: 。中国北京，中国农业出版社。https://doi.org/10.4060/ 粮农组织。2025。《全球黑土现状》 cc3124zh 20-CP 语学院安排并对翻译的准确性及质量负全部责任。如有出入，应以英文原版为准。
 FAO	兽药残留对肠道微生物组和人体健康的影响	FAO	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc5301zh	Citation requested in the work: 粮农组织。2025。《兽药残留对肠道微生物组和人体健康的影响——食品安全视角》。中国 北京，中国农业出版社。https://doi.org/10.4060/cc5301zh 20-CP
 FAO	农药残留对肠道微生物组和人体健康的影响	FAO	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc5306zh	Citation requested in the work: 粮农组织。2025。《农药残留对肠道微生物组和人体健康的影响——食品安全视角》。中国 北京，中国农业出版社。https://doi.org/10.4060/cc5306zh 20-CP
-FAO	卓越数字农业报告	​FAO; ITU	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc4764zh	Citation requested in the work: 粮 农 组 织 和 国 际 电 信 联 盟。2025。《 卓 越 数 字 农 业 报 告 —— 粮 农 组 织 与 国 际 电 联 促 进 欧洲和中亚数字农业良好做法区域竞赛》。中国北京，中国农业出版社。https://doi.org/ 10.4060/cc4764zh 20-CPP2021
+FAO	卓越数字农业报告	FAO; ITU	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc4764zh	Citation requested in the work: 粮农组织和国际电信联盟。2025。《卓越数字农业报告——粮农组织与国际电联促进欧洲和中亚数字农业良好做法区域竞赛》。中国北京，中国农业出版社。https://doi.org/10.4060/cc4764zh 20-CPP2021
 FAO	塑料挑战徽章训练手册	粮农组织;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd0922zh	Citation requested in the work: 粮农组织。2026。《塑料挑战徽章训练手册》。青年与联合国全球联盟学习和行动系列—— 挑战徽章⑬。中国北京，中国农业出版社。https://doi.org/10.4060/cd0922zh
 FAO	满足味蕾的养殖水产品	FAO	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc5140zh	Citation requested in the work: 《满足味蕾的养殖水产品——探索十二种地中海与黑海鱼类从海洋至餐桌之 粮农组织。2025。 旅》。中国北京，中国农业出版社。https://doi.org/10.4060/cc5140zh
 FAO	盐碱土探秘	FAO; IUSS;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc0530zh	Citation requested in the work: 粮农组织和国际土壤科学联合会。2025。《盐碱土探秘——全球精选十大儿童科普故事》。 中国北京，中国农业出版社。https://doi.org/10.4060/cc0530zh
 FAO	社会保护与前瞻行动——保护农业生计	粮农组织;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc7628zh	Citation requested in the work: 粮农组织。2026。《社会保护与前瞻行动——保护农业生计》。中国北京，中国农业出版社。 https://doi.org/10.4060/cc7628zh 20-CP 语学院安排并对翻译的准确性及质量负全部责任。如有出入，应以英文原版为准。
-FAO	细胞基食品食用安全解析	​ 粮农组织; 世界卫生组织;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc4855zh	Citation requested in the work: 粮农组织和世卫组织。2025。《细胞基食品食用安全解析》 。中国北京，中国农业出版社。 https://doi.org/10.4060/cc4855zh 20-CP。
+FAO	细胞基食品食用安全解析	粮农组织; 世界卫生组织;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc4855zh	Citation requested in the work: 粮农组织和世卫组织。2025。《细胞基食品食用安全解析》 。中国北京，中国农业出版社。 https://doi.org/10.4060/cc4855zh 20-CP。
 FAO	能源挑战徽章：生物能源补充训练手册	粮农组织;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd1397zh	Citation requested in the work: 粮农组织。2026。《能源挑战徽章：生物能源补充训练手册》。青年与联合国全球联盟学 习和行动系列。中国北京，中国农业出版社。https://doi.org/10.4060/cd1397zh
 FAO	食品法典委员会程序手册	FAO; WHO;	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cd7978zh	Citation requested in the work: 粮农组织和世卫组织。2026。《食品法典委员会程序手册》。第三十一版。罗马。https://doi.org/10.4060/cd4216zh
 FAO	鱼：知之，烹之，食之	FAO	zh	CC BY 4.0	https://openknowledge.fao.org/handle/20.500.14283/cc1395zh	Citation requested in the work: 粮农组织。2025。 《鱼：知之，烹之，食之》。中国北京，中国农业出版社。https://doi.org/10.4060/ cc1395zh
@@ -933,7 +935,7 @@ Kanripo	周禮疑義擧要	江永 (清)	zh	CC BY-SA 4.0	https://github.com/kanri
 Kanripo	嘉靖以來首輔傳	王世貞 (明)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR2g0037	
 Kanripo	埤雅	陸佃 (宋)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR1j0011	
 Kanripo	天經惑問	游藝 (清)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR3f0023	
-Kanripo	尚書(正文)		zh	CC BY-SA 4.0	https://github.com/kanripo/KR1b0001	
+Kanripo	尚書(正文)	unknown (traditionally attributed to 孔子, Confucius, as compiler)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR1b0001	
 Kanripo	尚書全解	林之奇 (宋)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR1b0007	
 Kanripo	尚書疑義	馬明衡 (明)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR1b0039	
 Kanripo	折獄龜鑑	鄭克 (宋)	zh	CC BY-SA 4.0	https://github.com/kanripo/KR3c0007	
@@ -1028,13 +1030,13 @@ NDLA	NDLA: Yrkesfaglig fordypning (EL-ELE vg1)	Albertine Aaberge; Bjørn Dølvin
 NDLA	NDLA: Yrkesliv i barne- og ungdomsarbeiderfag (HS-BUA vg2)	Bente Elisabeth Vetland; Camilla Øvstebø ; Cathrine Dunker Furuly; Einar Martin Kålen; Gro Nedberg Grønlid; Guri Bente Hårberg; Hege Nikolaisen; Karl Henrik Aanesen; Kristin Aase; Kristin Sundstrøm; Riborg Anna Ringereide; Rita Enstad-Karlsen, Terranova Media; Siv Stai; Siv Stai ; Tove Engesvik; Vig	no	CC BY-SA 4.0	https://ndla.no/subject:1:03e810db-3560-47b5-a5f6-e7afe1d0a2d6	
 NDLA	NDLA: Yrkesliv i helsearbeiderfag (HS-HEA vg2)	Albertine Aaberge; Aleksandra Krogh; Birgit Flaten; Einar Martin Kålen; Hege Nikolaisen; Hege Nikolaisen ; Helene Grotle; Ingrid Schiefloe Myhre; Johannes Leiknes Nag; Karl Henrik Aanesen; Kathrine Synnøve Karlsen; Lars Sandlie/Høgskolen i Lillehammer; Lene Fossbråten; Marit Smith Sørhøy; NAKU; Odd	no	CC BY-SA 4.0	https://ndla.no/subject:1:f644f829-4e7a-4e74-a63a-342ef786f68a	
 OAPEN	100 Cartas para Paulo Freire de quienes pretendemos Enseñar	Gárate Vergara, Francisco	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/51168	
-OAPEN	20 år med fysikkprestasjoner i fritt fall: Analyser fra TIMSS Advanced og andre internasjonale studier	Hole, Arne; Onstad, Torgeir; Hagen, Tor Espen	no	CC BY (OAPEN per-file license code CC-BY)	http://library.oapen.org/handle/20.500.12657/23254	
+OAPEN	20 år med fysikkprestasjoner i fritt fall: Analyser fra TIMSS Advanced og andre internasjonale studier	Hole, Arne; Onstad, Torgeir; Hagen, Tor Espen	no	CC BY (version not recorded)	http://library.oapen.org/handle/20.500.12657/23254	
 OAPEN	Ai margini del contado: Terra, signoria ed élites locali a Sabbion e nel territorio di Cologna Veneta (secoli XII-XIII)	STELLA, Attilio	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60437	
 OAPEN	Bekymringsarbeidet: Politiets forebygging av radikalisering og voldelig ekstremisme	Førde, Kristin Engh; Andersen, Arnfinn Jomar; Moum Hellevik, Per	no	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/63675	Citation requested in the work: Førde, K. E, Andersen, A. J. & Hellevik, P. M. (2023). Bekymringsarbeidet. Politiets forebygging av radikalisering og voldelig ekstremisme. Cappelen Damm Akademisk. https://doi.org/10.23865/noasp.185
 OAPEN	Bewältigung des Scheiterns: Autobiographische Schriften früherer Parteifunktionäre von NSDAP und SED	Danner, Hans-Ulrich	de	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0)	https://library.oapen.org/handle/20.500.12657/91039	
 OAPEN	Bologna dopo la pandemia: Impatto territoriale e scenari futuri	Castrignanò, Marco; RIMONDI, TOMMASO	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60518	
 OAPEN	Construyendo espacios: la ciudad iberoamericana virreinal: Teoría y estudios de caso	Paniagua Pérez, Jesús; Arciello, Daniele	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0)	https://library.oapen.org/handle/20.500.12657/51907	
-OAPEN	Corsi universitari	Fortini, Franco	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/89263	
+OAPEN	Corsi universitari	Fortini, Franco	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/89263	
 OAPEN	Das kolonisierte Heiligtum: Diskriminierungskritische Perspektiven auf das Verfahren der Musealisierung	Balzar, Christoph	de	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	https://library.oapen.org/handle/20.500.12657/60744	
 OAPEN	Das sogenannte ‚Königliche Gerichtsbuch‘ – Aufzeichnungen des Michael von Pfullendorf zu den Anfängen des Kammergerichts am römisch-deutschen Königshof (1442 bis 1451): Einführung und Edition	Luger, Daniel	de	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/53427	
 OAPEN	Del Palacio Negro a la Selva Lacandona: Louis Althusser en México	Ortega Reyna, Jaime	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/63196	
@@ -1045,27 +1047,27 @@ OAPEN	Documentación digital y léxico en la traducción e interpretación en lo
 OAPEN	En contra de los impíos. La Actuación de la Buena Prensa Católica en la Arquidiócesis de Santiago, 1906-1936	Loyola, Manuel	es	CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/)	http://library.oapen.org/handle/20.500.12657/32093	
 OAPEN	Esercizi di ricerca: Dottorato e politiche per la formazione	Boffo, Vanna; Togni, Fabio	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/62865	
 OAPEN	Estudios Interculturales desde el Sur: procesos, debates y propuestas	Samaniego, Mario	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/50252	
-OAPEN	Europa: um projecto em construção: Homenagem a David Sassoli	Graziani, Michela; Rita, Annabela	pt	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/62866	
+OAPEN	Europa: um projecto em construção: Homenagem a David Sassoli	Graziani, Michela; Rita, Annabela	pt	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/62866	
 OAPEN	Fabulations nocturnes: Écologie, vitalité et opacité dans le cinéma d’Apichatpong Weerasethakul	Bordeleau, Érik; Pape, Toni; Rose-Antoinette, Ronald; Szymanski, Adam	fr	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	http://library.oapen.org/handle/20.500.12657/31350	
 OAPEN	Fallbuch Asylrecht: Mit Bezügen zum Aufenthaltsrecht	Mantel, Johanna; Nachtigall, Rhea; Wasnick, Lars	de	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)	https://library.oapen.org/handle/20.500.12657/63515	
 OAPEN	Firenze prima degli Uberti: Il ceto dirigente fiorentino nell'XI secolo fra riforme diocesane e affermazione personale e familiare	Contessa, Maria Pia	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/62867	
-OAPEN	Francesco da Barberino al crocevia: Culture, società, bilinguismo	Bischetti, Sara; Montefusco, Antonio	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/52298	
+OAPEN	Francesco da Barberino al crocevia: Culture, società, bilinguismo	Bischetti, Sara; Montefusco, Antonio	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/52298	
 OAPEN	Fuentes para una Constitución con Poder Indígena	Valenzuela, Esteban; Romero, Natacha	es	CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)	http://library.oapen.org/handle/20.500.12657/32033	
 OAPEN	Förändring och kontinuitet: Reformer inom högre utbildning och forskning 1940–2020	Gribbe, Johan	sv	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0)	https://library.oapen.org/handle/20.500.12657/59842	Citation requested in the work: [Johan Gribbe, 2022, Förändring och kontinuitet: Reformer inom högre utbildning och forskning 1940–2020] Universitetskanslersämbetet. DOI: https://doi.org/10.53340/UKAP-4. Licens: CC-BY 4.0
 OAPEN	Führt Moral unumgänglich zur Religion?: Zur Kritik der Kantischen Religionsphilosophie bei Jürgen Habermas – eine Entgegnung	Langthaler, Rudolf	de	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/51766	
 OAPEN	Geplante Obsoleszenz: Hinter den Kulissen der Produktentwicklung	Poppe, Erik; Longmuß, Jörg	de	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/24340	
-OAPEN	Gli altri noi: Rom e residenti nella Svizzera italiana: etnografia a mediazione	Bizzini, Nadia	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/41430	
+OAPEN	Gli altri noi: Rom e residenti nella Svizzera italiana: etnografia a mediazione	Bizzini, Nadia	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/41430	
 OAPEN	Gouvernance du secteur de la Sécurité: Leçons des expériences ouest-africaines	Bryden, Alan; Chappuis, Fairlie	fr	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/32914	Citation requested in the work: Bryden, A et Chappuis, F (dir. publ.) 2015 Gouvernance du secteur de la Sécurité : Leçons des expériences ouest-africaines. London: Ubiquity Press. DOI: http://dx.doi.org/10.5334/bav. Licence: CC-BY 4.0
-OAPEN	Guida al mentoring: Aiutare mentori e allievi ad avere successo	Chopra, Vineet; Vaughn, Valerie; Saint, Sanjay	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/96163	
+OAPEN	Guida al mentoring: Aiutare mentori e allievi ad avere successo	Chopra, Vineet; Vaughn, Valerie; Saint, Sanjay	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/96163	
 OAPEN	Guiden till Spaniensverige: Diaspora, integration och transnationalitet bland svenska föreningar i södra Spanien	Olsson, Erik	sv	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/27488	Citation requested in the work: Olsson, Erik. 2018. Guiden till Spaniensverige: Diaspora, integration och transnationalitet bland svenska föreningar i södra Spanien. Stockholm: Stockholm University Press. DOI: https://doi.org/10.16993/bao. License: CC-BY
 OAPEN	Hacia una historia de las tendencias trotskistas después de Trotsky	Gaido, Daniel	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/58559	
 OAPEN	Handlungsoptionen auf dem Weg in die Gigabit-Gesellschaft: Eine rechtliche Analyse von Konzessions- und Kooperationsmodellen sowie regulatorischer Entflechtungsbestimmungen	Toros, Fabian	de	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode)	https://library.oapen.org/handle/20.500.12657/50285	
 OAPEN	Harpe og sverd: Litteraturhistoriske essay om den norske balladen	Solberg, Olav	no	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/50341	
 OAPEN	Högskolans ansvar: Principer för utveckling av den högre	Casson, Andrew	sv	CC BY 3.0 (https://creativecommons.org/licenses/by/3.0/)	http://library.oapen.org/handle/20.500.12657/33044	Citation requested in the work: Casson, A 2015 Högskolans ansvar: Principer för utveckling av den högre utbildningen. London: Ubiquity Press. DOI: http://dx.doi.org/10.5334/bap. License: CC-BY 3.0
-OAPEN	Il Fantasma dell’Io. La massa e l’inconscio mimetico: The Phantom of the Ego: Modernism and the Mimetic Unconscious	Lawtoo, Nidesh	it	CC BY (OAPEN per-file license code CC-BY)	http://library.oapen.org/handle/20.500.12657/25157	
+OAPEN	Il Fantasma dell’Io. La massa e l’inconscio mimetico: The Phantom of the Ego: Modernism and the Mimetic Unconscious	Lawtoo, Nidesh	it	CC BY (version not recorded)	http://library.oapen.org/handle/20.500.12657/25157	
 OAPEN	Il video a 360° nella didattica universitaria: Modelli ed esperienze	Ranieri, Maria; Luzzi, Damiana; Cuomo, Stefano	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60444	
 OAPEN	Im Brennpunkt der Wirtschaftspolitik: Innovation, Globalisierung und Klimawandel	Keuschnigg, Christian	de	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/90933	
-OAPEN	Immaginare l’altrove nell’epoca dell’Antropocene: Media, confini e cambiamenti climatici	CAPPI, VALENTINA	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/61650	
+OAPEN	Immaginare l’altrove nell’epoca dell’Antropocene: Media, confini e cambiamenti climatici	CAPPI, VALENTINA	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/61650	
 OAPEN	Inklusionsorientierte Schulentwicklung: Interdisziplinäre Rückblicke, Einblicke und Ausblicke	Frohn, Julia; Bengel, Angelika; Piezunka, Anne; Simon, Toni; Dietze, Torsten	de	CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)	https://library.oapen.org/handle/20.500.12657/60549	
 OAPEN	Innvielse til læreryrket: En analyse av praksislæreres veiledningssamtaler	Reier Jensen, Andreas	no	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/24989	
 OAPEN	Interessekonflikter i forskning	Ingierd, Helene; Bay-Larsen, Ingrid; Hiis Hauge, Kjellrun	no	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/25321	
@@ -1088,7 +1090,7 @@ OAPEN	La traiettoria storica dell’Etiopia di Meles Zenawi: Fra democrazia rivo
 OAPEN	La trama dell’allegoria: Scritture di ricerca e istanza allegorica nel secondo Novecento italiano	Caporiccio, Elisa	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/58406	
 OAPEN	La trichera letrada. Intelectuales latinoamericanos y Guerra Fría	Alburquerque, Germán	es	CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/)	http://library.oapen.org/handle/20.500.12657/32089	
 OAPEN	Les normes de prononciation du français: Une étude perceptive panfrancophone	Chalier, Marc	fr	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/51444	
-OAPEN	Letras na América Portuguesa: Autores – Textos – Leitores	Rodrigues-Moura, Enrique	pt	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/90323	
+OAPEN	Letras na América Portuguesa: Autores – Textos – Leitores	Rodrigues-Moura, Enrique	pt	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/90323	
 OAPEN	Lo sguardo territorialista di Leonardo: Il cartografo, l’ingegnere idraulico, il progettista di città e territori	Poli, Daniela	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/62874	
 OAPEN	L’intervista immaginata: Da genere mediatico a invenzione letteraria	GALLERANI, Guido Mattia	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/58403	
 OAPEN	L’URSS dentro e fuori: La narrazione italiana del mondo sovietico	Traini, Cheti	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60443	
@@ -1102,7 +1104,7 @@ OAPEN	Paradigmas y polifuncionalidad: Estudio diacrónico de «preciso»/«preci
 OAPEN	Poéticas espectatoriales en Hispanoamérica y Brasil (1800–1847): Ilustración – emancipación – convivencias excluyentes	Fernández, Hans	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/59652	
 OAPEN	Problemáticas étnicas y sociales desde el pensamiento latinoamericano: Temas, Conceptos, Enfoques	Kozel, Andrés; Rawicz, Daniela; Devés, Eduardo	es	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/63199	
 OAPEN	PROGETTO STREAMING - STRategiE di mitigazione e gestione dei rischi AMbientalI: casi di studio Nel territorio reGionale Toscano: Azioni locali di sostenibilità: cinque progetti per il futuro del territorio toscano	Bartalucci, Chiara; Fagioli, Federico; Giachetti, Andrea; NICCOLAI, ALBERTO; Verdi, Leonardo	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/58409	
-OAPEN	Pubblicità, educazione e diritto in Kant	Perni, Romina	it	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/62877	
+OAPEN	Pubblicità, educazione e diritto in Kant	Perni, Romina	it	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/62877	
 OAPEN	Raccontare la Resistenza a scuola: Esperienze e riflessioni	Bravi, Luca; Martinelli, Chiara; Oliviero, Stefano	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60445	
 OAPEN	Roher Diamant Dalmatien: Die habsburgische Verwaltung, ihre Probleme und das Land, wie beschrieben von seinem Gouverneur Lilienberg fuer Kaiser Franz I. (1834)	Clewing, Konrad	de	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/legalcode)	http://library.oapen.org/handle/20.500.12657/26672	Citation requested in the work: Konrad Clewing (Hg.), Roher Diamant Dalmatien. Die habsburgische Verwaltung, ihre Probleme und das Land, wie beschrieben von seinem Gouverneur Lilienberg für Kaiser Franz I. (1834). München, Berlin, Leipzig, Washington/D.C. 2015
 OAPEN	Samarbeid om selvhjelp: En antologi om den nye selvhjelpsbevegelsen i Norge	Gotaas, Nora; Hatleskog Zeiner, Hilde	no	CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/)	http://library.oapen.org/handle/20.500.12657/24940	
@@ -1114,7 +1116,7 @@ OAPEN	Urbane Transformation durch soziale Innovation: Schlüsselbegriffe und Per
 OAPEN	Wissenschaftskarrieren und Gender Bias: Chancengerechtigkeit an Hochschulen zwischen formellen Vorgaben und informellen Einflüssen	Dahmen-Adkins, Jennifer; Wolffram, Andrea	de	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/90818	
 OAPEN	«Parlare di tutto». Un’idea della critica: Il carteggio Baldacci-Fortini	Baldacci, Luigi; Fortini, Franco	it	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/62858	
 OAPEN	Å kjøpe for Norge	Langseth, Marius; Similä, Jan Ole	no	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/49452	
-OAPEN	Коммуникативный анализ нехудожественного текста для студентов-магистрантов РКИ	Perotto, Monica	ru	CC BY (OAPEN per-file license code CC-BY)	https://library.oapen.org/handle/20.500.12657/89272	
+OAPEN	Коммуникативный анализ нехудожественного текста для студентов-магистрантов РКИ	Perotto, Monica	ru	CC BY (version not recorded)	https://library.oapen.org/handle/20.500.12657/89272	
 OAPEN	Конструкции с опорным глаголом в русском и итальянском языках / Support Verb Constructions. A Russian-Italian Contrastive Analysis	MAIKO, TATSIANA	ru	CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)	https://library.oapen.org/handle/20.500.12657/60450	
 OpenStax	Algebra and Trigonometry 2e	OpenStax	en	https://creativecommons.org/licenses/by/4.0/	https://github.com/openstax/osbooks-college-algebra-bundle/blob/4922e46ebc04326979e19392ccc2384cb8b9076c/collections/algebra-and-trigonometry-2e.collection.xml	Attribution requested in the work: OpenStax and its content contributors.
 OpenStax	American Government 4e	OpenStax	en	https://creativecommons.org/licenses/by/4.0/	https://github.com/openstax/osbooks-american-government/blob/5c90dd7907dbf25f42266417e63ca3bb0012f031/collections/american-government-4e.collection.xml	Attribution requested in the work: OpenStax and its content contributors.

EVALUATION.md

@@ -1,8 +1,9 @@
 # Source-1 evaluation
 
 Source-1 was compared with its teacher and 16 public quality scorers on three test sets. An independent proprietary
-LLM grader scored every chunk with Source-1's 13-field rubric. Each number says how closely a model's ranking agrees
-with the grader's, so it measures agreement with Source-1's own rubric, on home ground.
+LLM grader scored every chunk with Source-1's 13-field rubric: its grades were never trained on, and on the held-out
+set it was given the same instructions as Source-1's teacher. Each number says how closely a model's ranking agrees
+with the grader's, so it measures agreement with this grader applying Source-1's own rubric, on home ground.
 
 ## Summary
 
@@ -38,7 +39,10 @@ were never trained on.
 
 **Home ground.** The held-out documents come from the same kinds of sources as the training data. The public scorers
 were built for their own definitions of quality, most for educational value, and are not wrong when they disagree with
-this rubric.
+this rubric. Even a scorer that matched the grader's own `educational_value` scores exactly would reach only 0.874 on
+the held-out set, 0.900 on the English exam and 0.817 on the 12-language exam, so a small part of the gap to the
+educational-value classifiers comes from what this measure asks for;
+[Educational value alone](#educational-value-alone) is the fairer comparison for them.
 
 **The three test sets.**
 
@@ -46,7 +50,7 @@ this rubric.
 |---|---|---|---|---|
 | held-out set (main result) | 495, one per document | 53 | 64 | documents from Source-1's held-out test split, never trained or calibrated on |
 | English exam | 414, from 332 documents (413 scored by the teacher) | English | 25 | 200 chunks drawn at random, plus 214 harder cases added on purpose |
-| 12-language exam | 352, one per document | 12 | 46 | web text drawn at random from FineWeb-2, about 30 chunks per language |
+| 12-language exam | 352, one per document | 12 | 46 | web text drawn from FineWeb-2's test split (blocks of 10 consecutive rows at random positions; Spanish from its first rows), about 30 chunks per language |
 
 **95% intervals.** Each difference between two models comes with a 95% interval from a paired bootstrap over
 documents. When the interval excludes zero, chance alone is an unlikely explanation for the difference.
@@ -67,6 +71,8 @@ grammar its card recommends; the serving engine mainly affects speed, so we clai
   publishes one.
 - The encoder classifiers ran in Hugging Face transformers as their model cards show, in the precision each card
   states (bfloat16 where it says so, float32 otherwise).
+- Source-1 itself ran in bfloat16, its default. In float32 its rank agreement on the held-out set is unchanged to
+  three decimal places.
 - The two fastText models ran in the fasttext library. They read the whole text with newlines turned into spaces, as
   the DCLM code does.
 - EAI-Distill ran in transformers in float32 with greedy decoding (its repository's default settings sample).
@@ -132,10 +138,12 @@ What grades from the grader's model family did and did not influence:
 - **Held-out set** (the main result): 495 chunks in 53 languages, one per document. All come from Source-1's held-out
   test split, drawn from the four data stages (web 129, multilingual web 276, conversations/code/synthetic 48, open
   books 42). The grader drops 64 of them. 159 chunks are English and 29 Chinese; most other languages have 1 to 12
-  chunks. Source-1 never trained or calibrated on these documents. They come from the same kinds of sources as the
-  training data. 42 of the 495 chunks (8.5%) come from sources that were later removed from training under the
-  license rules (19 from DCLM-baseline, 16 raw Common Crawl pages, and 7 from collections with unreliable or gated
-  license terms). The test split also keeps documents that the license filtering removed from training.
+  chunks. Chinese was oversampled on purpose: 15 of its 29 chunks were added to the proportional sample, and they hold
+  7 of the 64 grader drops. 26 of the 53 languages have 5 or fewer chunks. Source-1 never trained or calibrated on
+  these documents. They come from the same kinds of sources as the training data. 42 of the 495 chunks (8.5%) come
+  from sources that were later removed from training under the license rules (19 from DCLM-baseline, 16 raw Common
+  Crawl pages, and 7 from collections with unreliable or gated license terms). The test split also keeps documents
+  that the license filtering removed from training.
 - **English exam**: 414 chunks from 332 whole documents split into chunks. 200 chunks from 196 documents were drawn at
   random from web, wiki, Common Pile, math and code sources (the **random-sample** chunks and documents). 214 chunks
   from 136 documents were added on purpose to cover harder cases (long documents 57 chunks, academic 50, math 28,
@@ -145,12 +153,14 @@ What grades from the grader's model family did and did not influence:
   one of the 414 chunks (a random-sample chunk). So the comparisons with the teacher and the public scorers' English
   rows use the 413 chunks it scored (411 or 412 where a public scorer also lacks one). The 512-token comparison uses
   all 414.
-- **12-language exam**: 352 chunks, one per document, all drawn at random from FineWeb-2 web text in 12 languages
-  (ar, bn, de, es, hi, ja, ko, ru, sw, th, vi, zh; about 30 each). It has no code or math: 351 of the 352 chunks are
-  plain text by the grader's label. The grader drops 46.
+- **12-language exam**: 352 chunks, one per document, all FineWeb-2 web text in 12 languages (ar, bn, de, es, hi,
+  ja, ko, ru, sw, th, vi, zh; about 30 each), drawn from FineWeb-2's test split (blocks of 10 consecutive rows at
+  random positions; Spanish from its first rows). It has no code or math: 351 of the 352 chunks are plain text by the
+  grader's label. The grader drops 46.
 
 Every exam document was checked against the training data (exact text, URL, title, long-line and shingle matching).
-No copies were found.
+None has half or more of its text in a training document; the largest share of an exam text found in a training
+document is 41%.
 
 #### Near-duplicate check
 
@@ -162,9 +172,9 @@ each, all short templated texts. 9 in all are covered 20% or more. These chunks
 
 #### Safety filter
 
-A safety filter fixed before any result was seen removed a small number of documents from every evaluation set and
-from Source-1's training, validation and test data. Documents that substantially copy removed text were removed from
-its data as well. Every model is compared on the same filtered chunks.
+A safety filter, with a rule fixed before it was first run, removed a small number of documents from every evaluation
+set and from Source-1's training, validation and test data. Documents that substantially copy removed text were
+removed from its data as well. Every model is compared on the same filtered chunks.
 
 </details>
 
@@ -178,6 +188,10 @@ This file calls either "the grader". That model family also includes one of the
 rubric text. The exams were graded with an older revision of the rubric, from before the rubric settled how to score
 ads.
 
+The held-out grader was given the teacher's own instructions word for word: the 13-field rubric and the special rules
+in [Appendix A](#appendix-a-rubric-anchors), for example that ads are scored `spam_seo` 3 and kept. The public scorers
+follow none of these rules.
+
 The grader returns the 13 fields only. Its reference overall score and keep flag are computed from its scores in the
 same way as Source-1's: the overall formula of the rubric (see [README.md](README.md#what-you-get)), and keep =
 false when the rubric's hard filters match (`toxicity >= 4`, `spam_seo >= 4` or `boilerplate >= 4.5`). "The grader's
@@ -188,10 +202,9 @@ under [How the public scorers were run](#how-the-public-scorers-were-run).
 
 #### Grader self-agreement
 
-The exam grader also graded part of each exam a second time. The two gradings agree at rank 0.904 on 97 English
-random-sample documents and 0.945 on 58 documents of the 12-language exam. This is a rough noise ceiling: it shows how
-much one grading moves when repeated. It is not a hard upper bound, because a model can agree with one grading more
-closely than two noisy gradings agree with each other.
+The exam grader also graded part of each exam a second time. The two gradings agree at rank 0.902 on 97 English
+random-sample chunks and 0.945 on 58 chunks of the 12-language exam. On exactly these chunks Source-1 reaches 0.886
+and 0.892 and the teacher 0.879 and 0.882, below the grader's agreement with itself (within noise for English).
 
 </details>
 
@@ -252,9 +265,10 @@ the grader keeps ([Metric definitions](#metric-definitions)).
 | propella-1 1.7B | 495 | 0.736 | 0.855 | 0.896 | 0.599 (about 38/64) |
 | JQL-Edu (mean of 3 balanced heads) | 495 | 0.600 | 0.737 | 0.826 | 0.328 (21/64) |
 
-- At this matched rate Source-1 catches 46 of the grader's 64 drops and the teacher 44, a difference within noise. At
-  each model's own drop line (for the teacher, its own keep flags) they catch 43 and 46
-  ([The drop line](#the-drop-line)). Source-1 and the teacher are also level within noise on rank and AUC.
+- At this matched rate Source-1 catches 46 of the grader's 64 drops and the teacher 44, a difference within noise
+  (without the 15 added Chinese chunks: about 39 and 39 of 57). At each model's own drop line (for the teacher, its
+  own keep flags) they catch 43 and 46 ([The drop line](#the-drop-line)). Source-1 and the teacher are also level
+  within noise on rank and AUC.
 - The lead over propella-1 4B, the closest public scorer, is at least as large outside English: 0.916 against 0.765
   on the 335 non-English chunks.
 
@@ -530,7 +544,7 @@ At the shipped drop line, counted per document (the teacher at its own keep flag
 |---|---|---|---|---|---|
 | English, all 332 | 24 | 14 (0.583) | 96.7% | 14 (0.583) | 96.1% |
 | English, the 196 random-sample documents | 13 | 9 (0.692) | 97.4% | 6 (0.462) | 94.9% |
-| 12 languages, all 352 (all random) | 46 | 31 (0.674) | 94.3% | 28 (0.609) | 93.5% |
+| 12 languages, all 352 (none added on purpose) | 46 | 31 (0.674) | 94.3% | 28 (0.609) | 93.5% |
 
 The exams were graded with an older revision of the rubric, before it settled how to score ads (`spam_seo` 3, kept).
 So part of the gap is rubric drift that affects the teacher and Source-1 alike.
@@ -707,7 +721,8 @@ the public scorers or the teacher was made under the same conditions, so none is
 Precision and batching: computing in bfloat16, both weight files give the same scores. On these 1,261 chunks, scoring
 each chunk alone instead of in the default batches moved `overall` by up to 0.04 and a single field by up to 0.10 (3
 labels changed, no keep decision). float32 differs from bfloat16 by a similar amount (up to 0.03 on `overall` and 0.09
-on a single field; 4 labels and 1 keep decision changed).
+on a single field; 4 labels and 1 keep decision changed); computing in float32 with the default bfloat16 weights moved
+`overall` by up to 0.05. In float32, scores do not depend on the batch.
 
 </details>
 
@@ -716,7 +731,11 @@ on a single field; 4 labels and 1 keep decision changed).
 - **Home ground.** It measures agreement with Source-1's rubric, as applied by graders from one proprietary model
   family whose grades also steered development. It does not show how Source-1 does on text from other sources, or
   that filtering with it trains better language models.
+- **The numbers are agreement with one grader family, not accuracy.** Another grader applying the same rubric would
+  give different values.
 - **The exams helped choose the teacher**, so they are not independent of the grader.
+- **Agreement is lower among good texts.** Among the chunks the grader keeps, Source-1's rank agreement is 0.87, and in
+  the better half of those 0.73.
 - **It misses about a third of the grader's drops** at the shipped line: it catches 43 of 64 on the held-out set, the
   teacher, at its own keep flags, 46.
 - **Like its teacher, it rates some qualities higher than the grader does.** On the English exam, `educational_value`
@@ -739,7 +758,9 @@ on a single field; 4 labels and 1 keep decision changed).
 <summary>The full text of each limitation</summary>
 
 - **Home-ground evaluation.** The benchmark measures agreement with Source-1's rubric as applied by independent LLM
-  graders from one proprietary model family (one graded the exams, another the held-out set). Grades from that family,
+  graders from one proprietary model family (one graded the exams, another the held-out set). They are independent in
+  that their grades were never trained on; the held-out grader was given the teacher's own instructions word for word
+  ([The grader in detail](#the-grader-in-detail)). Grades from that family,
   among other proprietary LLM graders, also steered the rubric revisions, the choice of the teacher and its prompt
   setup, and the drop-line candidates and floor ([Independence from development](#independence-from-development)).
   The held-out documents come from the same kinds of sources as the training data. The public scorers were built for
@@ -747,8 +768,15 @@ on a single field; 4 labels and 1 keep decision changed).
   more generous reading of propella-1 halves its gap to Source-1 ([How propella-1 is read](#how-propella-1-is-read)).
   The comparison does not show how Source-1 does on text from other sources, or that filtering with Source-1 trains
   better language models; neither has been tested.
+- **The numbers are agreement with one grader family, not accuracy.** Another grader applying the same rubric would
+  give different values, and differences of a few hundredths near the top (Source-1 against its teacher) may reflect
+  this grader's own habits.
 - **The exams are not independent of the grader.** They are the samples on which the teacher and its prompt setup were
   chosen against the exam grader's labels. The held-out set, which played no part in that choice, is the main result.
+- **Agreement is lower among good texts.** About 13% of the held-out chunks are spam, boilerplate or toxic, which are
+  easy to tell apart. Among the chunks the grader keeps, Source-1's rank agreement is 0.87, and in the better half of
+  those 0.73 (teacher 0.75, propella-1 4B 0.61). Every scorer drops like this on already-filtered text; if you rank
+  filtered text, expect the lower figure.
 - **It misses about a third of the chunks the grader drops at the shipped line.** On the held-out set the shipped line
   catches 67.2% of the grader's drops (43/64; interval 55.0% to 77.4%); the teacher catches 71.9% (46/64). 17 of
   Source-1's 21 misses are also missed by the teacher, so most of what Source-1 misses its teacher misses too. Most
@@ -842,8 +870,9 @@ cannot be recomputed from this repository alone.
 | code_quality | Not usable code: garbled, minified, obfuscated | Very poor: likely non-functional fragments, no structure, or auto-generated boilerplate | Poor: may work but messy | Acceptable: readable, plausibly correct, minimal docs | Good: clean, idiomatic, documented | Excellent: exemplary, production quality, instructive |
 | math_quality | Garbled math | Mostly wrong or incoherent | Some correct math, but errors or skipped steps | Generally correct, key steps shown | Correct, clean notation, complete steps | Rigorous and elegant, every step justified |
 
-The rubric anchors and the head layout are in `source1.json`. The teacher's prompt also had a few special rules that
-are not in `source1.json`, and Source-1 was trained on labels that follow them, as far as the teacher did:
+The rubric anchors and the head layout are in `source1.json`. The teacher's prompt, which the held-out grader also
+received word for word, had a few special rules that are not in `source1.json`. Source-1 was trained on labels that
+follow them, as far as the teacher did:
 
 - Pages whose main purpose is to promote or sell a business, product or service (company "about us" pages, product
   and landing pages, shop listings, brochures) are ads: format `product_page` and `spam_seo` 3, even when cleanly
@@ -931,8 +960,9 @@ filtering removed, for evaluation only):
 - Separately, a pre-specified safety filter removed documents from every split, and a rule fixed in advance also
   removed documents that substantially copy text the safety filter removed (every split).
 - Book pages that are mostly a table of contents were left out of training (99 chunks).
-- Every exam document was checked against the training data (exact text, URL, title, long-line and shingle matching);
-  no copies were found.
+- Every exam document was checked against the training data (exact text, URL, title, long-line and shingle matching).
+  None has half or more of its text in a training document; the largest share of an exam text found in a training
+  document is 41%.
 
 #### Labels
 

NOTICE

@@ -63,10 +63,11 @@ are not part of this release.
 
 Source-1 was evaluated with an independent proprietary LLM grader, used for evaluation
 only: the grader's outputs were never used as training targets or training data, and
-the shipped drop line was chosen by a fixed rule on the teacher's labels. Grades and
-reviews by proprietary LLMs from the grader's model family did inform some design
-choices (the rubric revision, the choice of teacher and its prompt on the exam sets,
-the candidate drop lines and which collections were filtered out); see the
+the shipped drop line was chosen by a fixed rule on the teacher's labels. On the
+held-out set the grader was given the same instructions as the teacher. Grades and
+reviews by proprietary LLMs, among them the grader's model family, did inform some
+design choices (the rubric revision, the choice of teacher and its prompt on the exam
+sets, the candidate drop lines and which collections were filtered out); see the
 development note in README.md and "Independence from development" in EVALUATION.md.
 
 
@@ -100,13 +101,16 @@ states no open licence; web pages under NC or ND terms, and pages on sites that
 other people's documents; and code whose licence or file header is copyleft,
 proprietary or not on a permissive allow-list.
 
-Every book that is not public domain or CC0 is also credited individually in the file
-CREDITS_BOOKS.tsv, which is part of this notice: title, authors, language, licence and
-source URL. Where a work's own text asks to be cited or attributed in a particular way,
-the file gives that citation or attribution: FAO's "Required citation", OpenStax's
-attribution request, the citations requested by Eurydice, JRC and other EU reports, and
-those of some other books and reports (university presses, Frontiers ebooks, research and
-project reports). The World Bank works are credited in Appendix A.
+Every book in the open-books part of the training data that is not public domain or
+CC0 is also credited individually in the file CREDITS_BOOKS.tsv, which is part of this
+notice: title, authors, language, licence and source URL. Where a work's own text asks
+to be cited or attributed in a particular way, the file gives that citation or
+attribution: FAO's "Required citation", OpenStax's attribution request, the citations
+requested by Eurydice, JRC and other EU reports, and those of some other books and
+reports (university presses, Frontiers ebooks, research and project reports). The World
+Bank works are credited in Appendix A. Books and book chapters that came through
+Common Pile v0.1 (DOAB, Pressbooks, LibreTexts, OER Commons) are credited at collection
+level in 3.5.
 
 License links used below:
   ODC-By 1.0       https://opendatacommons.org/licenses/by/1-0/
@@ -372,9 +376,12 @@ licence were removed before training.
 4. CONTACT AND REMOVAL REQUESTS
 ================================================================================
 
-Questions about these credits, corrections, and removal or opt-out requests:
-the Community tab of the model repository,
-https://huggingface.co/msmth/Source-1/discussions.
+Questions about these credits, corrections, and removal or opt-out requests: open a
+discussion in the Community tab of the model repository,
+https://huggingface.co/msmth/Source-1/discussions. If your request involves personal
+information, open a discussion without the details and we will arrange a private way to
+reach us. We review every request and, where the content is in our training data,
+exclude it from future versions; published weights cannot be changed.
 
 
 ================================================================================

README.md

@@ -146,16 +146,19 @@ languages, up to 8,192 tokens at a time. It has 307M parameters and is free to u
 ## Highlights
 
 The numbers are rank agreement with an independent proprietary LLM grader: how closely a model puts texts in the same
-order as the grader does (1.0 means the same order, 0 means no link).
-
-- **Beats all 16 public quality scorers we tested, including FineWeb-Edu and propella-1, on all three test sets.** On the main test set (495 held-out
-  texts in 53 languages): 0.90 vs 0.76 for the best of them, propella-1 4B, a model 13x larger.
-- **Nearly matches its teacher**, the open-weight 27B LLM that labeled its training data, at 1/88 the size: 0.90 vs 0.91
-  on the main test set.
+order as the grader does (1.0 means the same order, 0 means no link). The grader's grades were never trained on, and on
+the main test set it was given the same instructions as Source-1's teacher.
+
+- **Beats each of the 16 public quality scorers we tested, including FineWeb-Edu and propella-1, on every test set it
+  was run on** (the 11 English-only scorers were not run on the 12-language exam). On the main test set (495 held-out
+  texts in 53 languages): 0.90 vs 0.76 for the best of them, propella-1 4B, a model with 13x more parameters.
+- **Nearly matches its teacher**, the open-weight 27B LLM that labeled its training data, with 1/88 of its parameters:
+  0.90 vs 0.91 on the main test set.
 - **Also ahead when judged on educational value alone**, the thing most public scorers were built for.
 
 *Caveat: the grader scored with Source-1's own rubric (its scoring guide), so this test plays to Source-1's
-strengths. See [Limitations](#limitations) and [EVALUATION.md](EVALUATION.md).*
+strengths. Even a scorer that matched the grader's educational-value scores exactly would reach only 0.87 here. See
+[Limitations](#limitations) and [EVALUATION.md](EVALUATION.md).*
 
 ## Quick start
 
@@ -178,6 +181,16 @@ doc = model.score("Photosynthesis is how plants turn light, water and carbon dio
 print(doc["overall"], doc["keep"])   # a 0-5 score and the keep/drop decision
 ```
 
+### For AI agents and scripts
+
+- **Load it only through `source1.py`.** `pipeline("text-classification", ...)` and `AutoModel...` classes load the
+  backbone without Source-1's 13 trained heads and return meaningless `LABEL_0` / `LABEL_1` scores.
+- **Check the setup** with the `examples/sample.jsonl` command above: its output must equal
+  `examples/expected_output.jsonl`.
+- **Output:** one JSON object per document (the 13 fields, `overall`, `keep`, `drop_reasons`). **Exit codes:** 0 done;
+  2 bad arguments or an unreadable input, before the model loads; 1 a bad record during a run.
+- No `trust_remote_code` and no prompts. Loading from a local folder makes no network calls.
+
 <details>
 <summary>More usage: many texts, precision, options, speed, files</summary>
 
@@ -206,16 +219,22 @@ python source1.py --model . --input page.txt --device cpu --precision fp32 --dty
 ```
 
 - **Output.** One flat dict per document with the 13 fields, `overall`, `keep` and `drop_reasons`. It also gives the
-  length in tokens, the number of chunks (`parts`), whether a chunk was cut, and per-chunk results (`chunks`).
+  length in tokens, the number of chunks (`parts`), whether a chunk was cut, and per-chunk results (`chunks`). Empty
+  text (or only spaces, zero-width or control characters) gives `keep` false, `drop_reasons` `["empty text"]` and
+  `null` for `overall` and every field, so leave those records out before sorting by `overall`.
 - **Precision.** By default it loads `model.safetensors` (bfloat16, 0.6 GB). For the full float32 weights (1.2 GB),
   remove `--exclude` from the download and pass `precision="fp32"`. It computes in bfloat16 on NVIDIA Ampere or newer
   GPUs and in float32 elsewhere. float16 is not supported. In bfloat16, scores can shift by up to about 0.04
   depending on which texts share a batch; pass `dtype="fp32"` or `batch_tokens=1` to avoid that.
 - **Options.** `drop_line` sets your own drop rule, such as `"toxicity >= 4 or spam_seo >= 3 or boilerplate >= 4.5"`.
   `max_chunks` scores only some chunks of very long documents, for speed. `revision` pins a Hub version. The command
-  line reads JSONL, JSON or plain text, and `python source1.py --help` lists every flag.
+  line reads JSONL, JSON or plain text (also gzip, bzip2 or xz compressed), and `python source1.py --help` lists
+  every flag.
 - **Examples and speed.** `examples/` holds six documents and their expected CPU output. A GPU can differ by a few
   hundredths. Source-1 scores about 30 chunks (50,000 tokens) per second on one RTX 3090 in bfloat16.
+- **On a CPU.** By default it scores 16,384 tokens per batch there and uses about 3 GB of RAM. On 4 threads it reads
+  about 900 tokens per second on typical chunks and about 600 on full-length ones (about 12 seconds per
+  7,000-token chunk). Use `--max-chunks` for long documents.
 
 **Files**
 
@@ -229,12 +248,12 @@ python source1.py --model . --input page.txt --device cpu --precision fp32 --dty
 | `calibration.json` | the drop line and the quality-score offsets, with how they were chosen |
 | `tokenizer.json`, `tokenizer_config.json` | mmBERT's tokenizer (same vocabulary and merges, re-saved) |
 | `source1.py` | standalone loader, Python API and command line |
-| `requirements.txt` | `torch`, `transformers`, `safetensors`, `tokenizers` |
+| `requirements.txt` | `torch`, `transformers`, `safetensors`, `tokenizers`, `huggingface_hub` |
 | `examples/` | six sample inputs (`sample.jsonl`) and their expected command-line output (`expected_output.jsonl`) |
 | `EVALUATION.md` | the full evaluation, the rubric anchors and the training details |
 | `images/` | the benchmark chart above |
 | `LICENSE`, `NOTICE`, `AUTHORS` | license text, third-party notices and credits, authors |
-| `CREDITS_BOOKS.tsv` | per-work credits for the training books that are not public domain (part of `NOTICE`) |
+| `CREDITS_BOOKS.tsv` | per-work credits for the open-books part of the training data, for books that are not public domain or CC0 (part of `NOTICE`) |
 
 </details>
 
@@ -260,9 +279,11 @@ Source-1 returns these 13 fields. Higher is better for quality scores and worse
 scores are `null` when they do not apply. You also get:
 
 - `overall`: one 0-5 score, the quality scores minus penalties for red flags.
-- `keep`: false (drop) if `toxicity >= 4` or `spam_seo >= 3.5` or `boilerplate >= 4.5`.
+- `keep`: false (drop) if `toxicity >= 4` or `spam_seo >= 3.5` or `boilerplate >= 4.5`. It checks only these red
+  flags, so also set a threshold on `overall`.
 
-Long documents are split into chunks. Each chunk is scored, and the results are combined into one.
+Long documents are split into chunks. Each chunk is scored, the results are combined into one, and `keep` is decided
+on the combined scores (each chunk's own result is in `chunks`).
 
 <details>
 <summary>All fields in detail, the scoring formula, the drop line and long documents</summary>
@@ -277,8 +298,10 @@ Long documents are split into chunks. Each chunk is scored, and the results are
 **Scores.** Each 0-5 score is a decimal such as 2.73: the model estimates how likely each level from 0 to 5 is, and
 the score is the average level weighted by those chances. `code_quality` applies when `content_type` is
 text_with_code or code_only, or `format` is code_file. `math_quality` applies when `content_type` is math_heavy or
-`topic` is math. Otherwise they are `null`. What each level means is in `source1.json` and
-[EVALUATION.md Appendix A](EVALUATION.md#appendix-a-rubric-anchors), with the extra rules the teacher was given.
+`topic` is math. Otherwise they are `null`. For a split document they average the chunks where they apply, so they can
+be set even when the document's `content_type` is plain_text. What each level means is in `source1.json` and
+[EVALUATION.md Appendix A](EVALUATION.md#appendix-a-rubric-anchors), with the extra rules that the teacher and the
+held-out grader were given.
 
 **Formula.**
 
@@ -299,10 +322,17 @@ fixed rule on the teacher's labels. Stricter lines and what they cost are in
 [EVALUATION.md](EVALUATION.md#the-drop-line). You do not have to use `keep`: ranking by `overall`, or your own rules
 on single fields, may work better for you.
 
+`keep` applies only this red-flag line, so very short or degenerate text (a single word, an emoji, one letter
+repeated) can still be kept. Combine it with a threshold on `overall`. A drop line can also use `overall`, `tokens`
+and `parts` (the last two for whole documents only), for example
+`"toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5 or overall < 1"`.
+
 **Long documents.** A document longer than about 7,800 tokens is split into balanced chunks at natural breaks. Each
 chunk is scored with a one-line header, as in training (source type, title if given, "Part i of n"). The document gets
 the label that covers the most tokens and the token-weighted average of each score. `toxicity` takes the maximum, and
-the code and math scores average only the chunks where they apply. `overall` and `keep` are then recomputed.
+the code and math scores average only the chunks where they apply. `overall` and `keep` are then recomputed from these
+combined scores, so a document can be kept even when some of its chunks would be dropped. Each chunk's own scores and
+`keep` are in `chunks`.
 
 </details>
 
@@ -398,8 +428,10 @@ Data sources, filtering, recipe and development details:
 
 ## Contact
 
-Questions, corrections and removal requests: the [Community tab](https://huggingface.co/msmth/Source-1/discussions).
-Send the URL or dataset of your content and we will remove it from the next release's training data.
+Questions, corrections and removal requests: open a discussion in the
+[Community tab](https://huggingface.co/msmth/Source-1/discussions). If your request involves personal information, open
+a discussion without the details and we will arrange a private way to reach us. We review every request and, where the
+content is in our training data, exclude it from future versions; published weights cannot be changed.
 
 ## Citation
 

requirements.txt

@@ -1,8 +1,11 @@
 # source1.py needs Python >= 3.10 and these packages. The minimum versions are the oldest this release was tested
 # with (torch 2.11 and 2.14, transformers 5.17), with both weight files: model.safetensors (bfloat16, the default)
-# and model.fp32.safetensors (float32). Older versions may work but were not tested. No GPU is needed: on a CPU, or a
-# GPU without native bfloat16, the bfloat16 weights are upcast to float32 when loaded.
-torch>=2.11
-transformers>=5.17
+# and model.fp32.safetensors (float32). Older versions may work but were not tested, and the next major versions
+# are left out until they are. huggingface_hub downloads the model when it is loaded by its Hugging Face repo id.
+# No GPU is needed: on a CPU, or a GPU without native bfloat16, the bfloat16 weights are upcast to float32 when
+# loaded.
+torch>=2.11,<3
+transformers>=5.17,<6
 safetensors>=0.8
 tokenizers>=0.23
+huggingface_hub>=1.0

source1.py

@@ -17,6 +17,7 @@ CPU.
 
     python source1.py --input docs.jsonl --output scores.jsonl     # one JSON object per line, "text" field
     python source1.py --input page.txt                             # a .txt file is one document
+    python source1.py --input docs.jsonl.gz                        # gzip, bzip2 and xz files are decompressed
 
 Output: one flat dict per document
 ----------------------------------
@@ -42,8 +43,8 @@ calibration offsets of calibration.json to the quality scores (about 0.02 at mos
 
 How a document is scored (the same steps the model was trained and evaluated with)
 ------------------------------------------------------------------------------------
-1. ``clean_text``: line endings to "\\n", control characters dropped, Unicode NFC, trailing spaces dropped, at most
-   two blank lines in a row.
+1. ``clean_text``: line endings to "\\n", ASCII control characters other than tab and newline dropped, Unicode NFC,
+   trailing spaces dropped, at most two blank lines in a row.
 2. ``split_text``: a document longer than 7,808 tokens is split into balanced chunks, each ending at the most
    natural boundary near its ideal end (headings, then paragraphs, lines, sentences, spaces; definitions in code).
 3. ``build_input``: each chunk gets a one-line header, a blank line, then the chunk text:
@@ -51,7 +52,7 @@ How a document is scored (the same steps the model was trained and evaluated wit
    Every training input had a Source line, almost always "dataset record", so that is the default; code files had
    "<language> source file" (pass ``code_language="Python"``). Title is added when given, Part when the document has
    more than one chunk. ``url`` is accepted but not shown to the model unless ``show_url=True``: no training input had
-   one. On 1,261 held-out benchmark chunks (495 graded held-out chunks from the test split from the test split and 766 exam chunks),
+   one. On 1,261 held-out benchmark chunks (495 graded held-out chunks from the test split and 766 exam chunks),
    dropping the whole header moved overall by 0.03 on average (at most 0.655), dropping a title by 0.05 on the chunks
    that had one; adding a URL moved it by up to 0.7 and did not improve its rank agreement with the independent
    graders of the model card's evaluation.
@@ -85,13 +86,17 @@ from __future__ import annotations
 
 import argparse
 import ast
+import bz2
+import gzip
 import json
+import lzma
 import math
 import operator
 import re
 import sys
 import time
 import unicodedata
+import zlib
 from bisect import bisect_left
 from collections.abc import Iterable, Iterator, Mapping
 from pathlib import Path
@@ -105,7 +110,8 @@ __version__ = "1.0.0"
 MAX_LENGTH = 8192  # tokens per model input, <bos> and <eos> included
 CHUNK_TOKENS = 7808  # document tokens per chunk: 8,192 minus 384 kept for the header (as in training)
 DEFAULT_SOURCE = "dataset record"
-DEFAULT_BATCH_TOKENS = 65536  # padded tokens per forward pass
+DEFAULT_BATCH_TOKENS = 65536  # padded tokens per forward pass on a GPU
+DEFAULT_BATCH_TOKENS_CPU = 16384  # on a CPU: a quarter of the memory, and no slower there
 LEVELS = (0, 1, 2, 3, 4, 5)
 # The backbone weights of each precision: bfloat16 (the default) and the full-precision float32 copy. The fp32 file
 # follows transformers' variant naming (model.<variant>.safetensors), so AutoModel loads it with variant="fp32".
@@ -126,7 +132,9 @@ def hub_files(precision: str = DEFAULT_PRECISION) -> tuple[str, ...]:
 
 
 _JUNK = re.compile("[\x00-\x08\x0b\x0e-\x1f\x7f\ud800-\udfff\ufeff\u200b\ufffe\uffff]")
-_TRAILING_WS = re.compile(r"[ \t]+\n")
+# The lookbehind keeps this linear: without it, a long run of spaces that ends in no newline is scanned again from
+# each of its positions, so a page of spaces takes quadratic time. The result is the same.
+_TRAILING_WS = re.compile(r"(?<![ \t])[ \t]+\n")
 _BLANK_LINES = re.compile(r"\n{4,}")
 _SPACE_RUN = re.compile(r"[ \t]{2,}")
 _BLANK_RUN = re.compile(r"\n(?:[ \t]*\n){3,}")
@@ -134,8 +142,9 @@ _BLANK_RUN = re.compile(r"\n(?:[ \t]*\n){3,}")
 
 def clean_text(text: str) -> str:
     """The document as the scorer sees it before chunking: "\\n" line endings (a form feed counts as a paragraph
-    break), control characters, zero-width spaces and byte-order marks dropped, Unicode NFC, no trailing spaces,
-    at most two blank lines in a row, no blank lines at either end."""
+    break), ASCII control characters other than tab and newline, lone surrogates, zero-width spaces and byte-order
+    marks dropped (C1 control characters such as U+0085 are kept), Unicode NFC, no trailing spaces, at most two
+    blank lines in a row, no blank lines at either end."""
     text = text.replace("\r\n", "\n").replace("\r", "\n").replace("\x0c", "\n\n")
     text = _JUNK.sub("", text)
     text = unicodedata.normalize("NFC", text)
@@ -296,25 +305,107 @@ _CMP = {ast.Eq: operator.eq, ast.NotEq: operator.ne, ast.Lt: operator.lt, ast.Lt
         ast.Gt: operator.gt, ast.GtE: operator.ge}
 
 
+class DropLineError(ValueError):
+    """A drop line that is not valid (see ``DropLine``)."""
+
+
 class DropLine:
-    """A drop line such as ``toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5``: field names, numbers,
-    comparisons, ``and`` / ``or`` / ``not`` and parentheses. A comparison with a missing score (None) is false.
-    A chunk or document is kept when the line does not match."""
+    """A drop line such as ``toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5``: comparisons of a score (or
+    ``overall``, ``tokens``, ``parts``) with a number or another score, and of a label with one of its values
+    (``format == 'news'``; ``==`` and ``!=`` only), joined with ``and`` / ``or`` / ``not`` and parentheses; ``True``
+    and ``False`` match always and never. A comparison with a missing score (None) is false, ``!=`` included. A
+    chunk or document is kept when the line does not match.
+
+    ``names`` are the fields the line may use and ``labels`` maps each label field to its values; the other names
+    are numbers. The line is checked when it is created: a bare field name or constant used as a condition, a string
+    that is not a value of its label, a score compared with a string or a label with a number raise DropLineError
+    (a ValueError)."""
 
     _NODES = (ast.Expression, ast.BoolOp, ast.And, ast.Or, ast.UnaryOp, ast.Not, ast.USub, ast.Compare, ast.Name,
               ast.Load, ast.Constant, *_CMP)
 
-    def __init__(self, source: str, names: Iterable[str]):
+    def __init__(self, source: str, names: Iterable[str], labels: Mapping[str, Iterable[str]] | None = None):
         self.source = source.strip()
-        tree = ast.parse(self.source, mode="eval")
+        self.names = set(names)
+        self.labels = {name: list(values) for name, values in (labels or {}).items()}
+        try:
+            tree = ast.parse(self.source, mode="eval")
+        except (SyntaxError, ValueError, RecursionError, MemoryError) as e:
+            raise DropLineError(f"drop line {source!r} is not a valid expression ({e})") from None
         for node in ast.walk(tree):
             if not isinstance(node, self._NODES):
-                raise ValueError(f"{type(node).__name__} is not allowed in a drop line: {source!r}")
-            if isinstance(node, ast.Name) and node.id not in set(names):
-                raise ValueError(f"unknown name {node.id!r} in drop line {source!r}")
+                raise DropLineError(f"{type(node).__name__} is not allowed in a drop line: {source!r}")
+            if isinstance(node, ast.Name) and node.id not in self.names:
+                raise DropLineError(f"unknown name {node.id!r} in drop line {source!r}")
         self.tree = tree.body
+        try:
+            self._condition(self.tree)
+        except RecursionError:
+            raise DropLineError(f"drop line {source!r} is nested too deeply") from None
         top_or = isinstance(self.tree, ast.BoolOp) and isinstance(self.tree.op, ast.Or)
         self.terms = list(self.tree.values) if top_or else [self.tree]
+        # Evaluated once on stand-in scores, and once with every score missing, so that a line that cannot be
+        # evaluated fails here and not in the middle of a run.
+        try:
+            self.reasons({n: self.labels[n][0] if self.labels.get(n) else 0.0 for n in self.names})
+            self.reasons({})
+        except RecursionError:
+            raise DropLineError(f"drop line {source!r} is nested too deeply") from None
+
+    def _fail(self, node: ast.AST, problem: str) -> None:
+        raise DropLineError(f"{ast.unparse(node)!r} {problem}, in drop line {self.source!r}")
+
+    def _condition(self, node: ast.AST) -> None:
+        """Check that ``node`` is a condition: a comparison, True / False, or and / or / not of conditions."""
+        if isinstance(node, ast.BoolOp):
+            for value in node.values:
+                self._condition(value)
+        elif isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not):
+            self._condition(node.operand)
+        elif isinstance(node, ast.Compare):
+            kinds = [self._operand(x) for x in (node.left, *node.comparators)]
+            for op, (a, b) in zip(node.ops, zip(kinds, kinds[1:])):
+                self._check_pair(node, op, a, b)
+        elif not (isinstance(node, ast.Constant) and isinstance(node.value, bool)):
+            self._fail(node, "is not a condition: compare it with something, as in 'toxicity >= 4'")
+
+    def _operand(self, node: ast.AST) -> tuple[str, Any]:
+        """What one side of a comparison is: ("label", name), ("str", value), ("num", name) for a numeric field or
+        ("num", None) for a number."""
+        if isinstance(node, ast.Name):
+            return ("label", node.id) if node.id in self.labels else ("num", node.id)
+        if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub):
+            kind = self._operand(node.operand)
+            if kind[0] != "num":
+                self._fail(node, "negates something that is not a number")
+            return kind
+        if isinstance(node, ast.Constant):
+            v = node.value
+            if isinstance(v, str):
+                return ("str", v)
+            if (isinstance(v, int) and not isinstance(v, bool)) or (isinstance(v, float) and math.isfinite(v)):
+                return ("num", None)
+            self._fail(node, "is not allowed in a comparison: use a finite number or a label value in quotes")
+        self._fail(node, "is not allowed in a comparison: compare fields, numbers and label values")
+        raise AssertionError  # not reached
+
+    def _check_pair(self, node: ast.Compare, op: ast.cmpop, a: tuple[str, Any], b: tuple[str, Any]) -> None:
+        kinds = {a[0], b[0]}
+        if all(k == "str" or name is None for k, name in (a, b)):
+            self._fail(node, "compares two constants")
+        if kinds == {"num"}:
+            return
+        if kinds == {"label", "str"}:
+            (_, label), (_, value) = (a, b) if a[0] == "label" else (b, a)
+            if not isinstance(op, (ast.Eq, ast.NotEq)):
+                self._fail(node, f"compares the label {label} by order: use == or !=")
+            if value not in self.labels[label]:
+                self._fail(node, f"uses {value!r}, which is not a value of {label} (one of: "
+                                 f"{', '.join(self.labels[label])})")
+            return
+        if kinds == {"label"}:
+            self._fail(node, "compares two labels: compare a label with one of its values, as in format == 'news'")
+        self._fail(node, "compares a score with a string, or a label with a number")
 
     def reasons(self, env: dict) -> list[str]:
         """The terms of the line that match ``env`` (each ``or`` branch on its own); [] means keep."""
@@ -341,16 +432,11 @@ class DropLine:
             left = self._eval(node.left, env)
             for op, comp in zip(node.ops, node.comparators):
                 right = self._eval(comp, env)
-                if not isinstance(op, (ast.Eq, ast.NotEq)) and (left is None or right is None):
-                    return False
-                try:
-                    if not _CMP[type(op)](left, right):
-                        return False
-                except TypeError:
-                    return False
+                if left is None or right is None or not _CMP[type(op)](left, right):
+                    return False  # a comparison with a missing value is false, != included
                 left = right
             return True
-        raise ValueError(f"unsupported drop-line node {type(node).__name__}")
+        raise DropLineError(f"unsupported drop-line node {type(node).__name__}")
 
 
 def _r3(x: float | None) -> float | None:
@@ -567,6 +653,23 @@ def _load_calibration(path: Path, drop_line: str | None, apply_offsets: bool) ->
     return calibration
 
 
+def _make_drop_line(config: dict, calibration: dict | None, drop_line: str | None = None) -> DropLine:
+    """The ``DropLine`` of ``drop_line`` for the schema of source1.json (``config``): None or "calibrated" =
+    calibration.json's line, "default" = the schema's hard filters, anything else = the expression itself, checked
+    against the schema's fields and label values (ValueError when it is not a valid drop line)."""
+    schema = config["schema"]
+    if drop_line in (None, "calibrated"):
+        drop_line = ((calibration or {}).get("drop_line") or {}).get("line")
+        if not drop_line:
+            raise ValueError("no calibrated drop line (calibration.json missing or incomplete); pass "
+                             "drop_line='default' or your own drop line")
+    elif drop_line == "default":
+        drop_line = " or ".join((schema.get("composite") or {}).get("hard_filters") or []) or "False"
+    labels = {name: list(spec["values"]) for name, spec in schema["labels"].items()}
+    numbers = [n for g in ("quality", "red_flags", "gated") for n in schema.get(g, {})] + ["overall", "tokens", "parts"]
+    return DropLine(drop_line, [*labels, *numbers], labels)
+
+
 class Source1(nn.Module):
     """Source-1: the mmBERT-base encoder, mean pooling and one linear head per field. Build it with
     ``Source1.from_pretrained``; score documents with ``score`` / ``score_batch``."""
@@ -590,15 +693,7 @@ class Source1(nn.Module):
         self.labels = {name: list(spec["values"]) for name, spec in self.schema["labels"].items()}
         self.fields = list(self.labels) + [n for g in ("quality", "red_flags", "gated") for n in self.schema.get(g, {})]
         self.calibration = calibration or {}
-        default_line = " or ".join((self.schema.get("composite") or {}).get("hard_filters") or []) or "False"
-        if drop_line in (None, "calibrated"):
-            drop_line = (self.calibration.get("drop_line") or {}).get("line")
-            if not drop_line:
-                raise ValueError("no calibrated drop line (calibration.json missing or incomplete); pass "
-                                 "drop_line='default' or your own drop line")
-        elif drop_line == "default":
-            drop_line = default_line
-        self.drop_line = DropLine(drop_line, set(self.fields) | {"overall", "tokens", "parts"})
+        self.drop_line = _make_drop_line(config, self.calibration, drop_line)
         self.offsets = {k: float(v["offset"]) for k, v in (self.calibration.get("offsets") or {}).items()}
         self.apply_offsets = apply_offsets
         self.show_url = show_url
@@ -658,11 +753,21 @@ class Source1(nn.Module):
             raise FileNotFoundError(f"{path} lacks {', '.join(missing)}")
         config = json.loads((path / "source1.json").read_text(encoding="utf-8"))
         calibration = _load_calibration(path, drop_line, apply_offsets)
+        _make_drop_line(config, calibration, drop_line)  # a bad drop line fails here, before the weights load
         tok = Tokenizer.from_file(str(path / "tokenizer.json"))
         tok.no_truncation()
         tok.no_padding()
         variant = {"variant": "fp32"} if precision == "fp32" else {}
-        backbone = AutoModel.from_pretrained(str(path), **{_dtype_kwarg(): dtype}, **variant, **backbone_kwargs)
+        backbone_kwargs.pop("output_loading_info", None)  # always requested, to check the weights
+        backbone, info = AutoModel.from_pretrained(str(path), **{_dtype_kwarg(): dtype}, **variant, **backbone_kwargs,
+                                                   output_loading_info=True)
+        keys = ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
+        bad = {k: info[k] for k in keys if info.get(k)}
+        if bad:  # transformers would otherwise fill the missing weights with random values, without an error
+            detail = "; ".join(f"{len(v)} {k.replace('_', ' ')} ({', '.join(sorted(map(str, v))[:3])}"
+                               f"{', ...' if len(v) > 3 else ''})" for k, v in bad.items())
+            raise RuntimeError(f"Source-1: the backbone weights in {path / weights} do not match the model: {detail}. "
+                               "Download the weights again, and use the package versions in requirements.txt")
         model = cls(backbone, tok, config, calibration, drop_line, apply_offsets, show_url)
         model.heads.load_state_dict(load_file(str(path / "heads.safetensors")))
         model.heads.to(dtype)
@@ -754,12 +859,19 @@ class Source1(nn.Module):
         rec["drop_reasons"] = reasons
         return rec
 
-    def score_inputs(self, inputs: list[str], batch_tokens: int = DEFAULT_BATCH_TOKENS) -> list[dict]:
+    def default_batch_tokens(self) -> int:
+        """The padded tokens per forward pass when ``batch_tokens`` is None: 16,384 on a CPU, 65,536 on a GPU."""
+        return DEFAULT_BATCH_TOKENS_CPU if self.device.type == "cpu" else DEFAULT_BATCH_TOKENS
+
+    def score_inputs(self, inputs: list[str], batch_tokens: int | None = None) -> list[dict]:
         """Score ready-made model inputs (``build_input`` output: header, blank line, chunk text), one dict per
         input with the 13 fields, overall, keep, drop_reasons, input_tokens and truncated.
 
-        Inputs are sorted by length and batched with at most ``batch_tokens`` padded tokens per forward pass;
-        a batch that runs out of GPU memory is split in half and retried."""
+        Inputs are sorted by length and batched with at most ``batch_tokens`` padded tokens per forward pass
+        (default: 16,384 on a CPU, 65,536 on a GPU); a batch that runs out of GPU memory is split in half and
+        retried."""
+        if batch_tokens is None:
+            batch_tokens = self.default_batch_tokens()
         encoded = self.encode(inputs)
         ids = [e[0] for e in encoded]
         results: list[dict | None] = [None] * len(ids)
@@ -789,13 +901,13 @@ class Source1(nn.Module):
             run(batch)
         return results  # type: ignore[return-value]
 
-    def score_batch(self, docs: Iterable[str | bytes | dict], batch_tokens: int = DEFAULT_BATCH_TOKENS, *,
+    def score_batch(self, docs: Iterable[str | bytes | dict], batch_tokens: int | None = None, *,
                     text_field: str = "text", max_chunks: int = 0) -> list[dict]:
         """Score a list of documents: strings (bytes are decoded as UTF-8), or dicts with the text under
         ``text_field`` and optionally ``title``, ``url``, ``source_type`` and ``code_language`` (see ``score``).
         A None text is scored as an empty document (keep False, drop_reasons ["empty text"]). Chunks of all
         documents are batched together. ``max_chunks`` > 0 scores only that many evenly spaced chunks of a long
-        document (the Part numbers still count every chunk).
+        document (the Part numbers still count every chunk). ``batch_tokens``: see ``score_inputs``.
 
         In bfloat16 a document's scores can shift slightly (up to about 0.04 on overall, 0.10 on a single field)
         depending on which other documents share its batch, because the batch shape changes the kernels' rounding;
@@ -831,7 +943,7 @@ class Source1(nn.Module):
 
     def score(self, text: str | bytes, title: str | None = None, url: str | None = None, *,
               source_type: str | None = None, code_language: str | None = None, max_chunks: int = 0,
-              batch_tokens: int = DEFAULT_BATCH_TOKENS) -> dict:
+              batch_tokens: int | None = None) -> dict:
         """Score one document (a str; bytes are decoded as UTF-8; anything else raises TypeError).
 
         title: shown to the model in the header when given (as in training, where about a quarter of inputs had one).
@@ -869,14 +981,108 @@ class Source1(nn.Module):
 
 
 class BadInput(ValueError):
-    """A record of the input file that cannot be scored (the message starts with file:line)."""
+    """A record of the input file that cannot be scored (the message starts with file:line), or, with
+    ``skippable=False``, an input file that cannot be read at all."""
+
+    def __init__(self, message: str, skippable: bool = True):
+        super().__init__(message)
+        self.skippable = skippable
+
+
+# Compressed inputs read transparently, recognized by their first bytes whatever their name.
+_DECOMPRESS = {"gzip": gzip.open, "bzip2": bz2.open, "xz": lzma.open}
+_COMPRESSED_SUFFIXES = (".gz", ".bz2", ".xz")  # docs.jsonl.gz is read as .jsonl
+_UNREADABLE = {".zst": "is zstd-compressed: decompress it first (zstd -d)",
+               ".zstd": "is zstd-compressed: decompress it first (zstd -d)",
+               ".parquet": "is a Parquet file: convert it to JSON Lines first",
+               ".zip": "is a zip archive: extract it first",
+               ".7z": "is a 7z archive: extract it first"}
+_READ_ERRORS = (OSError, EOFError, zlib.error, lzma.LZMAError)  # what damaged or truncated compressed files raise
+_SNIFF_BYTES = 65536  # how much of an input file _input_problem looks at
+_MAX_INVALID = 0.2  # share of invalid UTF-8 sequences among the characters above which an input file is refused
+_MAX_NUL = 0.01  # share of NUL bytes above which an input file is refused as binary
+
+
+def _compression(head: bytes) -> str | None:
+    """The compression that a file's first bytes show: "gzip", "bzip2", "xz", "zstd" or None."""
+    if head.startswith(b"\x1f\x8b"):
+        return "gzip"
+    if head[:3] == b"BZh" and b"1" <= head[3:4] <= b"9" and head[4:10] in (b"1AY&SY", b"\x17rE8P\x90"):
+        return "bzip2"
+    if head.startswith(b"\xfd7zXZ\x00"):
+        return "xz"
+    if head.startswith(b"\x28\xb5\x2f\xfd"):
+        return "zstd"
+    return None
+
+
+def _open_input(path: Path) -> Any:
+    """``path`` opened for reading bytes, decompressed when it is a gzip, bzip2 or xz file."""
+    with open(path, "rb") as f:
+        head = f.read(10)
+    return _DECOMPRESS.get(_compression(head) or "", open)(path, "rb")
+
+
+def _format_suffix(path: Path) -> str:
+    """The suffix that says how to read ``path``: its last one, or the one before .gz, .bz2 or .xz."""
+    suffixes = [s.lower() for s in path.suffixes]
+    if suffixes and suffixes[-1] in _COMPRESSED_SUFFIXES:
+        suffixes.pop()
+    return suffixes[-1] if suffixes else ""
+
+
+def _input_problem(path: Path) -> str | None:
+    """Why ``path`` cannot be scored as UTF-8 text or JSON (zstd, Parquet, an archive, binary, UTF-16, mostly invalid
+    UTF-8, damaged), judged from its name and its first 64 KiB once decompressed; None when it looks readable."""
+    if path.suffix.lower() in _UNREADABLE:
+        return _UNREADABLE[path.suffix.lower()]
+    try:
+        with _open_input(path) as f:
+            head = f.read(_SNIFF_BYTES)
+    except _READ_ERRORS as e:
+        return f"cannot be read ({e})"
+    if _compression(head) == "zstd":
+        return _UNREADABLE[".zst"]
+    if head.startswith((b"\xff\xfe", b"\xfe\xff")):
+        return "is UTF-16 or UTF-32: save it as UTF-8"
+    if head.count(b"\x00") > _MAX_NUL * len(head):  # a stray NUL in a text is dropped like other control characters
+        return "holds NUL bytes, so it is not UTF-8 text (binary, UTF-16, or compressed in a format not read here)"
+    text = head.decode("utf-8", errors="replace")
+    invalid = text.count("\ufffd") - head.count("\ufffd".encode())  # a U+FFFD already in the text is valid UTF-8
+    if invalid >= 8 and invalid > _MAX_INVALID * len(text):
+        return f"is not UTF-8 text: {invalid / len(text):.0%} of its first {len(text):,} characters are invalid UTF-8"
+    return None
+
+
+def _json_error(e: BaseException) -> str:
+    """Why json.loads failed, in a few words."""
+    if isinstance(e, json.JSONDecodeError):
+        return f"{e.msg} at column {e.colno}"
+    if isinstance(e, RecursionError):
+        return "nested too deeply"
+    return str(e).split(";")[0]  # e.g. an integer of more than 4,300 digits
+
+
+def _unwritable(value: Any) -> str | None:
+    """Why ``value`` cannot be written to the JSON Lines output, or None when it can."""
+    try:
+        json.dumps(value, ensure_ascii=False, allow_nan=False).encode("utf-8")
+    except UnicodeEncodeError:
+        return "holds a lone surrogate, which UTF-8 cannot encode"
+    except ValueError:
+        return "holds NaN or an infinite number, which JSON cannot hold"
+    except RecursionError:
+        return "is nested too deeply"
+    return None
 
 
 def _read_docs(path: Path, text_field: str, skip_bad: bool = False) -> Iterator[dict]:
     """Documents of an input file: .jsonl / .ndjson (one JSON object per line), .json (a JSON array of objects, one
-    object, or JSON Lines), or any other file as one plain-text document. Invalid UTF-8 is replaced (with a warning
-    for JSON). A bad record raises BadInput, or with ``skip_bad`` is reported on stderr and skipped. A null text is
-    kept and scored as an empty document."""
+    object, or JSON Lines), or any other file as one plain-text document; gzip, bzip2 and xz files are decompressed
+    (docs.jsonl.gz is read as .jsonl). Invalid UTF-8 is replaced, with a warning. A bad record (not a JSON object,
+    no string text, an id that cannot be written back) raises BadInput, or with ``skip_bad`` is reported on stderr
+    and skipped. A file that cannot be read at all (see ``_input_problem``) raises BadInput with skippable=False.
+    A null text is kept and scored as an empty document."""
 
     def usable(rec: Any, where: str) -> bool:
         if not isinstance(rec, dict):
@@ -885,6 +1091,8 @@ def _read_docs(path: Path, text_field: str, skip_bad: bool = False) -> Iterator[
             problem = f"no {text_field!r} field"
         elif rec[text_field] is not None and not isinstance(rec[text_field], str):
             problem = f"{text_field!r} is a {type(rec[text_field]).__name__}, not a string"
+        elif "id" in rec and (why := _unwritable(rec["id"])):
+            problem = f"the id {why}"
         else:
             return True
         if not skip_bad:
@@ -900,37 +1108,52 @@ def _read_docs(path: Path, text_field: str, skip_bad: bool = False) -> Iterator[
                   file=sys.stderr)
             return raw.decode("utf-8", errors="replace")
 
-    suffix = path.suffix.lower()
+    def read_all() -> bytes:
+        try:
+            with _open_input(path) as f:
+                return f.read()
+        except _READ_ERRORS as e:
+            raise BadInput(f"{path} cannot be read ({e})", skippable=False) from None
+
+    problem = _input_problem(path)
+    if problem:
+        raise BadInput(f"{path} {problem}", skippable=False)
+    suffix = _format_suffix(path)
     if suffix not in (".jsonl", ".ndjson", ".json"):
-        yield {text_field: path.read_text(encoding="utf-8", errors="replace"), "id": path.name}
+        yield {text_field: decode(read_all(), str(path)), "id": path.name}
         return
     if suffix == ".json":
         try:
-            data = json.loads(decode(path.read_bytes(), str(path)).lstrip(""))
-        except json.JSONDecodeError:
-            data = None  # not a single JSON value: read the file as JSON Lines below
+            data = json.loads(decode(read_all(), str(path)).lstrip(""))
+        except (ValueError, RecursionError):  # JSONDecodeError is a ValueError
+            data = None  # not one JSON value (or too large or too deep to read as one): read it as JSON Lines below
         if data is not None:
             for n, rec in enumerate(data if isinstance(data, list) else [data]):
                 if usable(rec, f"{path}[{n}]"):
                     yield rec
             return
-    with open(path, "rb") as f:
-        for n, raw in enumerate(f, 1):
-            where = f"{path}:{n}"
-            line = decode(raw, where)
-            if n == 1:
-                line = line.lstrip("")
-            if not line.strip():
-                continue
-            try:
-                rec = json.loads(line)
-            except json.JSONDecodeError as e:
-                if not skip_bad:
-                    raise BadInput(f"{where}: invalid JSON ({e.msg} at column {e.colno})") from None
-                print(f"source1: skipped {where}: invalid JSON ({e.msg} at column {e.colno})", file=sys.stderr)
-                continue
-            if usable(rec, where):
-                yield rec
+    n = 0
+    try:
+        with _open_input(path) as f:
+            for n, raw in enumerate(f, 1):
+                where = f"{path}:{n}"
+                line = decode(raw, where)
+                if n == 1:
+                    line = line.lstrip("")
+                if not line.strip():
+                    continue
+                try:
+                    rec = json.loads(line)
+                except (ValueError, RecursionError) as e:  # also integers too long to read, and very deep nesting
+                    problem = f"invalid JSON ({_json_error(e)})"
+                    if not skip_bad:
+                        raise BadInput(f"{where}: {problem}") from None
+                    print(f"source1: skipped {where}: {problem}", file=sys.stderr)
+                    continue
+                if usable(rec, where):
+                    yield rec
+    except _READ_ERRORS as e:
+        raise BadInput(f"{path} cannot be read after line {n} ({e})", skippable=False) from None
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -938,10 +1161,11 @@ def main(argv: list[str] | None = None) -> int:
     p.add_argument("--model", default=str(Path(__file__).resolve().parent),
                    help="Source-1 directory or Hugging Face repo id (default: this file's directory)")
     p.add_argument("--input", required=True, help=".jsonl (one document per line), .json (an array of objects) or "
-                   "a text file (one document)")
+                   "a text file (one document); gzip, bzip2 and xz files are decompressed (docs.jsonl.gz)")
     p.add_argument("--text-field", default="text", help="JSON field holding the text (default: text); "
                    "title, url, source_type and code_language fields are used when present")
-    p.add_argument("--output", help="output .jsonl (default: standard output)")
+    p.add_argument("--output", help="output .jsonl (default: standard output); written to <output>.tmp and renamed "
+                   "at the end, so a run that fails leaves an older output as it was")
     p.add_argument("--skip-bad", action="store_true", help="skip (and report on stderr) records that are not valid "
                    "JSON objects with a string text, instead of stopping")
     p.add_argument("--revision", help="branch, tag or commit, when --model is a Hugging Face repo id")
@@ -951,7 +1175,8 @@ def main(argv: list[str] | None = None) -> int:
     p.add_argument("--dtype", choices=DTYPE_CHOICES, default="auto", help="what to compute in; auto (default): "
                    "bfloat16 on a GPU with native bfloat16, else float32 (bf16 weights upcast); float16 is not "
                    "supported")
-    p.add_argument("--batch-tokens", type=int, default=DEFAULT_BATCH_TOKENS, help="padded tokens per forward pass")
+    p.add_argument("--batch-tokens", type=int, help="padded tokens per forward pass (default: "
+                   f"{DEFAULT_BATCH_TOKENS_CPU} on a CPU, {DEFAULT_BATCH_TOKENS} on a GPU)")
     p.add_argument("--max-chunks", type=int, default=0, help="score at most N evenly spaced chunks per document")
     p.add_argument("--drop-line", help='"calibrated" (default), "default" (the schema\'s hard filters) or an '
                    "expression such as 'toxicity >= 4 or spam_seo >= 3'")
@@ -960,50 +1185,109 @@ def main(argv: list[str] | None = None) -> int:
     p.add_argument("--no-chunks", action="store_true", help="leave out the per-chunk list of split documents")
     p.add_argument("--group", type=int, default=256, help="documents scored together")
     args = p.parse_args(argv)
-    if not Path(args.input).is_file():
+    source = Path(args.input)
+    if not source.is_file():
         p.error(f"--input {args.input}: no such file")
+    problem = _input_problem(source)
+    if problem:
+        p.error(f"--input {args.input} {problem}")
+    if args.batch_tokens is not None and args.batch_tokens < 1:
+        p.error("--batch-tokens must be at least 1")
+    # Checked before the model loads. A regular output file is written as <output>.tmp, which replaces the output
+    # only when the run succeeds; a device or pipe (such as /dev/stdout) is written directly.
+    target = tmp = None
+    if args.output:
+        out_path = Path(args.output)
+        if out_path.is_dir():
+            p.error(f"--output {args.output} is a folder; give a file name")
+        if out_path.exists() and out_path.samefile(source):
+            p.error("--output is the same file as --input; refusing to overwrite it")
+        if not out_path.resolve().parent.is_dir():
+            p.error(f"--output {args.output}: the folder does not exist")
+        if out_path.is_file() or not out_path.exists():
+            target = out_path.resolve()
+            tmp = target.with_name(target.name + ".tmp")
+            if tmp.is_dir() or (tmp.exists() and tmp.samefile(source)):
+                p.error(f"--output {args.output}: the temporary file it is written to first, {tmp}, is "
+                        + ("a folder" if tmp.is_dir() else "the --input file"))
 
     t0 = time.time()
-    model = Source1.from_pretrained(args.model, device=args.device, dtype=args.dtype, precision=args.precision,
-                                    drop_line=args.drop_line, apply_offsets=args.apply_offsets,
-                                    show_url=args.show_url, revision=args.revision)
+    try:
+        model = Source1.from_pretrained(args.model, device=args.device, dtype=args.dtype, precision=args.precision,
+                                        drop_line=args.drop_line, apply_offsets=args.apply_offsets,
+                                        show_url=args.show_url, revision=args.revision)
+    except DropLineError as e:
+        p.error(str(e))
     compute = str(next(model.parameters()).dtype).replace("torch.", "")
     print(f"source1: loaded {model.weights_file} ({model.weights_dtype or '?'} weights) on {model.device}, computing "
           f"in {compute}, in {time.time() - t0:.1f} s; drop line: {model.drop_line.source}", file=sys.stderr)
-    out = open(args.output, "w", encoding="utf-8") if args.output else sys.stdout
-    done = 0
+    out = open(tmp or args.output, "w", encoding="utf-8") if args.output else sys.stdout
+    done = seen = 0
     t0 = time.time()
 
     def flush(group: list[dict]) -> None:
-        nonlocal done
+        nonlocal done, seen
         for rec, res in zip(group, model.score_batch(group, args.batch_tokens, text_field=args.text_field,
                                                      max_chunks=args.max_chunks)):
+            seen += 1
             if args.no_chunks:
                 res.pop("chunks", None)
             if "id" in rec:
                 res = {"id": rec["id"], **res}
-            out.write(json.dumps(res, ensure_ascii=False, allow_nan=False) + "\n")
-        done += len(group)
+            try:  # the reader lets no unwritable id through; this keeps a half-written line out of the output
+                line = json.dumps(res, ensure_ascii=False, allow_nan=False)
+                line.encode("utf-8")
+            except (ValueError, RecursionError) as e:
+                where = f"document {seen:,}" + (f" (id {rec['id']!r})" if "id" in rec else "")
+                if not args.skip_bad:
+                    raise BadInput(f"{where}: its scores cannot be written as JSON ({e})") from None
+                print(f"source1: skipped {where}: its scores cannot be written as JSON ({e})", file=sys.stderr)
+                continue
+            out.write(line + "\n")
+            done += 1
         print(f"source1: {done:,} documents, {done / max(time.time() - t0, 1e-9):.1f}/s", file=sys.stderr)
 
+    def written() -> str:
+        """What a run that stopped early left behind."""
+        what = f"The {done:,} documents before it were" if done != 1 else "The document before it was"
+        if tmp is None:
+            return f"{what} written"
+        if not done:
+            tmp.unlink(missing_ok=True)
+            return f"Nothing was written, and {args.output} was not changed"
+        return f"{what} written to {tmp}, and {args.output} was not changed"
+
+    docs = _read_docs(source, args.text_field, args.skip_bad)
+    group: list[dict] = []
     try:
-        group: list[dict] = []
         try:
-            for rec in _read_docs(Path(args.input), args.text_field, args.skip_bad):
+            while True:
+                try:
+                    rec = next(docs)
+                except StopIteration:
+                    break
+                except Exception:
+                    if group:
+                        flush(group)  # the documents read before a bad record or a read error are still scored
+                    raise
                 group.append(rec)
                 if len(group) >= args.group:
                     flush(group)
                     group = []
-        except BadInput as e:
             if group:
-                flush(group)  # the documents read before the bad record are still scored and written
-            raise SystemExit(f"source1: {e}. The {done:,} documents before it were written; --skip-bad skips "
-                             "bad records") from None
-        if group:
-            flush(group)
-    finally:
-        if out is not sys.stdout:
-            out.close()
+                flush(group)
+        finally:
+            if out is not sys.stdout:
+                out.close()
+    except BadInput as e:
+        raise SystemExit(f"source1: {e}. {written()}"
+                         + ("; --skip-bad skips bad records" if e.skippable else "")) from None
+    except BaseException as e:
+        if tmp is not None:
+            print(f"source1: stopped by {type(e).__name__}. {written()}", file=sys.stderr)
+        raise
+    if tmp is not None:
+        tmp.replace(target)
     return 0