Oh, thank you!
Stefano Fiorucci PRO
anakin87
AI & ML interests
Language Models: orchestration, post-training, GRPO, synthetic data...
Contributing to Haystack LLM framework ๐๏ธ
Recent Activity
repliedto their post about 12 hours ago
How LLM training with RL Environments works?
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs reacted to theirpost with โค๏ธ 1 day ago
How LLM training with RL Environments works?
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs posted an update 1 day ago
How LLM training with RL Environments works?
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMsOrganizations
replied to their post about 12 hours ago
reacted to their post with โค๏ธ 1 day ago
Post
6603
How LLM training with RL Environments works?
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs
- 2 replies
posted an update 1 day ago
Post
6603
How LLM training with RL Environments works?
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs
It all starts with ๐ฅ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐ฉ๐ฒ๐ฟ๐ถ๐ณ๐ถ๐ฎ๐ฏ๐น๐ฒ ๐ฅ๐ฒ๐๐ฎ๐ฟ๐ฑ๐
- question asked
- model generates reasoning + answer
- answer checked against ground truth
- reward drives RL training
In this setup, the environment is simple: fixed questions and answers, rollout logic, reward(s)
Consider a more complex tic-tac-toe env โโญ
It adds:
- dynamic game generation/handling
- tunable opponent skill
- multi-turn interactions
(envs can also include tools)
---
What happens at training?
We use ๐๐ฟ๐ผ๐๐ฝ ๐ฅ๐ฒ๐น๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ผ๐น๐ถ๐ฐ๐ ๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป with a tic-tac-toe env
No critic model needed, the group is the baseline
Simpler than PPO
1๏ธโฃ Rollout generation: from the same board, model plays N games via sampling
2๏ธโฃ Each game scored with deterministic rewards (win, format, ...)
3๏ธโฃ Mean score computed across the group
4๏ธโฃ Each rollout's advantage = its score minus the group mean
5๏ธโฃ Model updated to favor trajectories above baseline
๐ Repeat
For a deep dive, check out
๐ฑ https://github.com/anakin87/llm-rl-environments-lil-course
a free hands-on course on RL environments for LLMs
- 2 replies
liked a
model 3 days ago
reacted to their post with ๐ค 5 days ago
Post
1603
Your RL environment is an SFT data factory ๐ญ
In LLM post-training it's common to do Supervised Fine-Tuning warm-up before Reinforcement Learning.
When teaching a new task, RL needs some signal to amplify and SFT builds a good initial basis, for example by teaching format.
If you've built an RL env, generating SFT synthetic data is basically free.
An env already has: task data, rollout logic, rewards.
1๏ธโฃ pick a strong model
2๏ธโฃ run it through the env
3๏ธโฃ filter rollouts by reward
works out of the box with Verifiers (Prime Intellect) and Atropos (Nous Research)
๐งโ๐ป Example: https://github.com/anakin87/llm-rl-environments-lil-course/blob/main/chapters/05.md
In LLM post-training it's common to do Supervised Fine-Tuning warm-up before Reinforcement Learning.
When teaching a new task, RL needs some signal to amplify and SFT builds a good initial basis, for example by teaching format.
If you've built an RL env, generating SFT synthetic data is basically free.
An env already has: task data, rollout logic, rewards.
1๏ธโฃ pick a strong model
2๏ธโฃ run it through the env
3๏ธโฃ filter rollouts by reward
works out of the box with Verifiers (Prime Intellect) and Atropos (Nous Research)
๐งโ๐ป Example: https://github.com/anakin87/llm-rl-environments-lil-course/blob/main/chapters/05.md
posted an update 5 days ago
Post
1603
Your RL environment is an SFT data factory ๐ญ
In LLM post-training it's common to do Supervised Fine-Tuning warm-up before Reinforcement Learning.
When teaching a new task, RL needs some signal to amplify and SFT builds a good initial basis, for example by teaching format.
If you've built an RL env, generating SFT synthetic data is basically free.
An env already has: task data, rollout logic, rewards.
1๏ธโฃ pick a strong model
2๏ธโฃ run it through the env
3๏ธโฃ filter rollouts by reward
works out of the box with Verifiers (Prime Intellect) and Atropos (Nous Research)
๐งโ๐ป Example: https://github.com/anakin87/llm-rl-environments-lil-course/blob/main/chapters/05.md
In LLM post-training it's common to do Supervised Fine-Tuning warm-up before Reinforcement Learning.
When teaching a new task, RL needs some signal to amplify and SFT builds a good initial basis, for example by teaching format.
If you've built an RL env, generating SFT synthetic data is basically free.
An env already has: task data, rollout logic, rewards.
1๏ธโฃ pick a strong model
2๏ธโฃ run it through the env
3๏ธโฃ filter rollouts by reward
works out of the box with Verifiers (Prime Intellect) and Atropos (Nous Research)
๐งโ๐ป Example: https://github.com/anakin87/llm-rl-environments-lil-course/blob/main/chapters/05.md
updated a
dataset 6 days ago
published a
dataset 6 days ago
liked a
model 6 days ago
liked a
Space 9 days ago
replied to their post 10 days ago
๐ฅ For a video walkthrough, check out "Let LLMs Wander - Engineering RL Environments": https://www.youtube.com/watch?v=71V3fTaUp2Q
reacted to their post with ๐ 10 days ago
Post
4135
๐ Let LLMs wander - Engineering RL Environments
Reinforcement Learning Environments are little worlds
where models can act, get rewards, and learn.
I've been exploring how to design them, figuring out what works and what doesn't.
If you want to learn how to build them, I recorded a practical intro video.
You'll also see how to turn Liquid AI LFM2-2.6B into a Tic-tac-toe master ๐
๐ฅ Engineering RL Environments video: https://www.youtube.com/watch?v=71V3fTaUp2Q
---
๐ฑ LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
Reinforcement Learning Environments are little worlds
where models can act, get rewards, and learn.
I've been exploring how to design them, figuring out what works and what doesn't.
If you want to learn how to build them, I recorded a practical intro video.
You'll also see how to turn Liquid AI LFM2-2.6B into a Tic-tac-toe master ๐
๐ฅ Engineering RL Environments video: https://www.youtube.com/watch?v=71V3fTaUp2Q
---
๐ฑ LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
posted an update 10 days ago
Post
4135
๐ Let LLMs wander - Engineering RL Environments
Reinforcement Learning Environments are little worlds
where models can act, get rewards, and learn.
I've been exploring how to design them, figuring out what works and what doesn't.
If you want to learn how to build them, I recorded a practical intro video.
You'll also see how to turn Liquid AI LFM2-2.6B into a Tic-tac-toe master ๐
๐ฅ Engineering RL Environments video: https://www.youtube.com/watch?v=71V3fTaUp2Q
---
๐ฑ LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
Reinforcement Learning Environments are little worlds
where models can act, get rewards, and learn.
I've been exploring how to design them, figuring out what works and what doesn't.
If you want to learn how to build them, I recorded a practical intro video.
You'll also see how to turn Liquid AI LFM2-2.6B into a Tic-tac-toe master ๐
๐ฅ Engineering RL Environments video: https://www.youtube.com/watch?v=71V3fTaUp2Q
---
๐ฑ LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
liked a
dataset 12 days ago
upvoted an article 12 days ago
Article
Multimodal Embedding & Reranker Models with Sentence Transformers
โข
50
reacted to their post with ๐ฅ 12 days ago
Post
3292
๐ฃ I just published a free course on Reinforcement Learning Environments for Language Models!
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
- 1 reply
reacted to their post with โค๏ธ 13 days ago
Post
3292
๐ฃ I just published a free course on Reinforcement Learning Environments for Language Models!
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
- 1 reply
liked a
Space 13 days ago
upvoted a collection 13 days ago
posted an update 13 days ago
Post
3292
๐ฃ I just published a free course on Reinforcement Learning Environments for Language Models!
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
๐ COURSE: https://github.com/anakin87/llm-rl-environments-lil-course
Over the past year, we've seen a shift in LLM Post-Training.
Previously, Supervised Fine-Tuning was the most important part: making models imitate curated Question-Answer pairs.
Now we also have Reinforcement Learning with Verifiable Rewards. With techniques like GRPO, models can learn through trial and error in dynamic environments. They can climb to new heights without relying on expensively prepared data.
But what actually are these environments in practiceโ And how do you build them effectivelyโ
Fascinated by these concepts, I spent time exploring this space through experiments, post-training Small Language Models.
I've packaged everything I learned into this short course.
What you'll learn
๐น Agents, Environments, and LLMs: how to map Reinforcement Learning concepts to the LLM domain
๐น How to use Verifiers (open-source library by Prime Intellect) to build RL environments as software artifacts
๐น Common patterns: How to build single-turn, multi-turn, and tool-use environments
๐น Hands-on: turn a small language model (LFM2-2.6B by LiquidAI) into a Tic Tac Toe master
๐ธ Build the game Environment
๐ธ Use it to generate synthetic data for SFT warm-up
๐ธ Group-based Reinforcement Learning
If you're interested in building "little worlds" where LLMs can learn, this course is for you.
---
๐ค๐น๏ธ Play against the trained model: anakin87/LFM2-2.6B-mr-tictactoe
๐ HF collection (datasets + models): https://huggingface.co/collections/anakin87/lfm2-26b-mr-tic-tac-toe
- 1 reply