EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Paper โข 2603.02041 โข Published
AI & ML interests
None defined yet.
Recent Activity
adorkinย
authored a
paper about 1 month ago
julien-cย
submitted a
paper to Daily Papers 2 months ago
adorkinย
authored a
paper 4 months ago
nouamanetaziย
posted an update 5 months ago
Post
4644
After training ๐๐ฆ๐จ๐ฅ๐๐๐ on ๐๐๐ ๐๐๐๐๐ฌ for nearly a month, I've come to realize something most people overlook: ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ฆ๐๐ค๐-๐จ๐ซ-๐๐ซ๐๐๐ค ๐๐๐๐ญ๐จ๐ซ ๐ข๐ง ๐๐๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ . ๐ฅ
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious ๐๐๐๐ ๐๐ซ๐ซ๐จ๐ซ๐ฌ, or when your expensive GPU cluster is running at ๐๐% ๐๐๐๐ข๐๐ข๐๐ง๐๐ฒ, the problem isn't your model. It's most probably a ๐ฆ๐ข๐ฌ๐ฎ๐ฌ๐ ๐จ๐ ๐ญ๐ก๐ ๐ก๐๐ซ๐๐ฐ๐๐ซ๐. ๐ ๏ธ
Questions that seemed simple but had no clear answers: Why is ๐๐จ๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฅ๐จ๐ฐ๐๐ซ ๐ญ๐ก๐๐ง ๐๐๐ง๐ฌ๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ? Which ๐๐๐๐ ๐๐ฅ๐๐ ๐ฌ should we actually set? How often should we checkpoint without killing throughput?
That's why we built ๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค ๐: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ฅ๐๐ฒ๐๐ซ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: ๐๐๐๐ ๐ก๐ข๐ญ๐ญ๐ข๐ง๐ ๐ ๐๐/๐ฌ, ๐๐๐๐ข๐ง๐ค ๐.๐ ๐ซ๐๐๐๐ก๐ข๐ง๐ ๐๐๐ ๐๐/๐ฌ, ๐๐๐๐ ๐๐๐ง๐ ๐๐ญ ๐๐.๐ ๐๐/๐ฌ. Then we ran collective operations across ๐๐๐ ๐๐๐๐ฌ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ๐๐๐ ๐๐/๐ฌ on a single node to ๐๐๐-๐๐๐ ๐๐/๐ฌ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค: https://lnkd.in/e5MKXUHS
Shared with โค๏ธ by the HuggingFace team
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious ๐๐๐๐ ๐๐ซ๐ซ๐จ๐ซ๐ฌ, or when your expensive GPU cluster is running at ๐๐% ๐๐๐๐ข๐๐ข๐๐ง๐๐ฒ, the problem isn't your model. It's most probably a ๐ฆ๐ข๐ฌ๐ฎ๐ฌ๐ ๐จ๐ ๐ญ๐ก๐ ๐ก๐๐ซ๐๐ฐ๐๐ซ๐. ๐ ๏ธ
Questions that seemed simple but had no clear answers: Why is ๐๐จ๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฅ๐จ๐ฐ๐๐ซ ๐ญ๐ก๐๐ง ๐๐๐ง๐ฌ๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ? Which ๐๐๐๐ ๐๐ฅ๐๐ ๐ฌ should we actually set? How often should we checkpoint without killing throughput?
That's why we built ๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค ๐: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ฅ๐๐ฒ๐๐ซ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: ๐๐๐๐ ๐ก๐ข๐ญ๐ญ๐ข๐ง๐ ๐ ๐๐/๐ฌ, ๐๐๐๐ข๐ง๐ค ๐.๐ ๐ซ๐๐๐๐ก๐ข๐ง๐ ๐๐๐ ๐๐/๐ฌ, ๐๐๐๐ ๐๐๐ง๐ ๐๐ญ ๐๐.๐ ๐๐/๐ฌ. Then we ran collective operations across ๐๐๐ ๐๐๐๐ฌ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ๐๐๐ ๐๐/๐ฌ on a single node to ๐๐๐-๐๐๐ ๐๐/๐ฌ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค: https://lnkd.in/e5MKXUHS
Shared with โค๏ธ by the HuggingFace team
adorkinย
authored a
paper 5 months ago
vumichienย
authored 2
papers 6 months ago
MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
Paper โข 2509.25531 โข Published โข 10
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Paper โข 2510.08697 โข Published โข 39
w11woย
authored a
paper 8 months ago
adorkinย
authored a
paper 10 months ago
DrishtiSharmaย
authored a
paper 11 months ago
w11woย
authored a
paper 11 months ago
Post
10272
BOOOOM: Today I'm dropping TINY AGENTS
the 50 lines of code Agent in Javascript ๐ฅ
I spent the last few weeks working on this, so I hope you will like it.
I've been diving into MCP (Model Context Protocol) to understand what the hype was all about.
It is fairly simple, but still quite powerful: MCP is a standard API to expose sets of Tools that can be hooked to LLMs.
But while doing that, came my second realization:
Once you have a MCP Client, an Agent is literally just a while loop on top of it. ๐คฏ
โก๏ธ read it exclusively on the official HF blog: https://huggingface.co/blog/tiny-agents
the 50 lines of code Agent in Javascript ๐ฅ
I spent the last few weeks working on this, so I hope you will like it.
I've been diving into MCP (Model Context Protocol) to understand what the hype was all about.
It is fairly simple, but still quite powerful: MCP is a standard API to expose sets of Tools that can be hooked to LLMs.
But while doing that, came my second realization:
Once you have a MCP Client, an Agent is literally just a while loop on top of it. ๐คฏ
โก๏ธ read it exclusively on the official HF blog: https://huggingface.co/blog/tiny-agents
- 1 reply
philschmidย
posted an update 12 months ago
Post
5754
Gemini 2.5 Flash is here! We excited launch our first hybrid reasoning Gemini model. In Flash 2.5 developer can turn thinking off.
**TL;DR:**
- ๐ง ย Controllable "Thinking" with thinking budget with up to 24k token
- ๐ย 1 Million multimodal inputย context for text, image, video, audio, and pdf
- ๐ ๏ธย Function calling, structured output, google search & code execution.
- ๐ฆย $0.15 1M input tokens; $0.6 or $3.5 (thinking on) per million output tokens (thinking tokens are billed as output tokens)
- ๐กย Knowledge cut ofย January 2025
- ๐ย Rate limits - Free 10 RPM 500 req/day
- ๐ Outperforms 2.0 Flash on every benchmark
Try it โฌ๏ธ
https://aistudio.google.com/prompts/new_chat?model=gemini-2.5-flash-preview-04-17
**TL;DR:**
- ๐ง ย Controllable "Thinking" with thinking budget with up to 24k token
- ๐ย 1 Million multimodal inputย context for text, image, video, audio, and pdf
- ๐ ๏ธย Function calling, structured output, google search & code execution.
- ๐ฆย $0.15 1M input tokens; $0.6 or $3.5 (thinking on) per million output tokens (thinking tokens are billed as output tokens)
- ๐กย Knowledge cut ofย January 2025
- ๐ย Rate limits - Free 10 RPM 500 req/day
- ๐ Outperforms 2.0 Flash on every benchmark
Try it โฌ๏ธ
https://aistudio.google.com/prompts/new_chat?model=gemini-2.5-flash-preview-04-17
- 1 reply
DrishtiSharmaย
authored a
paper 12 months ago
nouamanetaziย
authored a
paper 12 months ago
philschmidย
authored a
paper about 1 year ago
philschmidย
posted an update about 1 year ago
Post
4529
Gemini 2.5 Pro, thinking by default! We excited launch our best Gemini model for reasoning, multimodal and coding yet! #1 on LMSYS, Humanityโs Last Exam, AIME and GPQA and more!
TL;DR:
- ๐ปย Best Gemini coding model yet, particularly for web development (excels on LiveCodeBench).
- ๐ง ย Default "Thinking" with up to 64k token output
- ๐ย 1 Million multimodal inputย context for text, image, video, audio, and pdf
- ๐ ๏ธย Function calling, structured output, google search & code execution.
- ๐ย ย #1 on LMArena & sota on AIME, GPQA, Humanity's Last Exam
- ๐กย Knowledge cut ofย January 2025
- ๐คย Available for free as Experimental in AI Studio, Gemini API & Gemini APP
- ๐ย Rate limits - Free 2 RPM 50 req/day
Try it โฌ๏ธ
https://aistudio.google.com/?model=gemini-2.5-pro-exp-03-25
TL;DR:
- ๐ปย Best Gemini coding model yet, particularly for web development (excels on LiveCodeBench).
- ๐ง ย Default "Thinking" with up to 64k token output
- ๐ย 1 Million multimodal inputย context for text, image, video, audio, and pdf
- ๐ ๏ธย Function calling, structured output, google search & code execution.
- ๐ย ย #1 on LMArena & sota on AIME, GPQA, Humanity's Last Exam
- ๐กย Knowledge cut ofย January 2025
- ๐คย Available for free as Experimental in AI Studio, Gemini API & Gemini APP
- ๐ย Rate limits - Free 2 RPM 50 req/day
Try it โฌ๏ธ
https://aistudio.google.com/?model=gemini-2.5-pro-exp-03-25
- 3 replies
w11woย
authored a
paper about 1 year ago
Post
4449
Important notice ๐จ
For Inference Providers who have built support for our Billing API (currently: Fal, Novita, HF-Inference โ with more coming soon), we've started enabling Pay as you go (=PAYG)
What this means is that you can use those Inference Providers beyond the free included credits, and they're charged to your HF account.
You can see it on this view: any provider that does not have a "Billing disabled" badge, is PAYG-compatible.
For Inference Providers who have built support for our Billing API (currently: Fal, Novita, HF-Inference โ with more coming soon), we've started enabling Pay as you go (=PAYG)
What this means is that you can use those Inference Providers beyond the free included credits, and they're charged to your HF account.
You can see it on this view: any provider that does not have a "Billing disabled" badge, is PAYG-compatible.
- 9 replies
vumichienย
authored a
paper about 1 year ago