At last: Russian LLM fine-tune

[izard]

For a long time, there was no decent local LLM that understood and spoke Russian well. I've tested many fine-tunes, and they all behaved significantly dumber when answering in Russian than their respective English originals.

Russian internet has a huge potential training corpus second only to English; but smart French, German, Spanish and Arabic fine-tunes for local LLMs existed for months now, and there was no Russian one.

Until Serge Gotsuliak released his finetune few days ago.

Highly recommend it, works great with llama.cpp. I was missing RAG on my Russian books, will set it up now.

There is a drawback: 4-bit quant that I tested needs GPU[s] with ~48GB VRAM, or Apple silicon machine with 64GB+ RAM to run at decent speed.

Comments

[izard]

I was only using on Mac, and convert.py worked for me. probably some different python minor versions/llama.cpp commit tags.

[vak]

Right, it failed with python 3.12. So I tried 3.11, and it worked fine.