At last: Russian LLM fine-tune

[izard]

For a long time, there was no decent local LLM that understood and spoke Russian well. I've tested many fine-tunes, and they all behaved significantly dumber when answering in Russian than their respective English originals.

Russian internet has a huge potential training corpus second only to English; but smart French, German, Spanish and Arabic fine-tunes for local LLMs existed for months now, and there was no Russian one.

Until Serge Gotsuliak released his finetune few days ago.

Highly recommend it, works great with llama.cpp. I was missing RAG on my Russian books, will set it up now.

There is a drawback: 4-bit quant that I tested needs GPU[s] with ~48GB VRAM, or Apple silicon machine with 64GB+ RAM to run at decent speed.

Comments

[vak]

Wow, sounds promising. I never tried llama.cpp - I hope it's not so complicated.

[izard]

It is very minimalistic, well designed and has 0 dependencies.

[juan_gandhi]

I kind of missed it. You mean, one can install it on one's local machine, if it is M2 with 64GB RAM?

[izard]

That’s right. Also llama.cpp does not have any dependencies so compiles /installs quickly and easily.

[vak]

It worked pretty well on Linux: https://vak.dreamwidth.org/1184101.html

On Mac it failed though. When converting the model into gguf:

  File "llama.cpp/convert.py", line 793, in load
    fp = self.zip_file.open(info)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/zipfile/__init__.py", line 1643, in open
    raise BadZipFile(f"Overlapped entries: {zinfo.orig_filename!r} (possible zip bomb)")
zipfile.BadZipFile: Overlapped entries: 'pytorch_model-00004-of-00015/data/7' (possible zip bomb)

[izard]

I was only using on Mac, and convert.py worked for me. probably some different python minor versions/llama.cpp commit tags.

[vak]

Right, it failed with python 3.12. So I tried 3.11, and it worked fine.