150 GB Multilingual and Code Dataset for Central Asian AI Released After 10‑Hour OOM Battle
| Source: Mastodon | Original article
A developer assembled a 150 GB multilingual and code dataset for Central Asian AI, spending ten hours battling out‑of‑memory errors.
A developer has released a 150‑gigabyte open‑source dataset that combines multilingual text and source‑code samples aimed at boosting AI research for Central Asian languages. After weeks of crawling public repositories, scraping web forums and translating existing monolingual corpora, the creator uploaded the cleaned collection to Hugging Face, where it is now publicly available. The effort required extensive parsing and deduplication, and the final build process stalled for ten hours with out‑of‑memory errors before finally completing.
The dataset is notable for its breadth: it spans 49 languages and supports six common NLP tasks—topic classification, sentiment analysis, named‑entity recognition, question answering, natural‑language inference and summarisation—while also providing code snippets in several programming languages. By pairing language data with code, the resource targets “multilingual code” models that can understand and generate software instructions across linguistic boundaries, a capability that has been scarce for Central Asian tongues such as Kazakh, Uzbek and Kyrgyz.
Researchers and startups in the region see the release as a step toward more inclusive AI. Current large language models are heavily skewed toward high‑resource languages, limiting their usefulness for local developers, educators and enterprises. An open, sizable corpus lowers the barrier to pre‑training or fine‑tuning models that can handle regional dialects and programming tasks, potentially accelerating home‑grown solutions in finance, healthcare and public services.
The next watch points include early adopters’ experiments with the dataset, especially any fine‑tuned models that demonstrate measurable gains on the six benchmark tasks. Community feedback on the data‑building tool “dataset_build” will also shape future releases, and the creator has hinted at expanding the collection with more languages and larger code bases later this year.
Sources
Back to AIPULSEN