Bilibili Releases Open Small Language Models with Chinese and English Training
Bilibili has released a series of open small language models called Index-1.9B, trained on a massive dataset of Chinese and English tokens. These models are designed for various applications, from general use to specialized tasks like chat and character interaction.
Bilibili released Index-1.9B, a series of open small language models. The series includes four models: a base model with 1.9 billion non-embedding parameters, a control variant trained without any instruction-like data, a chat model fine-tuned for conversational use, and a character model for specialized interactions. These models were pre-trained on 2.8 trillion tokens, predominantly in Chinese and English.
This release matters because it provides open, accessible language models that can be used for a variety of tasks. For example, the chat model can be used to build conversational AI, while the character model could be used in gaming or virtual assistant applications. The control variant ensures that researchers can study the effects of instruction data on model behavior, and it can be used in environments where instruction data might be sensitive or inappropriate.
If you're interested in trying these models, you can read the technical report on arXiv linked in the source. The paper details the training recipes, alignment methods, and model specifications for the Index-1.9B series. These models are particularly useful for developers looking to build AI applications with a focus on Chinese and English languages.