failed09/bashkir-roberta

BashkirRoBERTa

A masked language model for Bashkir. Given a sentence with one [MASK] token, it predicts the most probable missing Bashkir token from its context. The model is useful for fill-mask experiments, spellchecking, and as a foundation for further fine-tuning.

Examples

Input	Top prediction
Башҡортостан — беҙҙең [MASK].	республика
Мин башҡорт телен [MASK].	яратам
Өфө — ҙур [MASK].	ҡала
Бөгөн Өфөлә яңы [MASK] асылды.	мәктәп

Model Architecture

Property	Value
Task	Masked language modelling / fill-mask
Architecture	Pre-LayerNorm Transformer encoder
Transformer blocks	8
Hidden size / attention heads	640 / 10
Feed-forward size	2,560
Context window	256 subword tokens
Parameters	50.04M
Tokenizer	SentencePiece BPE, 16,384 tokens

The output embedding matrix is tied to the input word embeddings. Token IDs are fixed: <pad> 0, <unk> 1, <s> 2, </s> 3, [CLS] 4, [SEP] 5, and [MASK] 6.

Training and evaluation

The model was pretrained with dynamic masked-language modelling on a Bashkir text collection assembled from encyclopedic, periodical, and literary sources. The source texts are not distributed in this repository. On a held-out Bashkir encyclopedic evaluation set, the project reports 24.7% top-1 and 54.0% top-5 accuracy for masked subword prediction.

These are diagnostic MLM results, not a general-purpose language-understanding score. A mask may represent a whole word or a SentencePiece subword fragment.

PyTorch Loading (Transformers)

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "failed09/bashkir-roberta"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
logits = model(**inputs).logits
mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
prediction_id = logits[0, mask_index].argmax().item()
print(tokenizer.decode([prediction_id]))  # яратам

trust_remote_code=True is required: the model preserves the original custom Pre-LayerNorm architecture rather than using the stock post-LayerNorm RoBERTa implementation.

⚡ ONNX Runtime (Fast CPU & Edge Deployment)

For resource-constrained devices, edge environments, and production without heavy PyTorch dependencies, pre-compiled ONNX models are available in the onnx/ folder:

• onnx/model_fp16.onnx (95.4 MB): Recommended for GPU and DirectML acceleration.
• onnx/model_int8.onnx (58.1 MB): Quantized INT8 checkpoint for ultra-fast CPU, server, and mobile (Android/iOS) inference with negligible accuracy difference.

import numpy as np
import onnxruntime as ort
import sentencepiece as spm
from huggingface_hub import hf_hub_download

# Download lightweight INT8 model and tokenizer (~58 MB total)
model_path = hf_hub_download("failed09/bashkir-roberta", "onnx/model_int8.onnx")
sp_path = hf_hub_download("failed09/bashkir-roberta", "spm_bashkir_bert_16k.model")

session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sp = spm.SentencePieceProcessor(model_file=sp_path)

# Tokenize sentence with [MASK]
tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(".") + [3]
mask_idx = tokens.index(6)
inputs = {"input_ids": np.array([tokens], dtype=np.int64)}

logits = session.run(None, inputs)[0][0, mask_idx]
top_tokens = np.argsort(logits)[::-1][:5]
print([sp.decode([int(t)]) for t in top_tokens])  # ['яратам', 'беләм', 'өйрәнә', ...]

🎯 Target Applications

• Grammar & Spellchecking: Contextual candidate ranking, detection of morphological errors and vowel harmony violations.
• Cloze & Multiple-Choice Testing: Automated solving and candidate evaluation for Bashkir educational tests.
• OCR Post-Correction: Resolving noisy characters and ambiguous glyphs in digitized historical print.
• Feature Extraction & Fine-Tuning: Backbone representations for Bashkir text classification, sentiment analysis, and NER.

License and provenance

The checkpoint is released under custom terms (other on the Hub) while the source-rights audit is completed. No training texts are redistributed. For provenance or removal requests, please contact the maintainer through the Hub.

Citation

@model{failed09_bashkir_roberta_2026,
  title     = {BashkirRoBERTa},
  author    = {failed09},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/failed09/bashkir-roberta}
}