mBERT vs MuRIL

Problem & motivation

Does a model that detects harmful Hinglish speech on one dataset still work on another? Hinglish mixes Hindi and English, and datasets differ in script, domain, and what they call harmful.

I compared mBERT and MuRIL to examine that change in setting. The central question is when a model’s apparent advantage survives a different label policy.

Technical approach

1. Prepare datasetsMinimal cleaning, deduplication, explicit label mappings
2. Train & comparemBERT / MuRIL; matched, transfer, and mixed conditions
3. Inspect errorsMacro F1, positive recall, confusion counts, probability diagnostics

Python, PyTorch, and Hugging Face support the training and evaluation pipeline. Matched experiments compare training and evaluation within a dataset; transfer experiments evaluate a trained checkpoint on another dataset. Mixed training is a separate condition. TF-IDF baselines provide a simpler reference.

Positive labels mean different things: Kaggle uses hate, CM uses offensive speech, and THAR uses targeted AntiReligion content. They should not be treated as interchangeable annotations of one task.

Dataset registry · Experiment setup.

Matched results across seeds

Macro F1 averages the F1 scores of both classes, so majority-class accuracy alone cannot hide poor positive-class performance. Values below are percentages: mean ± sample standard deviation for seeds 7, 13, and 42. These ranges are not confidence intervals.

Model ranking depends on the dataset. mBERT has the higher mean on Kaggle and CM; MuRIL has the higher mean on THAR. The close CM means should be read alongside their variation.

CM uses a fixed source split. Kaggle and THAR split membership varies with the seed, so their variation includes both training randomness and different evaluation examples. Evaluation splits were reused for selection; these are not untouched final-test results.

Multi-seed report · Per-seed data.

Transfer, errors & failed mixtures

The earlier seed-42 Kaggle-trained checkpoints transferred poorly. On CM, positive recall was 19.7% for mBERT and 14.3% for MuRIL. On THAR, it was 15.9% and 8.2%. These are single-checkpoint transfer results, separate from the multi-seed matched table.

Two MuRIL mixtures produced all-negative external predictions. In the Kaggle+CM diagnostic, the maximum positive probability was about 0.466—below the default 0.50 threshold. Lower thresholds improved scores on those same inspected splits, but that is a diagnostic, not independent validation of a fix.

Error analysis examines missed positives, false alarms, disagreement, and heuristic script/keyword groups. These groups guide further reading; they do not establish causes or replace human annotation. The noisy 79-row probe is excluded from primary conclusions.

Transfer results · Collapse diagnostic · Error analysis.

Limitations

Matched multi-seed results, mostly single-seed transfer results, and threshold diagnostics answer different questions. Reusing evaluation data for selection limits the strength of the reported scores. Label policies and source composition also complicate cross-dataset comparisons.

This is a research comparison, not a validated moderation service. The work does not establish that one model is universally better for Hinglish, and automated error categories are not independent human review.

What I learned & next steps

I learned to record label definitions and split policy beside every metric. Positive recall and saved predictions exposed failures that an overall accuracy score could conceal.

I would next separate model selection from independent evaluation, repeat the key transfer and mixed-training conditions across seeds, and have humans review the error categories before strengthening the conclusions.

Inspect & reproduce

Start with the per-seed CSV and aggregation report. Full training needs separately prepared datasets and compute; dataset terms apply independently.

The released checkpoint collection contains model artifacts and loading notes, not a hosted classification demo. The repository records AI coding and documentation assistance.