A team of researchers has unveiled the first systematic effort to generate audio descriptions in Hindi, addressing a significant accessibility gap in Indian cinema and streaming media. The work represents a crucial step toward making visual content available to blind and low-vision audiences across India's diverse linguistic landscape.

According to arXiv, the researchers introduced Andha-Dhun, a dataset of human-authored Hindi audio descriptions sourced from eight full-length movies. This collection marks the first time audio descriptions for any Indian language have been studied at scale, filling a void that has existed as regulatory bodies like India's Central Board of Film Certification increasingly mandate descriptive content in regional languages.

The Translation Problem

The research team explored two distinct approaches to generating Hindi audio descriptions. The first involved creating descriptions directly from dense English video captions, while the second relied on translating existing English audio descriptions into Hindi. What they discovered challenges conventional assumptions about how language technology should handle accessibility content.

Simple machine translation approaches proved inadequate. The researchers found that direct translation from English audio descriptions introduced cultural artifacts and reduced the diversity of the final content compared to descriptions originally authored in Hindi. English-to-Hindi machine translation also failed to properly adapt cultural references, while human-translated versions performed better but still fell short of native descriptions.

Why Direct Translation Fails

The findings highlight a fundamental principle often overlooked in AI localization work: accessibility content serves a specific audience with distinct needs. Rather than treating audio descriptions as mere text to be translated with fidelity to the source, the research demonstrates that effective descriptions must account for the cultural context and lived experience of Indian blind and low-vision viewers.

  • Naive translation narrows the range of descriptive choices available to audio describers
  • Cultural references embedded in English descriptions do not transfer meaningfully to Hindi-speaking audiences
  • Original Hindi descriptions capture nuances and context that translation-based approaches miss

The team evaluated their approaches using two metrics: perplexity to assess language fluency, and an LLM-as-a-judge framework to measure overall quality. This dual evaluation acknowledges that accessible content must be both grammatically sound and genuinely useful to its intended audience.

Implications for AI Localization

The research carries broader implications for how artificial intelligence systems approach language-specific accessibility features. As regulatory mandates expand across India and other multilingual regions, the templated approach of translating English-language AI systems into other languages may prove insufficient.

Rather than treating non-English accessibility as a downstream localization problem, the findings suggest that AI systems serving blind and low-vision audiences in different linguistic communities require purpose-built datasets, models, and evaluation frameworks. This approach demands investment in original content creation rather than reliance on translation pipelines.

With the release of the Andha-Dhun dataset and their analysis of generation methods, the researchers have established a foundation for future work in Hindi audio descriptions and potentially other Indian languages. Their conclusion carries a clear message for technology developers: accessibility innovation must prioritize the needs of the target audience over source-language fidelity.