Skip to main content
Search

Suvradip Paul

1 related articles

Subscribe to updates
  1. arXiv

    IndicBankBench tests 799 Indian retail-banking cases and finds eleven models reach only 43.7%–58.2% strict three-run reliability while at-least-once success runs 60%–74%

    The authors release IndicBankBench, a 799-case benchmark for Indian retail banking spanning five operational domains, a capability/refusal domain, and twenty primary axes, graded in four stages—safety, action and tool use, response adequacy, and advisory quality—with deterministic tool-use and most safety checks, an LLM judge only for semantic response adequacy, and a narrow resolver for ambiguous confirmation-before-write; running every case three times across eleven models yields strict pass3 of 43.7%–58.2% versus at-least-once success of 60%–74%, a gap of 10.8–21.4 percentage points.

You’re all caught up