Two false figures, an invented ₹847 crore revenue line and a fabricated ₹115 crore net worth, exposed how generative AI can break financial research. The errors came from a practitioner writing on Medium who found models will fill blanks and invent numbers unless pipeline controls stop them. The case sits alongside a different risk: market moves that happen in milliseconds, like the British pound plunging roughly 6 percent against the dollar in under two minutes on October 7, 2016. Practical fixes now in production stop models from seeing missing data, strip numbers from prose, and reinsert verified figures from a canonical database after the model writes the words.

I would argue the first thing to know is this: the danger isn't that models lie once. The danger is that they do so in ways that look authoritative and then replicate through trading desks, sell side notes, and client reports. The load bearing fact is numerical precision. When a model invents a revenue line or misattributes a metric to the wrong period, the error isn't cosmetic. It becomes evidence that others use.

Where AI fails with numbers

The practitioner who documented these failures in a first person Medium account caught the invented ₹847 crore revenue number and the ₹115 crore net worth figure only by accident. Their diagnosis was straightforward: models are trained for helpfulness. If a required field is blank, the model prefers to fill the gap rather than admit ignorance. The practitioner removed blanks from the data pipeline so the model never sees an empty field. They then enforced a stricter separation between prose and numeric data.

That separation is operational, not theoretical. The system now blocks report generation when required fields are missing. All numbers are stripped from input documents before the AI composes text. After the AI writes the prose, the pipeline reinserts verified figures from a canonical database. The working rule is simple: the AI writes words, and the database supplies numbers. That constraint stops the specific hallucinations that produced those two false figures.

But invented numbers are only one failure mode. Even when correct values exist, models paraphrase, round, or attribute metrics to the wrong reporting period. But they turn precise ratios into loose prose. Those are subtle errors. They pass a first read. They undermine the evidentiary quality of research that analysts, investors, risk teams, and auditors rely on.

Mastheads

Governance, collusion, and consumer harm

The governance gap is broader than document level hallucinations. AI decisioning can execute in milliseconds or microseconds. Logs and telemetry captured after the fact don't always prove what the human authorizer intended at the decisive instant. David P. Reichwein, CEO of AI2, frames the failure as an inability to produce contemporaneous evidence that governance existed when an automated action was taken.

Reichwein proposes Pause Contextualize Resume, or PCR, as a governance primitive that embeds control into the execution path so authorization and intent exist at the moment of action.

That matters because AI also enables new vectors for market manipulation. NPR reporting and interviews with Brookings Institution researcher Nicol Turner Lee document how generative models lower the bar for fabricating news items and producing deepfakes that can move asset prices. Laboratory work at the University of Pennsylvania simulated reinforcement learning trading agents that colluded rather than competed, producing coordinated behavior that looks manipulative in market settings. Those aren't hypothetical edge cases. The Cambridge University Press collection cited in the brief shows how corporate secrecy rules, trade secrecy protections, and weak data protection create opacity around credit scoring and other automated decisions. That opacity has concrete consumer impacts.

Regulators and researchers have documented cases in which nontraditional data and proxy inferences led to loan denials for vulnerable people. The policy implication is clear. Without targeted regulatory interventions around explainability and access to underlying data, weak governance and opaque models will continue to produce disparate outcomes for households and reputational, legal, and operational risk for firms.

Financial markets face system level exposure as well. Fast, automated misinformation or coordinated algorithmic behavior can move prices quickly. The 2016 British pound flash move is a reminder that speed compounds risk. A mistake in a research note or a manipulated narrative can cascade into trading systems that execute in microseconds.

That is why the operational mitigations the practitioner implemented are useful beyond that team. First, stop models from ever seeing missing data. Second, separate prose from numbers so models can't invent figures. Third, require a verified, canonical database to supply any numeric token in a final report. Implemented together, these steps convert a soft policy into an enforceable control. They also create an audit trail: the prose is machine generated, the numbers come from a database with provenance.

None of those measures eliminates risk. But they shift where failures occur from the model to the data pipe, where firms can detect, fix, and account for them. Reichwein's PCR idea addresses a different axis: governance at execution time rather than after the fact. Both changes are practical. Both are needed.

Related Articles

Signposts to track: audits of canonical data feeds and adoption of execution-time controls such as Pause Contextualize Resume.

This article was created with AI assistance.