How raw NSE Bhavcopy beat premium vendors at survivorship-free data
If you backtest Indian equities on a dataset assembled from a convenient API, your results are probably wrong in a direction that flatters you. Not slightly. Structurally.
This post is about why, and about rebuilding the data so it isn’t. The source turned out to be free and public, which is the mildly annoying part.
The bias nobody sees, because it is invisible by construction
Start with the obvious approach: take today’s index constituents, pull their price history, run your backtest.
Two things just went wrong.
Survivorship. Your universe contains only companies that still exist and still trade. Every firm that went bankrupt, got delisted, or collapsed to nothing is silently absent — not marked missing, absent, as though it had never listed. Any strategy that would have bought those names is measured as though it never could have. Buy-the-losers strategies are hit hardest, because the losers are exactly the names that vanish.
Universe look-ahead. Applying today’s index membership to history means your 2018 backtest is trading the companies that would go on to be successful enough to still be in the index in 2026. You have handed your strategy an eight- year-old newspaper.
Neither shows up as an error. There is no null, no warning, no gap. The backtest runs clean and returns a number that is too high, and you have no way to know by how much.
I started on a vendor dataset with both problems: about a million rows, ten years, and the current constituents of a broad index, pulled from a free API. It was fine for building the engine and useless for estimating an edge, and I labelled every output from it accordingly.
The fix is the exchange’s own daily file
NSE publishes a Bhavcopy every trading day: a plain file containing every security that traded that session, with OHLCV. It has been doing this for decades and the archive is public.
The insight is almost too simple. If you assemble history forward from daily files rather than backward from today’s constituents, both biases disappear at once. A company that traded in 2017 and collapsed in 2019 is in the 2017 files, because it traded in 2017. Nothing has to be reconstructed, because nothing was ever removed.
The rebuilt master came to roughly 2.5 million rows across about 2,550 symbols and ten years, at zero cost.
How to know it worked: you check for ghosts. The test is not “does the data load” but “are the dead present.” I wrote acceptance tests asserting that specific collapsed companies appear by name — DHFL, Jet Airways, Reliance Capital, RCom, RPower, Yes Bank, Sintex. Those tests failed by design against the old vendor dataset and only went green on the rebuilt one.
Write the test that fails on your current data first. Otherwise you have written a test that asserts the status quo.
The part that will actually hurt you: corporate actions
Raw Bhavcopy is unadjusted. This is where a free dataset stops being free and starts costing you a month.
When a stock splits 1:10, the raw close drops from ₹2,000 to ₹200 overnight. Nothing happened. Every shareholder is exactly as wealthy as before. But your data now contains a −90% daily return, and every indicator computed across that date is corrupted — momentum, volatility, drawdown, all of it.
A scan of my raw archive found 220 such cliffs. Each one needs a factor, and each factor needs verifying, because a wrong adjustment is worse than no adjustment: it corrupts silently, in a direction you can’t predict, and every downstream result inherits it.
The acceptance-test discipline matters more here than anywhere, and the crucial tests are the negative controls. It is easy to write a test saying “after adjustment, the HDFC Bank split is smooth.” It is far more important to write one saying Yes Bank’s collapse must survive adjustment. That was a real −80% move. Real money was really lost. If your CA pipeline “fixes” it, you have not cleaned your data, you have deleted history — and you will never notice, because clean-looking data is what you were hoping for.
Curation ended at 179 verified factors and 84 structural breaks. Demergers are handled as breaks rather than adjustments, because the pre- and post-demerger entities are genuinely different companies and pretending otherwise is a fiction.
The acceptance tests found a hole in my own calendar
Here is the part I did not expect.
The corporate-action detector flagged three symbols cliffing on the same date with non-canonical ratios. Three simultaneous unrelated corporate actions is not a thing that happens. So either the detector was broken, or the data was.
The data was. My archive had an eight-and-a-half-month hole — a stretch of 2022 where the original scrape had silently failed. The two dates either side of the gap sat adjacent in the file, so any calculation spanning them was computing a “one-day return” across eight months.
The evidence had been sitting in plain sight the whole time. The dataset documented its own row count and its own date range, and dividing one by the other implies about 170 missing sessions. Nobody had divided.
Two changes followed. Gaps became first-class data — recorded in their own file, with the rule that no lookback or indicator may span one. And diagnostics stopped assuming that consecutive rows are consecutive days.
The lesson generalises past this project: acceptance tests written against real data find the failure you did not think to register. That suite was built to catch bad corporate-action factors. It caught the dataset instead.
What this costs you, honestly
Roughly a month, if you are careful. Most of it is corporate actions, and most of that is verification rather than code.
You get: a decade of survivorship-free daily history, point-in-time universes that don’t peek, and — the part that turns out to matter most — a reason to believe your own backtests. After this, when a strategy fails, you know it failed on the market rather than on your data. Nineteen of the twenty-three hypotheses on my Ledger are dead, and I can be confident they are dead for real reasons.
Two things I’d say plainly. First, the exchange’s raw files beat the convenient API not because they are better data but because they are unprocessed data, and the processing is where the bias entered. Second, this is not clever. It is just the boring version, done properly, and almost nobody does it — which is exactly why the flattering numbers are so common.