quant-integrity

Statistical tests for whether a backtest means anything.

This library will not find you a profitable strategy. Its function is to reject them. Everything here exists to answer one question: given how many configurations you tried before reporting this one, does the result survive?

pip install quant-integrity

---

The problem, in one picture

A moving-average crossover on BTC-USD hourly bars. After sweeping parameters, the best configuration returned an annualised Sharpe of 0.95 — and a Probabilistic Sharpe of 0.86, which reads as "probably beats zero".

That is where most backtests stop. Here is the same result next to the bar that 28 attempts actually sets:

The best of 28 tries is large even when every underlying edge is exactly zero. Against that bar the deflated Sharpe is 0.157. The observed result is not marginal; it is below half of what luck alone produces.

A high PSR with a low DSR is the signature of an overfit search, not a borderline edge.

What the search was actually selecting on

The cause was visible in the trials themselves. Sorting the 28 recorded runs by trade count against Sharpe:

Every configuration that traded less looked better, at ρ = −0.96. That is not a signal being discovered — it is cost drag being measured, plus a slow moving average approximating buy-and-hold in a rising market. The search was not finding an edge. It was finding the configuration that traded least.

And the interval was never tight to begin with

Reported alone, Sharpe = 0.95 implies a precision the data does not support.

Every number above is reproducible

git clone https://github.com/abhayjnayakk/quant-integrity
cd quant-integrity && pip install -e .
python docs/make_figures.py        # regenerates all three figures and prints the values

Trials evaluated	28
Sharpe (annualised, after costs)	0.9500
Probabilistic Sharpe (PSR)	0.8556
E[max Sharpe] under the null, 28 trials	1.8525
Deflated Sharpe (DSR)	0.1568
Bootstrap 95% CI on Sharpe	[−0.93, 2.69] — straddles zero
Excess kurtosis	18.61
Spearman ρ(Sharpe, trades)	−0.9618 (p = 3.8e−16)
Minimum backtest length	1.94 years required, 1.25 available

The trial count is taken from the recorded runs rather than typed in. An earlier version of this page hardcoded 29 while the file recorded 28, and quoted a deflated Sharpe from one and a minimum backtest length from the other.

The figures and this table are generated by the same script from examples/btc_ma_runs.npz (24 KB, the actual return series and per-trial results). No server and no account. There is also a longer walkthrough in examples/btc_ma_postmortem.ipynb.

On one number that is not in this repo. The same specification was later run across a basket rather than a single asset, and pooled to a Sharpe of −1.376 over 10 assets with 3 of them positive. That result lives in the hosted server's ledger, not in this repository — the example file here carries only the single-asset trials, so you cannot reproduce the basket figure from it. It is quoted here because it is the strongest part of the story, and flagged because a number you cannot check should always be labelled as one.

---

What's in it

Function	Question it answers
deflated_sharpe(returns, n_trials)	Does this Sharpe beat what the best of N trials produces by luck?
probability_backtest_overfitting(matrix)	Does your selection procedure carry information at all?
combinatorial_purged_cv(...)	Cross-validation splits with purging AND embargo
stationary_bootstrap_ci(returns)	How wide is the interval really?
reality_check(family, benchmark)	Is the family's best member better than the benchmark it was chosen over?
effective_tests(matrix)	How many INDEPENDENT tests does a correlated basket give?
min_backtest_length(n_trials, ...)	Is your sample even long enough for the search you ran?
triple_barrier_labels(...), dollar_bars(...)	Labelling and activity-based sampling
spec_hash(spec)	Content-addressed strategy identity
verify_attestation(record, key)	Independently check a signed research record

import quant_integrity as qi

out = qi.deflated_sharpe(returns, n_trials=28)
print(out["dsr"], out["psr"])

n_trials means every configuration you evaluated, including the ones you discarded. Understating it produces a flattering answer. Nothing in this library can check it — which is exactly the problem the next section is about.

---

Commercial intent, stated on day one

The statistics in this repository are AGPL-3.0 and free forever. Every method here is published academic work. There is no moat in the arithmetic and no version of this library is crippled to sell you an upgrade — a teaser would fail as a funnel and would forfeit the only thing an integrity tool has.

Hosted attestation is a paid service. Verification is public and lives here. Issuance is not, for the reason below. Saying this up front so nobody can later claim a rug-pull.

Why self-hosting can't attest

A trial count is only meaningful if it cannot be revised downward. But if you run your own ledger, you can edit it. Not through malice, usually — through the ordinary temptation to restart the count after a rewrite, or to not record the sweep that went nowhere. A self-attested integrity record certifies nothing, because the person attesting is the person who benefits.

Verification is public. Issuance is the service.

Anyone can check an attestation with this library and an issuer's public key. No cooperation from the issuer is required, and none of the checking code is withheld. What you cannot do is make one, because a signature is only worth something when the signer is not the beneficiary.

The property that does the work is not any single signature — it is the chain. verify_chain confirms that, across a sequence of attestations:

• sequence numbers are consecutive, so no record was removed from the middle;
• each references the previous record's digest, so none was altered afterwards;
• n_trials never decreases.

An outside party can establish all of that without ever seeing the ledger. Someone who edits their own records cannot reproduce it, because they would have to re-sign every subsequent record with a key they do not hold.

That is the whole business: not the arithmetic, which is here and free, but operating an instance that has no stake in the answer. That instance is quant.avasis.ai — an MCP server, so the research runs inside Claude, Cursor, VS Code or any MCP client, and the trial count is kept by the server rather than by the conversation.

---

Scope

This is a statistical instrument. Deliberately absent, and staying absent:

• No buy, sell or hold signals. No target prices, position sizes or stop levels.
• No model portfolios or allocations.
• No claims about returns, and no examples implying any.

The library takes return series and trial counts and returns statistics about them. It does not know what you are trading and does not offer an opinion on it.

---

Install and contribute

pip install quant-integrity                 # statistics
pip install "quant-integrity[attestation]"  # + signature verification

Python 3.10+. Depends on numpy, scipy and pandas. Fully typed (py.typed).

Contributions require a CLA and a DCO sign-off — see CONTRIBUTING.md, which explains why without apology.

Licence

AGPL-3.0-or-later. See LICENSE and NOTICE.

The network-use clause is deliberate: it means a competitor cannot run a closed hosted fork of this code. It does not restrict you from using the library in your own research, hosted or otherwise, without publishing anything.