InnoWaveTec

Blog

  • Backtest Overfitting: A Practical Checklist for Strategy Research

    Backtest Overfitting: A Practical Checklist for Strategy Research

    Every quantitative researcher has produced a backtest with a Sharpe ratio above 3 that turned out to be worthless. The uncomfortable part is that the backtest was not wrong — the code ran correctly, the data was clean, the equity curve was real. The problem is that the number reported was never an estimate of future performance. It was an estimate of how hard the researcher had searched.

    Selection bias has a closed form

    The core result is simple enough to be uncomfortable. If you test N independent strategies that all have zero true edge, the maximum observed Sharpe ratio across those trials grows roughly with the square root of the logarithm of N. Test twenty variations of a moving-average crossover and the best one will look decent. Test two thousand and the best one will look extraordinary.

    This means the headline Sharpe of a strategy is uninterpretable without knowing the size of the search that produced it. A Sharpe of 1.8 from a single pre-registered hypothesis and a Sharpe of 1.8 from a parameter sweep over ten thousand configurations are entirely different claims about the world. Almost no research process tracks the second number.

    A practical checklist

    Formal deflation methods exist and are worth using, but the majority of the damage in real research pipelines comes from a handful of concrete, checkable mistakes.

    1. Count your trials and write the number down. Including the ones you abandoned, including the ones from last month on the same dataset. If you cannot state N, you cannot interpret your result.
    2. Audit for look-ahead in your data, not your code. Point-in-time correctness is the issue: restated fundamentals, survivorship-filtered universes, index membership applied retroactively. Your code can be perfect and still see the future because the vendor’s database does.
    3. Model costs before you model alpha. Half-spread, market impact scaled to your participation rate, borrow cost, and a realistic fill assumption. A strategy that is only profitable at the mid price is not a strategy.
    4. Check turnover against the holding period you claim. A signal with a one-day decay half-life and a stated monthly rebalance is measuring something other than what you think.
    5. Use combinatorial cross-validation rather than a single hold-out. One out-of-sample period is one draw. Once you have looked at it twice, it is in-sample.

    The hold-out set problem

    Everyone knows to reserve an out-of-sample period. Far fewer research processes actually protect it. The failure mode is gradual: a strategy fails out-of-sample, the researcher adjusts a filter and re-runs, and the hold-out silently becomes a second training set. After a dozen iterations there is no untouched data left, and no record that this happened.

    Out-of-sample data is a consumable resource. Every time you look at it, you spend some. Budget it like capital.

    The organisational fix is more effective than any statistical one: log every backtest run automatically, with its parameters and its result, to a store the researcher cannot edit. This makes the trial count observable and it makes hold-out consumption visible. Teams that do this find their reported Sharpe ratios drop noticeably — and their live-versus-backtest tracking error improves by more than the drop.

    Priors beat statistics

    The most reliable filter is not a test statistic. It is the question of whether there is a plausible economic reason for the effect to exist and to persist: someone with a non-economic motive on the other side of the trade, a structural constraint forcing a flow, a risk genuinely being borne. A strategy with a coherent story and a modest Sharpe survives contact with live markets far more often than a spectacular one with no explanation.

    That is not a rigorous criterion, and it will occasionally cause you to discard something real. It will more often stop you from allocating capital to a pattern that exists only in the shape of your search.

  • The Plumbing Nobody Sees: What Shortened Settlement Really Changed

    The Plumbing Nobody Sees: What Shortened Settlement Really Changed

    Settlement cycles are the least glamorous subject in capital markets. They are also the constraint that quietly determines how much cash a firm must hold idle, how much counterparty risk sits on the books overnight, and how many people are awake at 2 a.m. reconciling breaks. When the North American markets moved from T+2 to T+1, most retail investors noticed nothing. Operations teams rebuilt half their workflow.

    Why a single day matters so much

    The intuition that “one day faster” is a marginal change misses what the day was being used for. Under T+2, the gap between trade and settlement absorbed an enormous amount of slack: allocation errors got caught, FX for cross-border purchases got arranged, securities lending recalls got processed, and mismatched standing settlement instructions got repaired. Compressing to T+1 does not remove that work. It removes the buffer in which the work happened.

    The effect is asymmetric. A domestic institution trading domestic equities absorbs the change with modest process tightening. A European asset manager buying US equities faces something closer to a structural problem: the FX to fund the purchase now has to be executed and settled in a window that may not overlap with the liquidity hours of the currency pair involved.

    Where the pressure actually lands

    • Affirmation deadlines. Trade allocation and affirmation must complete on trade date. Batch processes that ran overnight now have to run intraday, which usually means they have to be rewritten rather than rescheduled.
    • Funding and FX. Pre-funding requirements rise. Some managers hold a standing currency buffer rather than trying to execute FX inside the compressed window — a real cost that shows up as drag on returns.
    • Securities lending. Recalls that used to have a comfortable margin now need to be issued near-immediately, which raises the operational cost of running a lending programme at all.
    • Fails management. Less time to repair a break means more breaks reach settlement. Fail rates ticked up initially in every market that has made this transition.

    The automation forcing function

    What makes shortened settlement interesting as a case study is that it works as a mandate for automation that no internal business case could have produced. For years, operations leaders have argued for replacing manual affirmation and email-based instruction repair. The projects lost every budget cycle to something revenue-facing. A regulatory deadline does what a cost-benefit analysis could not.

    The measurable outcomes have been consistent across markets: same-day affirmation rates climbing well above 90%, standing settlement instruction databases finally getting cleaned, and a meaningful reduction in the manual touch rate per trade. The initial spike in fails typically normalises within a couple of quarters, and post-transition fail rates often end up below the pre-transition baseline — because the automation that the deadline forced was genuinely better than the manual process it replaced.

    What comes next

    The obvious question is whether T+0 — atomic settlement — is the destination. Technically it is achievable; several tokenised settlement pilots have demonstrated it. Economically it is far less clearly desirable. Netting is the reason. Under a T+1 regime with end-of-day netting, a broker settling ten thousand offsetting trades moves a small fraction of the gross notional. Under true atomic settlement, every trade is funded gross, and the intraday liquidity requirement explodes.

    Faster settlement reduces counterparty risk and increases liquidity risk. There is no cycle length that minimises both, only a choice about which one you would rather manage.

    The likely equilibrium is not T+0 everywhere but T+0 as an option for participants who want it and can fund it, sitting alongside a netted default cycle for everyone else. That is a messier answer than a headline reform, which is usually a sign it is the right one.

  • Edge Inference Is Quietly Reshaping Application Architecture

    Edge Inference Is Quietly Reshaping Application Architecture

    For most of the last decade, the default answer to “where should this run?” was simple: the cloud. Push the request to a region, let a large machine handle it, send the bytes back. That answer is starting to fray — not because the cloud got worse, but because the economics of small models got dramatically better.

    The latency budget is where it starts

    A round trip from a device in Southeast Asia to a data centre in Northern Virginia costs somewhere between 180 and 260 milliseconds before a single token is generated. Add model queueing and you are comfortably past the threshold where an interaction stops feeling instant. Users do not read latency numbers; they just decide the feature is sluggish and stop using it.

    Edge inference collapses that budget. When a 3B-parameter model runs on the device or in a nearby point of presence, the network cost approaches zero and the entire experience becomes bounded by compute you actually control. The trade is capability: a small model will not match a frontier model on hard reasoning. The practical insight of the last two years is that a surprising share of production workloads never needed frontier reasoning in the first place.

    Classify locally, escalate remotely

    The architecture that keeps emerging in production systems is a two-tier router. A small local model handles the high-volume, low-ambiguity path — intent classification, entity extraction, redaction, formatting, retrieval query rewriting. Anything it flags as uncertain gets escalated to a larger hosted model.

    • Tier 1 (local): deterministic transforms, routing, PII scrubbing, cache lookups. Target p99 under 40 ms.
    • Tier 2 (regional): summarisation, multi-document synthesis, anything with a long context window.
    • Tier 3 (frontier): genuine reasoning, code generation, tasks where an error is expensive.

    Teams that have measured this report escalation rates between 8% and 20% depending on domain. That is an order-of-magnitude reduction in tokens billed at the top tier, and it arrives without a visible quality regression — because the escalation gate, not the small model, is what determines quality.

    The gate is the hard part

    Everyone underestimates this. Building a router that knows when it does not know is a harder engineering problem than either model it sits between. Naive confidence thresholds on output logits are poorly calibrated and drift as inputs shift. The approaches that hold up in production tend to be boring:

    1. Escalate on structural failure — malformed output, schema violation, empty extraction — rather than on a soft confidence score.
    2. Sample a fixed percentage of local-tier traffic to the frontier model regardless of confidence, and use the disagreement rate as a live quality signal.
    3. Keep a per-tenant override. There is always one customer whose data breaks your assumptions.

    What this means for how you build

    The operational implication is that model selection stops being an architectural decision made once and becomes a runtime decision made per request. That pushes real requirements into your infrastructure: versioned model artefacts shipped to the edge, a way to roll back a bad small-model deployment in minutes, and observability that tells you the escalation rate by route rather than just aggregate latency.

    The teams that win here are not the ones with the best model. They are the ones who can measure, per endpoint, what fraction of their traffic actually needed the expensive path.

    None of this is exotic. It is caching and tiering — the same pattern that reshaped web serving in the 2000s and CDNs in the 2010s, applied to a new kind of compute. The surprising part is how long the industry spent treating inference as an indivisible remote call before noticing.