There's a claim circulating in the NinjaTrader community that Strategy Analyzer backtests “can't be trusted” and that Market Replay is the only valid way to test a strategy. It sounds reasonable on the surface — Market Replay uses real tick data, so it must be more accurate, right?
Not exactly. And if you're making strategy decisions based on this belief, you might be using the wrong tool for the job.
I wrote a detailed breakdown of how NinjaTrader's testing methods actually work, when each one is appropriate, and why Strategy Analyzer is actually more reliable than Market Replay for most trading strategies — not less.
The Five Levels of Strategy Validation
Not all testing methods are created equal. Here's the hierarchy, ranked from most to least reliable:
| Level | Method | Strength |
|---|---|---|
| 1 | Live trading (real money) | The ultimate test |
| 2 | Live sim (Playback connection) | Real-time execution, no risk |
| 3 | Strategy Analyzer | Statistical power across years of data |
| 4 | Market Replay | Tick-perfect single-path replay |
| 5 | Chart overlay / visual inspection | Subjective, no fill simulation |
Yes — Strategy Analyzer ranks above Market Replay. That's intentional, and the article explains exactly why.
The OHLC Ambiguity Problem (And When It Doesn't Matter)
The main argument against Strategy Analyzer is that it doesn't know the intra-bar tick sequence within each candle. It only sees Open, High, Low, Close — so it has to guess which came first, the high or the low.
This is a real limitation. But it only matters under specific conditions:
- Your strategy uses OnEachTick or OnPriceChange calculation mode
- You're using limit orders or tight stop losses that could trigger at the high or low of a bar
- You're trading on very small timeframes (1-minute or less) where intra-bar sequence matters
If your strategy uses OnBarClose calculation on 5-minute or larger bars with market orders, the OHLC ambiguity is essentially irrelevant. The entry and exit prices are determined by bar closes — which are exact values, not estimates.
Why Market Replay Isn't the Gold Standard
Market Replay has its own serious limitations that rarely get discussed:
- Speed problem — Running Market Replay at realistic speed makes statistical analysis impractical. Testing one year of data takes… one year.
- High-speed decimation — Running at 10x or faster strips out 80-90% of ticks, destroying the precision that was supposed to be the whole point.
- Market impact — The historical tick stream doesn't include your orders. Your fills would have changed the order book.
- n=1 — Market Replay gives you exactly one data point. Strategy Analyzer gives you thousands of trades across years of varying conditions.
Read the Full Analysis
The complete article covers:
- The decision matrix — which testing method to use based on your strategy's specific architecture
- OnBarClose vs. OnEachTick — why calculation mode is the key factor, not the testing method
- When Market Replay actually matters — the specific scenarios where tick-level precision is important
- The confirmation workflow — how to use both tools together for maximum confidence
- Why “more data points” beats “more precision” for strategy validation
