Sight Lab
Forecast accuracy & backtest results
Walk-forward evaluation over the last 30 days. 3,388 (account, horizon) predictions across 11 accounts.
sight-v2evaluated May 19, 2026
Methodology notes
- Synthetic data: NovaPay accounts are scripted with engineered behavioral profiles in
ml/scripts/accounts_config.py. Real-world performance would require backtesting on live treasury flows. - Stacker eval is in-sample: the Ridge meta-learner is currently trained and evaluated on the same 30-day holdout. Reported stacker MAPE is optimistic. Honest walk-forward evaluation (rolling-origin OOF) is on the roadmap.
Overall accuracy
94.8%
100 − avg MAPE
7-day MAPE
5.51%
median horizon
14-day MAPE
5.24%
longest horizon
Deficit detection F1
0.922
precision × recall
Per-horizon comparison
Sight ensemble vs. three baseline forecasters. Lower is better.
| Horizon | Sight | Naive | 7-day MA | Seasonal Naive |
|---|---|---|---|---|
| 1 day | 5.06% | 5.48% | 6.48% | 6.36% |
| 3 days | 4.98% | 11.29% | 7.43% | 6.35% |
| 7 days | 5.51% | 6.45% | 9.12% | 6.45% |
| 14 days | 5.24% | 9.23% | 9.97% | 6.51% |
Real vs predicted
Walk-forward backtest at 7-day horizon. Predictions made on each cutoff date, then compared against what actually happened.
ActualPredictedP10–P90
Deficit detection · confusion matrix
P 0.930 · R 0.914 · F1 0.922True Positive
775
8.0% of pairs
Correctly flagged dips
False Negative
73
0.8% of pairs
Missed dips
False Positive
58
0.6% of pairs
False alarms
True Negative
8796
90.7% of pairs
Correctly healthy
Holdout MAPE by model · lower is better
Stacked ensemble wins on every account in the holdout.
Impact summary
What the backtest implies for treasury operations across 30 days of walk-forward evaluation.
Overdraft days
balance below required minimum
Without Sight
848
With Sight
73
−91%
Implied capital cost
$35K per overdraft event
Without Sight
$29.7M
With Sight
$2.6M
−91%
Treasury hours
manual monitoring + response
Without Sight
342 h
With Sight
105 h
−69%
Pipeline
What's behind the numbers.
Ensemble members
- Prophetbase model
- LightGBMbase model
- ARIMAbase model
- Holt-Wintersbase model
- Chronosbase model
- Stacked EnsembleRidge meta-learner
Methodology
- Trained on 365 days of NovaPay transaction history.
- Walk-forward backtest: 30 cutoffs, predictions at horizons 1d / 3d / 7d / 14d.
- Per-account model selection on rolling-origin out-of-fold holdout (5 windows × 7d = 35 OOF pairs). Ridge stacker is one candidate alongside the five base models.
- Intervals via time-weighted split conformal with held-out coverage check; mean empirical coverage ≈ 76% against 80% target.
Version · sight-v2