Shadow Mode: How to Prove ROI Before You're Ready to Commit
Most Shopify apps ask you to flip a switch and hope. Install, activate, and pray the numbers go up. If they don't? You've already been running for two weeks and your boss is asking questions.
Reclavio works differently. Shadow Mode lets you measure what would have happened before you commit a single dollar of margin. No offers issued. No discount codes minted. Just data—collected in the background—that shows you exactly what you'd recover, what you'd spend, and whether the math works for your store.
TL;DR: Shadow Mode runs the full decision pipeline without issuing offers. You see the opportunities. You evaluate the rules. You go live only when the data says it's worth it.
Author: Brodie, Founder @ Reclavio Status: Private beta (pre–Shopify App Store) Last updated: February 2026
Who This Is For
✅ Merchants evaluating Reclavio — test without risk before committing to offers ✅ Store owners who've been burned — by discount apps that trained customers to wait for coupons ✅ Data-driven teams — who want proof before action, not just dashboards with green arrows ❌ Not for: Stores looking for "set it and forget it" — Reclavio is a precision tool, not a blast cannon
The 60-Second Flow: What Shadow Mode Actually Does
Here's what happens when a shopper interacts with your store while Shadow Mode is active:
- Shopper types a message — "Is this available in size L?" or "Any discounts?"
- The full pipeline runs — intent classification, rule matching, exclusion checks, IPOE scoring
- The pipeline reaches a verdict — "Would have offered 10% off" or "No offer (cart below $100)"
- Nothing happens to the shopper — no widget popup, no discount code, no intervention
- The data is logged — cart value, session context, the offer that would have been issued, and why
You see every decision that would have been made. You see the blocked ones too.
What You See in the Dashboard
Shadow Mode populates three views that build your case for going live:
1. The Opportunities Card
This is the headline number: total at-risk cart value from sessions where Reclavio would have intervened.
"At-risk" means the shopper showed intent signals (asked about discounts, expressed price hesitation, or had an exit-like interaction) and your rules said they were eligible. It's not "all abandoned carts"—it's the subset where the pipeline would have acted.
Key details:
- Session-deduped — if a shopper updates their cart three times in one session, it counts once (at peak cart value), not three times
- Sparsity gating — fewer than 3 qualifying sessions? The card shows "Collecting signal…" instead of a misleading number
- Methodology visible — click the info icon to see exactly how the number is calculated
2. The Activity Log
Every "would-have" decision appears in your Activity Log with full context:
- What the shopper asked
- Which route the Conversation Engine selected
- Which rule matched (or why none did)
- Whether any exclusions fired
- What IPOE would have scored
This is the audit trail. You can search by session, filter by "would-have-offered" vs. "blocked," and drill into any individual decision.
3. Smart Blocks: The "No Offer" Breakdown
Not every session gets an offer—even in Shadow Mode. The Smart Blocks view shows you why decisions were blocked:
| Block Reason | What It Means |
|---|---|
| No matching rule | Cart didn't meet any rule conditions |
| All items excluded | Every item in cart was in an exclusion list |
| IPOE: negative EIP | Offer would have cost more in margin than it generated |
| Guardrail: G4 cooldown | Customer received an offer recently |
| Route: non-NEGOTIATION | Shopper asked a shipping question, not a price question |
This is where you tune. If 60% of blocks are "no matching rule" and those carts average $180, you know exactly which rule to add.
Holdout Groups: The Scientific Measurement
Shadow Mode tells you what could happen. Holdout groups tell you what did happen.
When you go live with Reclavio, a configurable percentage of sessions (0–50%) are randomly assigned to a holdout group. These shoppers never see Reclavio—no widget, no offers, no AI. They experience your store exactly as it was before.
The holdout group is your control. Treatment vs. control conversion rates, measured over time, tell you whether Reclavio is creating incremental revenue or just subsidizing purchases that would have happened anyway.
What the Holdout Comparison Card Shows
| Metric | What It Measures |
|---|---|
| Treatment conversion rate | Shoppers who saw Reclavio and converted |
| Holdout conversion rate | Shoppers who didn't see Reclavio and converted |
| Lift | The difference—your incremental value |
| Confidence interval | Normal-approximation 95% CI (z = 1.96) |
| Practical significance | Is the lift meaningful (>$0.50/session threshold)? |
| SRM check | Is the randomization working correctly? (Valid / Review / SRM Issue) |
The 7-Day Gate
The card won't show a verdict (green/amber/red) until at least 7 days of data have been collected. This prevents premature conclusions from lucky early data. We've all seen dashboards turn green on Day 2 and red by Day 10. The gate prevents that.
Exporting the Proof
One-click CSV export includes every data point behind the holdout comparison: session counts, conversion rates, CIs, and the raw numbers. Hand this to your CFO, your agency, or your ecommerce lead. The data speaks for itself.
The Playbook: Shadow → Live in Four Steps
Step 1: Run Shadow Mode for 1–2 Weeks
Install Reclavio, configure your rules, and leave IPOE in Shadow mode. No offers are issued. The Opportunities card fills up.
What to watch: Total at-risk value, session count, most common block reasons.
Step 2: Tune Your Rules
Use the Smart Blocks breakdown to adjust. Too many "no matching rule" blocks on high-value carts? Add a rule. Too many blocks on low-value carts? Good—your floor is working.
What to watch: Block reason distribution. You want intentional blocks, not misconfigured ones.
Step 3: Go Live with Holdout
Switch from Shadow to Live mode. Set a holdout percentage (we recommend starting at 10–20%). Reclavio starts issuing offers to the treatment group. The holdout group sees nothing.
What to watch: The Holdout Comparison Card. Wait for the 7-day gate.
Step 4: Read the Data and Decide
After 7+ days (or 500+ treatment exposures, whichever comes later):
- Green badge + positive lift → Reclavio is creating incremental value. Consider reducing the holdout to capture more revenue.
- Amber badge → Not enough data yet. Keep running.
- Red badge → Something isn't working. Check your rules, review Smart Blocks, and consider adjusting.
No lock-in. No guesswork. You decide based on your data.
FAQ
Q: Does Shadow Mode slow down my store? A: No. Shadow Mode runs the decision pipeline server-side without rendering any frontend components. No widget, no scripts, no impact on page load.
Q: Can shoppers tell I'm running Shadow Mode? A: No. The Reclavio widget doesn't appear during Shadow Mode. Shoppers experience your store exactly as before.
Q: How long should I run Shadow Mode? A: 1–2 weeks is typical. The goal is enough sessions to make the Opportunities card meaningful (minimum 3 qualifying sessions for it to display a number, but 50+ sessions gives you a reliable signal).
Q: What's the difference between Shadow Mode and IPOE Shadow Mode? A: Shadow Mode is the overall "observe without acting" state. IPOE has its own Shadow setting (Settings → IPOE → Mode) that lets the profit engine score offers without affecting delivery. You can run Shadow for everything or just for IPOE while using deterministic rules for offers.
Q: Can I switch between Shadow and Live without resetting data? A: Yes, with one caveat: changing the holdout percentage triggers an epoch reset (you'll see a confirmation dialog). This resets the holdout comparison data to prevent mixing apples and oranges. Your Shadow Mode Opportunities data is preserved.
📦 Free Download: Rule Templates Pack Get 6 ready-to-import recovery rules—Free Shipping Threshold, First-Time Buyer, Exit Intent, High-AOV Guardrail, Excluded Category, and Shadow Mode Starter—plus a testing checklist. Get the templates (free) →
What happens next: You'll get the pack + Shadow Mode checklist. Run Shadow Mode → see your Opportunities card → review candidates → adjust rules → go live when you're ready.
Shadow Mode isn't a demo. It's a commitment to letting you prove value before spending margin.
Every recovery tool asks you to trust the algorithm. Reclavio asks you to trust the data.
For the full decision pipeline, read The Four Engines: What Actually Decides Whether You Get a Discount. For the engineering architecture, read How We Built Profit-Aware Cart Recovery on Shopify. Have questions? Check the Help Center →.
References
How I verify claims: All Shadow Mode behaviors described here are verified against the codebase as of February 2026. Dashboard metrics use session-deduped calculations (SUM(MAX(cartValue) GROUP BY sessionId)). Holdout group lift is measured via a normal-approximation confidence interval (z = 1.96) on the per-session profit difference, using empirical variance from online moments. SRM detection uses chi-squared test with conservative thresholds.