Pausing Tests
Last updated
Last updated
For any active test, you can choose to pause your test at any time and resume it later on. This is particularly handy, for example, if you noticed that you made a mistake when editing your test variant and need to correct it, but don't want to set up a brand new test from scratch.
If you pause a test, visitors to your website will immediately stop seeing the tested experiences. Data collection will also pause, and paused periods of time will be represented in performance over time charts on the test report page with plots for when a test was paused, when it was resumed, and if it was paused and resumed in the same day.
Days plotted in the performance over time chart during which a test was paused will be represented with a "partial data" tag in the tooltip when hovering over that point on the chart.
While we would always recommend starting a new test if you plan on making changes to your tested variants (because changes can invalidated your collected data), you can always determine for yourself when a test should be re-run.
In Shoplift, tests are paused when there are changes to your store’s templates or themes that make it impossible to run the test as designed. There are two common reasons for this:
1. Missing Templates in a Published Theme
What happens: If you're running a test on a specific template (like a product page or collection page), and you publish a new theme that doesn't include that template, the test will be paused.
Why: The test uses the template to display different versions to visitors. If the template no longer exists, visitors can't access it, making the test invalid.
How to fix it: Add the missing template to your live theme, being sure to maintain the correct template name (i.e. "sl-12345"). For detailed information on adding a template to your theme, see Theme Compatibility.
How to avoid it: Ensure that any new themes you publish include all templates involved in your test, or make sure templates have the same name across themes.
2. Template Changes in Theme Tests
What happens: If you modify the original theme by creating a new template and assigning it to an asset (like a product or page), the test will be paused.
Why: The new template can't be accessed from the variant theme, so visitors on the variant will see a 404 error when trying to load that page.
How to fix it: Add the missing template(s) to your live theme, being sure to maintain the correct template name (i.e. "sl-12345") for each template. For detailed information on adding a template to your theme, see Theme Compatibility.
How to avoid it: Avoid creating new templates for assets during an active test, or make sure they are accessible from both themes.
