Mutual Exclusion

Shoplift doesn't limit how many tests you can run simultaneously, and depending on the tests you are running, our tracking methodologies will intelligently isolate test data when running tests that have the potential to overlap. That being said, there are some things to know when running multiple template tests, multiple theme tests, or multiple template and theme tests.

What happens when I run multiple template tests?

Shoplift may or may not isolate test data depending on what templates are being tested and what visitor segments you are testing.

If you are running multiple tests on a single template, but with different audience and visitor segments, then the segmentation rules will prevent visitors from being placed in both tests concurrently.

In the event that you run multiple tests on a single template, but your audience and visitor segments overlap, then Shoplift will dynamically place potential visitors into a single test and intelligently attribute the events throughout their journey to the tested experience they encounter.

If you are running multiple tests across different templates, Shoplift does not currently isolate tested visitor segments. Thus, it is important to ensure that your visitor segments have do not have potential to overlap on these tests, or your test results could be adversely impacted. We are working on a mutual exclusion solution so that you can confidently exclude visitor data that may have the potential to overlap when running multiple template tests across different areas of your store.

In instances where Shoplift proactively prevents tests with the potential for overlapping visitor segments from cross-contaminating datasets, it is important to know that your tests may reach statistical significance more slowly because the total potential traffic of each test is reduced, because it must be divided between each test.

What happens when I run multiple theme tests?

As with template tests, Shoplift will allocate visitors and attribute their event data according to your test targeting rules, such as your audience or visitor segments. Similarly, if multiple tests have visitor segments that overlap, Shoplift will dynamically isolate the visitor events so that your test data remains valid.

What happens when I run theme tests and template tests at the same time?

In some cases, you may need to run template tests and theme tests concurrently. When doing so, it is important to know that the visitors and data from your template test - while technically occurring on your Original (A) for your theme test - are isolated from your theme test and dataset.

We do this so that your Original (A) and Variant (B) themes remain equal in all respects, and can ensure that your template test does not influence the performance of your theme test, and vice-versa.

If you have any questions or concerns, our support team is always available to assist. Reach out from within the app using the live chat widget in the bottom right of the screen, or email [e-mail] to get in touch.