When designing a website, you’ve probably heard the term “below the fold.” But what exactly does that mean, and why does it matter for A/B testing?

What is ‘Below the Fold’?

“Below the fold” refers to the part of a webpage that is not immediately visible when the page first loads. It requires users to scroll down to view it. The term originates from the newspaper industry, where stories “below the fold” were physically below the fold of the newspaper and were typically seen after the more prominent headlines on top.

In web design, content below the fold can include images, product descriptions, buttons, or calls to action that users only interact with after scrolling. This area can be tricky to optimize because not every visitor will scroll down, making it more difficult to track user behavior accurately.

Challenges of A/B Testing Below the Fold

A/B testing elements below the fold brings specific challenges. Here are some of the key issues you might encounter:

1. Lower Visibility and Interaction: Since fewer users see the content below the fold, fewer interactions can be measured, making it harder to gather meaningful data. For example, testing different variations of a button below the fold might take much longer to reach statistical significance than testing something above the fold.

2. Sample Bias: People who scroll down to view the content below the fold are already more engaged. This introduces a selection bias, as these users might behave differently from users who leave the page early. Your test results could be skewed because they are based on this already engaged segment of users.

3. Delayed Impact: Changes below the fold may not immediately impact the user experience, meaning the conversion metrics might take longer to show a difference between variations. This extends the test duration and requires more traffic to reach significant conclusions.

Sample Ratio Mismatch (SRM) – What Is It?

When running an A/B test, each variation of your test should get an equal share of traffic. For example, in a 50/50 split test, each variation (A and B) should receive roughly 50% of the users.

However, if one variation ends up receiving significantly more or less traffic than intended, this creates a Sample Ratio Mismatch (SRM). SRM is a red flag that something is wrong with your test setup, and it can severely compromise your test results.

What Causes SRM?

SRM can occur due to various factors, such as:

• Technical Glitches: Errors in the A/B testing tool or on the website itself can lead to uneven traffic distribution.
• Ad Blockers: Some users with ad-blockers may prevent one variation from loading correctly, causing traffic to skew.
• User Segmentation: If you’re running segmented tests (like on mobile vs. desktop), uneven traffic to these segments can cause SRM.

How to Detect SRM

Detecting SRM early in your test can save you time and prevent you from drawing incorrect conclusions. One excellent resource to help you check for SRM is Lukas Vermeer’s website. He provides tools and insights into SRM and its impacts. Additionally, the Google Chrome SRM checker extension is a simple and effective way to monitor whether your test is distributing traffic correctly across variations.

Conclusion

A/B testing below the fold introduces unique challenges because of limited visibility and user interaction. Additionally, failing to detect or account for Sample Ratio Mismatch (SRM) can invalidate your test results, costing you time and resources. By understanding these challenges and using tools to check for SRM, you can run more effective and reliable A/B tests, even on trickier parts of your website like below-the-fold content.