Implementing an A/B test

Overview

An A/B test in Noibu is a feature flag with multiple named variations. Noibu owns the test definition: the title, the variations, the traffic split, the targeting rules, and the success metrics. Your code owns one thing: what each variation renders. The pattern is to read the visitor’s assigned variation, then branch your render code on the result.

Before you start

Two things must be in place:

1. The test exists in Noibu. Create it first, so you have the flag key and the generated code snippet. See Creating an A/B test.

   The test’s draft page shows a snippet generated from your setup — copy it into your site as a starting point for the patterns below.

   

2. The A/B testing SDK is initialized. See Initializing the A/B testing SDK.

Core API reference

const variation = client.getStringValue(flagKey, defaultVariationKey);

• flagKey: the test’s slug from Noibu.

• defaultVariationKey: the value the SDK returns when it cannot resolve the flag. This happens before the SDK is ready, when the key does not exist, or when the network call fails. Always set this default to the control’s key, so an unresolved flag fails toward the current experience rather than toward a variation.

• Flag reads are synchronous and never throw.

Each variation resolves to its own key, a slug of the variation’s name, so the string getStringValue returns is the variation identifier you branch on. getStringDetails returns the same value plus a variant and a reason field, so you can see why a flag resolved the way it did. See QA before launch below.

Note: Noibu also supports boolean, number, and object flags (getBooleanValue, getNumberValue, getObjectValue). These types exist for general feature flagging. A/B test variations are always string values.

Bucketing guarantees

• Assignment happens deterministically at the first evaluation. Noibu buckets the targeting key by the traffic-split percentage, so the same key always resolves to the same variation for a given test.

• If you do not set a custom targeting key, the SDK creates and stores a stable ID in the visitor’s browser. The same visitor therefore gets the same variation across pages and repeat sessions, for as long as that browser storage persists.

• A visitor excluded by a test’s targeting rules always resolves to the control’s value. Your code does not need to handle a separate “excluded” experience.

Copy-paste patterns for common cases

Every pattern below follows the same shape: wait for the client, read the flag, switch on the result with one case per variation key, and let the control fall through to default.

Vanilla JS or Liquid theme (content swap)

Noibu’s generated snippet uses this pattern. A single render function reads the flag and dispatches to a per-variation function.

function onFlagsReady(callback) {
  if (window.NoibuFeatureFlag) {
    var client = window.NoibuFeatureFlag.getClient();
    client.addHandler("PROVIDER_READY", function () {
      callback(client);
    });
  } else {
    window.addEventListener("noibuFeatureFlagReady", function ({ detail }) {
      callback(detail.client);
    });
  }
}

window.addEventListener("noibuFeatureFlagError", function ({ detail }) {
  console.warn("Feature flags unavailable:", detail.message);
  // Continue with the default control experience.
});

function renderStickyCheckoutButton(container) {
  // TODO: variation B implementation
}

function renderCurrentExperience(container) {
  // TODO: this is the current experience. Usually no code is needed here.
}

function render(client, container) {
  var variation = client.getStringValue("cart-drawer-redesign", "current-experience");

  switch (variation) {
    case "sticky-checkout-button":
      // Variation B
      return renderStickyCheckoutButton(container);
    case "current-experience":
      // Variation A, the control. Usually no code is needed here.
      // Let this case fall through to the default.
    default:
      // Visitors excluded by targeting see this. A flag load
      // failure also shows this.
      return renderCurrentExperience(container);
  }
}

onFlagsReady(function (client) {
  render(client, document.getElementById("container"));
});

In a Liquid theme, paste this code in a script tag. Put the tag at the bottom of the section you are testing, and target the section’s own container ID.

CSS class toggle

For layout or styling changes, add a class instead of swapping markup. This method is cheaper than a full re-render, and you can control it entirely in CSS.

onFlagsReady(function (client) {
  var variation = client.getStringValue("pdp-gallery-layout", "current-experience");
  document.documentElement.classList.add("nb-test-pdp-gallery-layout--" + variation);
});

.nb-test-pdp-gallery-layout--grid-2up .product-gallery { /* ... */ }

A/B/n tests (more than two variations)

The pattern stays the same. Add one case for each variation key.

switch (variation) {
  case "variation-b":
    return renderVariationB(container);
  case "variation-c":
    return renderVariationC(container);
  case "current-experience":
  default:
    return renderCurrentExperience(container);
}

Reading flags without flicker

The SDK bundle loads and resolves asynchronously, so window.NoibuFeatureFlag may not exist at first paint. You have two options:

• Gate the render on assignment. Every example above uses this method. Do not render the tested element until onFlagsReady or PROVIDER_READY fires. This works best when you can defer the tested surface without a performance cost — below-the-fold content, a modal, or a drawer.

• Render the control first, then swap after resolution. Render the control immediately, and re-render only if the resolved variation is different. This works best for above-the-fold content, where a delay would cause a visible layout shift. The trade-off is a possible flash of the wrong variation for a returning visitor in a non-control arm.

No single option avoids both trade-offs. Choose a method for each surface based on where the tested element sits on the page.

QA before launch

• Read the resolution reason. client.getStringDetails(key, default) returns { value, variant, reason }. A reason of "SPLIT" confirms a pseudorandom traffic-split assignment, which means the SDK bucketed you into the test correctly. A reason of "DEFAULT" or "ERROR" means you are seeing the fallback value instead.

• Sample multiple variations. Test in separate private or incognito windows, or clear site storage between page loads. Each new session gets a fresh browser ID and a new, independent assignment.

• Set an explicit targetingKey during QA. Use NoibuFeatureFlag.setContext({ targetingKey: "qa-scenario-b" }). This keeps one ID’s assignment stable across reloads while you test. It does not let you choose which variation that ID receives; it only stops the assignment from changing during your test.

• Test on a theme preview or staging domain if the test’s targeting rules or the flag itself target a different domain than your local or development environment. The SDK filters flags by hostname on the client, so a test aimed at your production domain does not resolve on localhost.

Important: There is no dedicated force-variant or preview tool. Such a tool would let you pin a variation with a query parameter regardless of the bucketing result. This is a known gap. If your QA process needs it, tell your Noibu contact.

Test lifecycle

A test moves one way through three states: Draft, Running, and Stopped. A stopped test cannot be restarted.

• Draft. You can edit the entire definition, including the variations, the traffic split, the targeting, and the primary metric. Use this state to write and deploy your variation code.

• Running. Only the hypothesis and the secondary metrics stay editable. Noibu freezes the variations, the traffic split, the targeting rules, and the primary success metric, and enforces this on the server rather than only in the interface.

• Stopped. This state is final. You can delete a test only while it is in the Draft state.

What is safe to change mid-test is the hypothesis text and the secondary metrics, because neither affects bucketing. Everything else is locked because changing it would invalidate the results. See Common mistakes to avoid below.

Ending a test. Stop the test in Noibu once it has reached a verdict. Noibu does not remove the flag or roll the winning variation into your default experience for you. Your team does that work: implement the winning variation directly, delete the switch statement so the winner is the only path, and remove the flag read.

Common mistakes to avoid

• Starting a test before your variation code is live. If the test moves to running before you merge and deploy, Noibu buckets visitors into variations that do not exist on your site. Those visitors fall through to the default case and see the control, but Noibu still counts them as part of the variation. You cannot correct this data afterwards.

• Editing variation code while a test is running. Noibu freezes the test definition, but nothing stops you from changing what renderStickyCheckoutButton does in your own codebase during the test. Do not do this. It silently mixes two different experiences under one variation’s results.

• Running overlapping tests on the same element. If two running tests modify the same element, such as the cart drawer, they confound each other’s results. Neither test’s results will attribute cleanly to its own variations.

• Leaving the default argument pointed at the wrong variation. If your getStringValue default is not the control’s key, a flag-load failure serves an arbitrary or empty experience instead of failing safely toward the current experience.

• Leaving a stopped test’s flag read in the code. After a winner ships, remove the switch statement and the flag read. A live read against a stopped test still resolves to the control, because of the fail-open default, but the code is dead weight and a trap for the next person who assumes the test still runs.

Next steps

Your test is now live in code. Return to Noibu to start the test and monitor the results. See Measuring A/B test results.