AI Agent Experiment Tracking

Use this guide when you want to run browser-side A/B tests on a real website or app by directing your AI agent to do the work for you.

This guide is about page elements like headlines, CTAs, pricing copy, and signup flows. It is not for prompt evals, model comparisons, or internal agent workflow testing.

Before you start

• Your site already has tracker.js installed.
• The project already exists in Agent Analytics.
• You are on a paid plan, because experiments are not available on the free tier.
• You have one business goal event in mind, such as signup, checkout, or another real conversion event.

If you still need the initial setup, start with Getting Started.

1. Pick one goal and one UI element to test

Start with a narrow change. Good first experiments are usually one CTA, one headline, or one pricing message tied to one business event.

Copyable prompt:

I want to run an A/B test on my-site. Help me choose one page element to test and one goal event that reflects a real business outcome. Keep the scope to a single CTA or headline.

Avoid using page_view as the goal unless the page itself is the product outcome. Most of the time, signup, checkout, or a core activation event is a better goal.

2. Ask your agent to verify or add the goal event

Before the experiment exists, make sure the goal event is already tracked and named consistently with the rest of the site.

Copyable prompt:

Add a signup tracking event to this CTA and keep the event name consistent with the existing site naming. If a matching event already exists, reuse it instead of inventing a new one.

If the event is simple, your agent should usually prefer declarative tracking in markup rather than extra JavaScript.

3. Ask your agent to create the experiment

Once the goal event is ready, ask the agent to create an experiment with a clear name and simple variants.

Copyable prompt:

Create an experiment for the signup CTA on my-site with control and new_cta variants, using signup as the goal event.

If you want the exact request and response shape behind that step, use the Experiments API reference.

Keep experiment names in snake_case, and keep the scope easy to explain later. signup_cta is better than something vague like homepage_test.

4. Ask your agent to wire the variant declaratively

For most sites, declarative variants are the cleanest path. The original HTML stays as the control, and the variant content lives in data-aa-variant-* attributes.

Copyable prompt:

Wire this hero headline into the signup_cta experiment using declarative tracker.js attributes. Keep the existing text as control and add the new variant in the markup.

Example result:

<h1 data-aa-experiment="signup_cta"
    data-aa-variant-new_cta="Start free today">
  Start your free trial
</h1>

Use window.aa?.experiment() only when the UI is too dynamic for declarative HTML. For the lower-level mechanics, see Tracker.js.

5. Ask your agent to QA both variants

Before you trust the data, make sure both variants actually render and the goal event still fires.

Copyable prompt:

Show me how to force each variant locally so I can QA both versions, then verify that the signup event still fires correctly for each one.

The forced-variant URLs look like this:

• ?aa_variant_signup_cta=control
• ?aa_variant_signup_cta=new_cta

Your agent should use those URLs to check both versions without waiting for hash-based assignment.

6. Ask your agent to read results and recommend a winner

Once the experiment has traffic, ask your agent to check whether you have enough data and what the current recommendation is.

Copyable prompt:

Check results for signup_cta and tell me whether we have enough data to pick a winner. If we do, recommend whether to keep running it, pause it, or complete it with a winner.

For the underlying HTTP endpoints for reading, pausing, resuming, completing, or deleting experiments, use the Experiments API reference.

Make the decision on the business goal event, not on raw traffic. A variant with more exposures is not automatically better if it does not improve signup or checkout.

Common mistakes

• Testing too many elements at once, which makes the result hard to interpret.
• Using page_view as the goal instead of a conversion event.
• Creating the experiment before the goal event is actually tracked.
• Forgetting to QA forced variants before sending traffic through the test.
• Packing layout, copy, and offer changes into one experiment when one focused change would be easier to learn from.
• Leaving an experiment active after you already know the winner.

Related

• Guides
• Getting Started
• Tracker.js
• CLI vs MCP vs API
• Experiments API
• API Reference