Day-7 compare read

POST /compare reads the same weekdays before and after a ship and returns keep, revert, or extend. It is free on every plan. One successful 200 counts as one monthly agent/API read. POST /query stays paid-only.

Schedule the read for ship day + 8 or later, in UTC. The compare window is the N days after the ship date, and the ship day is in neither window. If either window includes today UTC, the verdict is extend / incomplete_current_day and that overrides every other reason. A 7-day read on ship day + 7 ends today, so it extends.

CLI and MCP

compare and annotate are in @agent-analytics/cli@0.5.41. MCP exposes the same calls as compare and append_project_annotation.

npx --yes @agent-analytics/cli@0.5.41 compare <project> --goal <event> [--ship-date <YYYY-MM-DD>] [--days <N>] [--denominator <event>] [--exclude-denominator <field>=<value>] [--metric '<agg>(<property>)'] [--metric-event <event>] [--better higher|lower] [--keep-threshold <fraction>] [--revert-threshold <fraction>] [--min-denominator <N>] [--min-goal-events <N>] [--filter <json>] [--json]

--days is window_days and defaults to 7. For explicit windows, pass --baseline-from, --baseline-to, --compare-from, and --compare-to, and omit --ship-date and --days. --denominator defaults to page_view. --exclude-denominator is repeatable and maps to denominator_filters with op neq. --metric is optional (sum(<property>), avg(<property>), p50(<property>), p75(<property>), p95(<property>)). --metric-event is the event whose property is measured and defaults to the goal. --better is higher or lower and defaults to higher. Omit metric for the conversion-rate read. Thresholds are fractions.

npx --yes @agent-analytics/cli@0.5.41 annotate <project> --title <text> [--note <text>] [--at <ISO-8601 datetime>]

--at is occurred_at and defaults to the current UTC time. --note is sent only when you pass it.

The MCP tool compare takes project, goal_event, denominator_event, filters, optional denominator_filters, either ship_date and window_days or baseline and compare windows, an optional nested rule with keep_threshold, revert_threshold, min_denominator, and min_goal_events, and optional metric, metric_event, and better. The MCP tool append_project_annotation takes project, occurred_at, title, and note.

The HTTP API is still POST /compare and POST /project-context/annotations.

Windows

Pass either ship_date or both baseline and compare. Not both, and not window_days with explicit windows.

ship_date is YYYY-MM-DD in UTC. window_days defaults to 7. It must be an integer from 1 to 90 and a multiple of 7, so the before and after windows cover the same weekdays. The longest ship window is 84 days. The baseline is ship-N through ship-1. The compare window is ship+1 through ship+N.

Explicit windows are { "date_from", "date_to" }, inclusive, each at most 90 days. weekday_aligned is reported and is not required.

A window whose date_to is after today UTC is 400 VALIDATION_ERROR. A window that includes today is 200 with verdict extend and reason incomplete_current_day.

Request

goal_event is required (trimmed, max 256 characters). denominator_event defaults to page_view. filters use the /query filter shape (field, op, value, including properties.<key>). A filter on event is rejected. The two event names are the parameters above.

Invalid filters are checked before any count query. A bad field, operator, or property key is a plain 400 (INVALID_FILTER_FIELD, INVALID_FILTER_OP, or INVALID_PROPERTY_KEY) whose body is only { error, message }. It does not include available_properties or any other /properties payload, and it does not count as a read.

rule fields are optional. Omitted fields use the documented day-7 examples, not product policy: keep_threshold 0.15, revert_threshold -0.1, min_denominator 200, min_goal_events 20. Thresholds are relative fractions. 0.15 means +15%. An absolute value above 5 is 400 VALIDATION_ERROR. For keep_threshold 15 the message is rule.keep_threshold is 15. Thresholds are fractions (0.15 = +15%), and the absolute value must be 5 or less. keep_threshold must be greater than revert_threshold. min_goal_events is the minimum number of unique converters across both windows. The field name is unchanged. Minimums are integers of 0 or more. Omitted fields are listed in rule.defaults_applied.

{
  "project": "my-site",
  "goal_event": "signup",
  "denominator_event": "page_view",
  "ship_date": "2026-04-15",
  "window_days": 7,
  "rule": {
    "keep_threshold": 0.1,
    "revert_threshold": -0.1,
    "min_denominator": 200,
    "min_goal_events": 20
  }
}

Exclude 404 or bot page views from the denominator

A 404 or bot page view still counts as a denominator user. On grokbotpages.com, a compare of template_opened after the ship on 2026-11-04 returned revert because 45 one-hit visitors with a page_view title of Page not found | grokbotpages.com inflated the compare-window denominator.

denominator_filters removes those page views from denominator_unique_users only. filters is unchanged. Omit the field, or send [], and the denominator matches a compare with no exclude. At most 10 entries. Each value is a string or a number. Ops are eq, neq, gt, lt, gte, lte, and contains. A bad field, a bad op, a value that is not a string or number, or more than 10 entries is 400 VALIDATION_ERROR. The response echoes denominator_filters next to filters, and echoes [] when the request omitted it.

goal_converters still counts only goal users inside that filtered denominator. Goal users outside it are goal_users_outside_denominator.

{
  "denominator_filters": [
    {
      "field": "properties.title",
      "op": "neq",
      "value": "Page not found | grokbotpages.com"
    }
  ]
}

npx --yes @agent-analytics/cli@0.5.41 compare grokbotpages.com --goal template_opened --ship-date 2026-11-04 --exclude-denominator 'properties.title=Page not found | grokbotpages.com'

Pass --exclude-denominator more than once to add another neq entry. The MCP compare argument is the same denominator_filters array.

Metric and verdict

This section is the conversion-rate compare. It applies when metric is omitted.

count_mode is unique_users. A user who fires the goal three times counts once. goal_converters = min(goal_unique_users, denominator_unique_users) after denominator_filters. rate is goal_converters / denominator_unique_users. goal_users_outside_denominator counts goal users who are not in that denominator set. With no denominator_filters, that is goal users who did not fire the denominator event.

Relative lift is (compare_rate - baseline_rate) / baseline_rate, rounded to 6 decimal places. Thresholds are compared to that rounded lift.

The verdict is deterministic. Order:

extend / incomplete_current_day when either window includes today UTC. This overrides every later reason.
extend / below_min_sample when either window has fewer denominator users than min_denominator, or the two windows together have fewer unique converters than min_goal_events.
extend / zero_denominator only if a denominator is still 0 after that. That is reachable only when min_denominator is 0.
extend / zero_baseline when the baseline rate is 0. lift is null.
revert / revert_threshold_met only when lift is at or below revert_threshold and the 90% interval lies entirely below zero.
extend / inconclusive when that drop is inside the noise: lift is at or below revert_threshold, but the interval is not entirely below zero.
keep / keep_threshold_met when lift is at or above keep_threshold and the 90% interval for the rate difference lies entirely above zero (interval.low > 0).
extend / interval_includes_zero when lift clears the keep threshold but the interval includes zero.
extend / no_evidence_of_gain otherwise.

revert is only revert_threshold_met. Every other result is keep or extend.

interval.estimate is rate_difference: compare rate minus baseline rate, not the relative lift. The method is newcombe_wilson_90. The Newcombe interval treats the unique-converter rate as a binomial proportion. interval is null on below_min_sample and zero_denominator. lift is null when the baseline rate is 0 or a rate cannot be formed.

ship_date and window_days are omitted from the response when you passed explicit windows.

Numeric property metric

metric is optional. Omit it and the response stays the conversion-rate compare above. That response has no metric object, no raw_lift, no interval_method, and no window value, n, or skipped.

Set metric to one of sum(<property>), avg(<property>), p50(<property>), p75(<property>), or p95(<property>). The call stays free and still counts as one metered read. metric_event is the event whose property is measured. It defaults to goal_event. filters apply to the metric event. denominator_filters still apply to the denominator user count. The sum numerator ignores denominator_filters: every numeric property value on metric_event in the window is added, then that sum is divided by denominator unique users.

better is higher or lower. The default is higher. It is not inferred from the property name. Set lower for cwv_lcp and the other vitals.

Only JSON numbers are aggregated. Numeric strings, booleans, nulls, and missing keys are skipped. n is the numeric count. skipped is the other rows. Percentiles are nearest-rank: sort ascending, break ties by event id, and take rank ceil(p/100 * n) where p is 50, 75, or 95. Rank 1 is the smallest value. The result is an observed value. avg is the arithmetic mean, rounded to 6 decimal places. percentile_method is nearest_rank for a percentile and null for sum or avg. On POST /query, a percentile whose filtered set is larger than 200,000 events is 400.

sum sets per_user to true. avg and percentiles set per_user to false and compare the aggregate itself. Each window still includes the conversion rate and user counts, plus value, n, and skipped.

raw_lift is (compare_value - baseline_value) / baseline_value. lift is raw_lift, negated when better is lower, so the existing thresholds keep their meaning. A missing compare value, or a baseline of 0 or null, is zero_baseline with lift null. interval is null. interval_method is none.

min_denominator is still unique denominator users per window (default 200). When metric is set, min_goal_events is numeric n on metric_event across both windows (default 20).

The existing compare order still applies. A metric compare returns extend instead of keep or revert in v1.

extend / incomplete_current_day when either window includes today UTC.
extend / below_min_sample when n across both windows is under min_goal_events, or either window is under min_denominator.
extend / zero_denominator when a denominator is still 0.
extend / zero_baseline when the compare value is missing, or the baseline value is 0 or null. lift is null.
extend / no_interval_for_metric when lift meets keep_threshold or revert_threshold. lift is still reported. Example reason_detail: “Relative lift 0.5 meets the keep threshold 0.15, but metric compare does not compute an interval in v1, so the verdict is extend.”
extend / no_evidence_of_gain when lift is between the thresholds.

Default thresholds stay keep_threshold 0.15 and revert_threshold -0.1.

The response metric is an object with metric, fn, property, metric_event, better, per_user, and percentile_method.

Web Vitals

Compare p75 of cwv_lcp. The goal stays signup. Lower is better. p75 is the headline percentile.

Hosted stores cwv_* on page_view. Self-hosted core stores the same keys on $web_vitals. Hosted does not store $web_vitals. This example uses the hosted event.

The rated read (thresholds, good / needs-improvement / poor share, and cwv_pass) is vitals in @agent-analytics/cli@0.5.41, the MCP tool web_vitals, or POST /vitals. See Web Vitals.

On 2026-11-13, CLS and INP changed in place. A compare window that crosses that date mixes the old and new definitions.

npx --yes @agent-analytics/cli@0.5.41 compare my-site --goal signup --metric 'p75(cwv_lcp)' --metric-event page_view --better lower --ship-date 2026-11-05

{
  "project": "my-site",
  "goal_event": "signup",
  "metric": "p75(cwv_lcp)",
  "metric_event": "page_view",
  "better": "lower",
  "ship_date": "2026-11-05"
}

The stored LCP value is milliseconds. CLS is unitless.

Revenue per user

sum(revenue) compares purchase revenue per denominator user (per_user: true). The sum ignores denominator_filters. better stays higher.

npx --yes @agent-analytics/cli@0.5.41 compare my-site --goal purchase --metric 'sum(revenue)' --ship-date 2026-11-04

{
  "project": "my-site",
  "goal_event": "purchase",
  "metric": "sum(revenue)",
  "ship_date": "2026-11-04"
}

Track the event as aa.track('purchase', { revenue: 29, currency: 'USD' }). See Revenue.

Guardrails

Annotations and traffic mix do not change the verdict. incomplete_current_day does.

guardrails.annotations lists stored annotations whose occurred_at falls inside either window, UTC day inclusive. Each item includes windows: baseline, compare, or both. An annotation on the ship day is not included, because that day is in neither window. guardrails.confounded is true when the list is not empty.
guardrails.traffic_mix is utm_source and referrer on the denominator event. A value is a shift when its share of tagged events moves by at least 0.20. Each side keeps the top 20 values by event count. Request filters apply to the window counts and do not apply to the mix. filters_applied is true when the request included filters. If the mix query throws, traffic_mix is null and traffic_mix_note is omitted: traffic mix could not be computed.
guardrails.incomplete_current_day is true when either window includes today UTC. The verdict is then extend / incomplete_current_day.
guardrails.caveat is always: Before/after is not causal. A keep or revert verdict is a deterministic reading of this rule on these two windows, not proof that the change caused the difference.

Annotations

POST /project-context/annotations appends one annotation. Goals, activation events, the glossary, and existing annotations stay. The call is not a metered read. Auth matches PUT /project-context. An agent session needs projects:write.

Allowed fields: project, occurred_at, title, and optional note. occurred_at must be an ISO-8601 datetime and is stored as UTC ISO. title is required, max 120 characters. note is optional, max 500 characters. An empty note is omitted. Unknown fields are rejected. The body cap is 512KB (524288 bytes).

The identity of an annotation is normalized occurred_at plus title. The same pair already stored returns 200 with appended: false, the existing annotation (the new note is not written), and the unchanged annotation_count.

A new annotation when 100 are already stored returns 409 VALIDATION_ERROR and stores nothing. The same status is used when the stored document cannot be parsed, and when 32 compare-and-swap attempts are exhausted (project context changed concurrently; retry the append).

PUT /project-context still replaces the whole document. Append against append is safe. A PUT that read the context before someone else’s append can still overwrite that append. An append that is in flight when a PUT lands retries, so that PUT document and the new annotation can both remain.

Limits

Free accounts can call /compare and still receive PRO_REQUIRED from POST /query. A free account is limited to 10 requests per minute. The 11th compare in that minute is 429 RATE_LIMITED with reason: "rate_limit" and limit: 10. The free monthly read cap remains 500. The next compare after 500 successful reads is 429 RATE_LIMITED with reason: "monthly_read_limit". A 4xx or 5xx compare does not increment the counter. Pro uses the Pro request rate. One successful compare is still one metered read.

Another account’s project is 404 PROJECT_NOT_FOUND. An unexpected compare failure is 500 QUERY_FAILED with message failed to compare windows. An AnalyticsError from a count query is returned as that error’s code and message.