Skip to content

Commit 6314f8f

Browse files
GHaberisGHaberis
authored
Merge pull request #14812 from guardian/gh-ab-docs
Consolidate Beta AB Test Framework Docs
2 parents 0f572c4 + 42ac073 commit 6314f8f

File tree

2 files changed

+127
-79
lines changed

2 files changed

+127
-79
lines changed

‎ab-testing/README.md‎

Lines changed: 1 addition & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,73 +1,6 @@
11
# Beta AB Test framework
22

3-
This directory contains the code and configuration for the new **Beta** AB Test framework, developed by commercial-dev to support both client and server side A/B tests in DCR. The goal is to eventually replace the legacy A/B testing framework with this new framework.
4-
5-
Please reach out to commercial-dev if you're setting up a client or server side A/B tests in DCR and you're interested in using this new framework whilst it's in beta (this would be helpful for the team) or if you'd like up to date information on it's state of readiness. **For now the legacy framework is still available to use, and documentation for that can be found [here](https://github.com/guardian/dotcom-rendering/blob/main/dotcom-rendering/docs/development/ab-testing-in-dcr.md)**.
6-
7-
The [`abTest.ts`](./abTest.ts) module is where you should define your AB tests.
8-
9-
Add your AB tests to the `abTests` array in the `abTest.ts` file. Each test should have a unique name.
10-
11-
```ts
12-
{
13-
name: 'webex-example-test',
14-
description:
15-
'Test something interesting on the site',
16-
owners: ['[email protected]'],
17-
status: 'ON',
18-
expirationDate: '2050-12-30',
19-
type: 'client',
20-
audienceSize: 10 / 100,
21-
groups: ['control', 'variant'],
22-
shouldForceMetricsCollection: true
23-
}
24-
```
25-
26-
When you create a PR that modifies the `abTest.ts` file, a git hook and CI will run checks to ensure that your AB test is valid (not expired, enough space for the test etc.).
27-
28-
When your PR is merged, the AB test will be automatically deployed to Fastly and be available at the same time as your changes.
29-
30-
## Guidelines for AB Tests
31-
32-
### Naming Conventions
33-
34-
AB tests should be prefixed with the team associated with the test, for example `webex-example-test`. This helps to identify the team responsible for the test and is enforce by typescript validation.
35-
36-
### Test Size and Groups
37-
38-
The `audienceSize` is the size of the whole test and is divided between the test groups that you specify. The "resolution" of sizing is down to 0.1%, so groups will be rounded to the nearest 0.1%.
39-
40-
Convention is to have groups named control and variant, but you can name them as you wish.
41-
42-
A single group is also possible, for example if you're rolling out a new feature and don't need a control.
43-
44-
### Client vs Server Side Tests
45-
46-
All requests are processed by Fastly at the edge, however, ab testing of server-side logic in Frontend or DCR will need to be cached separately. Client side tests do not need to be cached separately, as they are applied in the browser after the response is delivered.
47-
48-
Ensure that the `type` field is set to either `client` or `server` to indicate the type of test so that server side tests can be cached correctly, and client side tests are not splitting the cache unnecessarily.
49-
50-
There's a limit of the number of concurrent server-side tests that can be run, enforce by the validation script, so it's important to use client-side tests where possible.
51-
52-
### Test Expiration
53-
54-
AB tests should have an expiration date set in the future. This is to ensure that tests do not run indefinitely.
55-
56-
Expired tests will cause the ab testing validation to fail, and will not be deployed.
57-
58-
Tests that expire while they are are in-flight will not be served by fastly, and should be removed from the `abTest.ts` file as soon as possible.
59-
60-
### Audience Spaces
61-
62-
Ideally AB tests would never overlap (users being in multiple tests), but sometimes this is unavoidable, for example when running a very large 50+% test without interrupting existing tests.
63-
64-
To add a test where there is not enough space in the default audience space (`A`), you can specify a different `audienceSpace` in the test definition.
65-
66-
For example if there are already 3 25% tests in space `A` totalling 75%, and you want to run a 50% test, you can set the `audienceSpace` to `B` to allow this test to overlap with the existing tests.
67-
68-
### Test Status
69-
70-
Tests can be set to `ON` or `OFF` using the `status` field. Only tests with status `ON` will be validated and deployed.
3+
This directory contains the code and configuration for the AB testing framework used on theguardian.com. If you're looking to set up a new server or client side AB test using the new framework then please visit the docs [here](https://github.com/guardian/dotcom-rendering/blob/main/dotcom-rendering/docs/development/ab-testing-in-dcr.md#beta-ab-test-framework).
714

725
## How it works
736

‎dotcom-rendering/docs/development/ab-testing-in-dcr.md‎

Lines changed: 126 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -78,34 +78,103 @@ You can verify that you have been correctly assigned to the variant by appending
7878

7979
You can access server-side `abTests` within DCR wherever the CAPI object is used (`CAPIArticle.config.abTests`).
8080

81+
---
82+
8183
# Beta AB Test framework
8284

8385
This is a new framework that has been developed by commercial-dev to support both client and server side A/B tests in DCR. The goal is to eventually replace the legacy A/B testing framework described above with this new framework.
8486

8587
Please get in touch with the commercial-dev team if you'd like up to date information on it's state of readiness.
8688

89+
If you're interested in how it works please visit the docs [here](https://github.com/guardian/dotcom-rendering/tree/main/ab-testing#beta-ab-test-framework).
90+
8791
## Creating a new A/B test
8892

89-
Create an ab test in [ab-testing/abTest.ts](../ab-testing/abTest.ts) both server and client side tests are defined here. More information on defining tests can be found in [ab-testing/README.md](../ab-testing/README.md).
93+
### 1. Configure your A/B test
94+
95+
Create an A/B test in [ab-testing/abTest.ts](../ab-testing/abTest.ts) **both server _and_ client side tests** are defined here.
96+
97+
Add your A/B tests to the `abTests` array in the `abTest.ts` file. Each test should have a unique name.
98+
99+
```ts
100+
{
101+
name: 'webex-example-test',
102+
description:
103+
'Test something interesting on the site',
104+
owners: ['[email protected]'],
105+
status: 'ON',
106+
expirationDate: '2050-12-30',
107+
type: 'client',
108+
audienceSize: 10 / 100,
109+
groups: ['control', 'variant'],
110+
shouldForceMetricsCollection: true
111+
}
112+
```
113+
114+
When you create a PR that modifies the `abTest.ts` file, a git hook and CI will run checks to ensure that your A/B test is valid (not expired, enough space for the test etc.).
115+
116+
When your PR is merged, the A/B test will be automatically deployed to Fastly and be available at the same time as your changes.
117+
118+
#### Guidelines for A/B tests
119+
120+
#### Naming Conventions
121+
122+
A/B tests should be prefixed with the team associated with the test, for example `webex-example-test`. This helps to identify the team responsible for the test and is enforce by typescript validation, you can inspect & edit the allowed team name definitions [here](https://github.com/guardian/dotcom-rendering/blob/main/ab-testing/types.ts#L9).
123+
124+
#### Test Size and Groups
125+
126+
The `audienceSize` is the size of the whole test and is divided between the test groups that you specify. The "resolution" of sizing is down to 0.1%, so groups will be rounded to the nearest 0.1%.
127+
128+
Convention is to have groups named control and variant, but you can name them as you wish.
129+
130+
A single group is also possible, for example if you're rolling out a new feature and don't need a control.
131+
132+
#### Client vs Server Side Tests
133+
134+
All requests are processed by Fastly at the edge, however, A/B testing of server-side logic in Frontend or DCR will need to be cached separately. Client side tests do not need to be cached separately, as they are applied in the browser after the response is delivered.
135+
136+
Ensure that the `type` field is set to either `client` or `server` to indicate the type of test so that server side tests can be cached correctly, and client side tests are not splitting the cache unnecessarily.
137+
138+
There's a limit of the number of concurrent server-side tests that can be run, enforce by the validation script, so it's important to use client-side tests where possible.
139+
140+
#### Test Expiration
141+
142+
A/B tests should have an expiration date set in the future. This is to ensure that tests do not run indefinitely.
143+
144+
Expired tests will cause the A/B testing validation to fail, and will not be deployed.
145+
146+
Tests that expire while they are are in-flight will not be served by fastly, and should be removed from the `abTest.ts` file as soon as possible.
147+
148+
#### Audience Spaces
149+
150+
Ideally A/B tests would never overlap (users being in multiple tests), but sometimes this is unavoidable, for example when running a very large 50+% test without interrupting existing tests.
151+
152+
To add a test where there is not enough space in the default audience space (`A`), you can specify a different `audienceSpace` in the test definition.
153+
154+
For example if there are already 3 25% tests in space `A` totalling 75%, and you want to run a 50% test, you can set the `audienceSpace` to `B` to allow this test to overlap with the existing tests.
155+
156+
#### Test Status
157+
158+
Tests can be set to `ON` or `OFF` using the `status` field. Only tests with status `ON` will be validated and deployed.
90159

91160
When the config is merged, the A/B test will be automatically deployed and be available at the same time as your changes.
92161

93-
Ab test on/off state is controlled only by the config. Expired tests will cause the ab testing validation to fail, they will also not be served. In effect expired tests are turned off "automatically", but their config needs to be cleaned up.
162+
A/B test on/off state is controlled only by the config. Expired tests will cause the A/B testing validation to fail, they will also not be served. In effect expired tests are turned off "automatically", but their config needs to be cleaned up.
94163

95164
The test will appear in https://frontend.gutools.co.uk/analytics/ab-testing once the config is deployed.
96165

97-
## Putting code changes behind an A/B test (group)
166+
### 2. Putting your code changes behind an A/B test
98167

99-
### Use in Components
168+
Once your A/B test has been configured you can conditionally put your code changes behind an A/B test participation. The instructions below describe how to do this, and are applicable to both client and server side tests.
100169

101-
Again, this applies to both client and server side tests.
170+
#### Use in Components
102171

103172
```ts
104173
// Within the components
105174
import { useBetaAB } from '../lib/useAB';
106175

107176
const someComponent = () => {
108-
// Example usage of AB Tests
177+
// Example usage of A/B tests
109178
const abTests = useBetaAB();
110179

111180
// Am I in the test at all?
@@ -140,13 +209,57 @@ const someComponent = () => {
140209

141210
```
142211

143-
### Other ways to check
212+
### 3. Ways to check your participation
213+
214+
#### In source code
215+
216+
As detailed above the `useAB` module exposes methods for getting a user's A/B test participations.
217+
218+
```ts
219+
import { useBetaAB } from '../lib/useAB';
220+
221+
const abTests = useBetaAB();
222+
223+
// Get all of the user's server/client-side A/B test participations
224+
const abTestParticipations = abTests?.getParticipations(); // EG. { commercial-dev-client-side-test: 'variant', commercial-dev-server-side-test: 'variant' }
144225

145-
The ab test API is also available on the window object as `window.guardian.modules.abTests`, this only works client side. It's best to use the `useBetaAB` hook in react components.
226+
// Is user in the AbTestTest test (any cohort)
227+
const isInTest = abTests?.isUserInTest('AbTestTest') ?? false;
228+
229+
// Is user in the AbTestTest test (control cohort)
230+
const isInControlGroup =
231+
abTests?.isUserInTestGroup('AbTestTest', 'control') ?? false;
232+
233+
// Is user in the AbTestTest test (variant cohort)
234+
const isInVariantGroup =
235+
abTests?.isUserInTestGroup('AbTestTest', 'variant') ?? false;
236+
```
237+
238+
#### On the Client
239+
240+
The A/B test API described above is also available on the window object as `window.guardian.modules.abTests`. **Note:** This only works client side, you should use the `useBetaAB` hook described above in React components.
241+
242+
#### On the Server
146243

147244
Server side tests are also available in the CAPI object e.g. `CAPIArticle.config.serverSideABTests`.
148245

149-
## Forcing yourself into a test
246+
#### In the response headers
247+
248+
Fastly sends a user's AB participations via the `x-gu-server-ab-tests` response header (server side A/B tests) and `gu_client_ab_tests` response cookie (client side A/B tests).
249+
250+
### 4. Testing your changes on CODE
251+
252+
If you want to test your changes on CODE you need to follow these steps:
253+
254+
1. Configure the A/B tests on your branch
255+
256+
2. Deploy your branch to CODE
257+
258+
3. Manually run the [🧪 AB testing CI (CODE)](https://github.com/guardian/dotcom-rendering/actions/workflows/ab-testing-ci-code.yml) worfklow using your branch. This deploys the test config to Fastly CODE.
259+
260+
The 3rd step is crucial as Fastly buckets users into tests/cohorts and returns your A/B test participations as response headers.
261+
262+
### 5. Forcing yourself into a test on PROD/CODE
150263

151264
Use the opt-in and opt-out URL fragments to force yourself into or out of a test.
152265

@@ -156,5 +269,7 @@ When opted-out, you'll return to random/mvt based assignment.
156269

157270
These links are also in the [frontend admin](https://frontend.gutools.co.uk/analytics/ab-testing).
158271

159-
- Opt-in Example: `https://theguardian.com/ab-tests/opt/in/commercial-test-example:variant`
160-
- Opt-out: `https://theguardian.com/ab-tests/opt/out`
272+
- Opt-in Example on PROD: `https://theguardian.com/ab-tests/opt/in/commercial-test-example:variant`
273+
- Opt-out on PROD: `https://theguardian.com/ab-tests/opt/out`
274+
275+
You can use the same routes on CODE.

0 commit comments

Comments
 (0)