Field Level Experimentation

Sanity has created a plugin that allows A/B/N testing experiments to individual fields. You can set the experiments and their variations as config or use an async function to return the information.

You should be able to follow this course if you have completed the Day One with Sanity Studio course. It will also build on the same studio, schema, and front end.

Go to the Day One with Sanity Studio course

To save you from having to fill out and publish a bunch of content to make this course interesting, we have prepared a dataset that you can import and work on.

Download production.tar.gz below and import it into your Sanity project by running the following command in your studio folder:

Download the dataset export

Run the following command in the terminal to Import the dataset into your project's production dataset

npx sanity@latest dataset import ~/path/to/production.tar.gz production

A successful import will give you a bunch of artists, venues, and events in the past and future between 2010–2030.

You could create your own schema types to support A/B Testing from external sources, but Sanity has created a plugin that gives an opinionated way of handling A/B testing inside of the Sanity Studio.

We are going to add a new field to run an experiment on the name of an event. Our hypothesis is that more information in the name of an event will encourage our users to read more about it.

First you need to install and add the plugin @sanity/personalization-plugin to your Sanity Studio configuration

Run the following command in the terminal to install the Personalization Plugin

npm install @sanity/personalization-plugin

When configuring the plugin you will need to specify which fields types to run experiments on, and the details of the experiments and their variants that we are running.

Update sanity.config.ts to configures the plugins field types and experiments

// ...all other imports
import {fieldLevelExperiments} from '@sanity/personalization-plugin'

export default defineConfig({
  // ... all other config settings
  plugins: [
    // ...all other plugins
     fieldLevelExperiments({
      // field types that you want to be able to emperiment on
      fields: ['string'], 
      // hardcoded experiments and variants
      experiments: [
        {
          id: 'event-name',
          label: 'Event Name',
          variants: [
            {
              id: 'control',
              label: 'Control',
            },
            {
              id: 'variant',
              label: 'Variant',
            },
          ],
        },
      ],
     })
  ],
})

Adding a field will create a new type that we can use in our schema types. This type will be prefixed with experiment. In this example the new type to use in the schema will be experimentString. With this we can add the a new Name experimentation field to the schema of our event.

Update eventType.ts to add a newName field to schema of event


export const eventType = defineType({
  name: 'event',
  title: 'Event',
  icon: CalendarIcon,
  type: 'document',
  groups: [
    {name: 'details', title: 'Details'},
    {name: 'editorial', title: 'Editorial'},
  ],
  fields: [
    defineField({
      name: 'name',
      type: 'string',
      group: ['editorial', 'details'],
    }),
    defineField({
      name: 'newName',
      type: 'experimentString',
      group: ['editorial', 'details'],
    }),
    // ... and the rest

You're adding a new field using the new experimentString type to ensure you don't accidentally clear the existing value.

This will render the new experimentation field with a default value input.

Let's add content for our new field in the Studio:

Add a default value to be used when the experiment is not running or if the user is not included in the experiment.
Select add experiment on the field.
Select which experiment you want to add the content for.
Add content for the control

Add content for the A/B test

When specifying the field types in the plugin config you can use Sanity Studio primitives, your own custom defined types, types from other plugins, or you can defined a type in the plugin config. You can add validation/options in the plugin config or on the schema definition.

// ...all other imports
import {defineArrayMember, defineConfig, defineField} from 'sanity'
import {fieldLevelExperiments} from '@sanity/personalization-plugin'

export default defineConfig({
   // ...all other config
  plugins: [
    // ... all other plugins
    fieldLevelExperiments({
      fields: [
        // primitives
        'string',
        'image',
        'slug',

        // custom defined types
        'path',

        // internationalized Array plugin types
        'internationalizedArrayString',

        // types defined in plugin config
        defineField({
          name: 'featuredAuthor',
          type: 'reference',
          to: [{type: 'author'}],
          hidden: ({document}) => !document?.title,
          validation: (Rule) => Rule.required()
        }),
        defineField({
          name: 'customArray',
          title: 'Custom Array',
          type: 'array',
          of: [defineArrayMember({type: 'object', fields: [{name: 'string', type: 'string'}]})],
        }),
      ],
      // experiements removed for brevity
    }),
  ],

})

import {defineField, defineType} from 'sanity'

export const path = defineType({
  name: 'path',
  type: 'string',
  validation: (Rule) =>
    Rule.required().custom((value: string | undefined, context) => {
      if (!value) return true
      if (!value.startsWith('/')) return 'Must start with "/"'
      return true
    }),
})

export const article = defineType({
  name: 'article',
  title: 'Article',
  type: 'document',
  fields: [
    defineField({
      name: 'title',
      title: 'Title',
      type: 'experimentString',
    }),
    defineField({
      name: 'path',
      title: 'Path',
      type: 'experimentPath',
    }),
    defineField({
      name: 'image',
      title: 'Image',
      type: 'experimentImage',
    }),
    defineField({
      name: 'image',
      title: 'Image',
      type: 'experimentImage',
      validation: (Rule) =>
        Rule.custom((value: any) => {
          if (!value.default) return 'Must have a default image'
          if (value.active && (!value.variants || value.variants?.length === 0)) {
            return 'Must have at least one variant'
          }
          return true
        }),
    }),
    defineField({
      name: 'featuredArtist',
      title: 'Featured Artist',
      type: 'experimentFeaturedArtist',
    }),
    defineField({
      name: 'array',
      type: 'experimentCustomArray',
    }),
  ],
  preview: {
    select: {
      title: 'title',
      media: 'image',
    },
    prepare({title, media}) {
      return {
        title: title.default ? title.default : undefined,
        subtitle: title.variants
          ? title.variants.map((variant: any) => variant.value).join(' | ')
          : undefined,
        media: media?.default ?? undefined,
      }
    },
  },
})

You might consider using an external service to help with your A/B testing as they help with traffic allocation of your experiment on the frontend, connect with analytics platforms to give you information about your experiments, and they often have other features such as feature flagging and personalization.

When choosing a service you need to consider if the service meets your needs, what other tools might you need, and what the cost of the service will be.

You can connect @sanity/personalization-plugin to an external service by using an async function for getting your experiments and variants. This function will be passed a Sanity Client so you can use information stored in the Content Lake if needed, as in the example below.

import { getExperiments } from './utils/experiments'

export default defineConfig({
  // ... all other config
  plugins: [
    // ... all other plugins
    fieldLevelExperiments({
      // .. field types
      experiments: (client: SanityClient) => getExperiments(client),
    }
  ],

})

import {SanityClient} from 'sanity'
import {ExperimentType} from '@sanity/personalization-plugin'

export const getExperiments = async (client: SanityClient) => {
  // secret is stored in the content lake using @sanity/studio-secrets
  const query = `*[_id == 'secrets.namespace'][0].secrets.key`
  const secret = await client.fetch(query)

  if (!secret) {
    return []
  }
  // call to external api to fetch experiments
  const response = await fetch('https://example.api/experiments', {
    headers: {
      Authorization: `Bearer ${secret}`,
    },
  })
  const {experiments: externalExperiments} = await response.json()

  // map and transform to get experiments and their variations
  const experiments: ExperimentType[] = externalExperiments?.map(
    (experiment: externalExperiment) => {
      const experimentId = experiment.id
      const experimentLabel = experiment.name
      const variants = experiment.variations?.map((variant) => {
        return {
          id: variant.variationId,
          label: variant.name,
        }
      })
      return {
        id: experimentId,
        label: experimentLabel,
        variants,
      }
    },
  )
  return experiments
}

In the following lesson you will get the data for the experiment and add it to your Next.js application.

You have 6 uncompleted tasks in this lesson

0 of 6

A/B Testing

Prerequisites

Import example content to your dataset

Configure Field Level Experiments

Complex Field Types

Getting experiments