Install and configure Sanity AI Assist

Sanity AI Assist puts the power of Large Language Models (LLM) right at your fingertips in the studio, where your content lives. Write reusable instructions in natural human language to a document-aware AI assistant that can handle chores and repetitive tasks while you focus on the creative stuff.

AI Assist is a plugin for Sanity Studio and is installed using your favorite package manager, such as npm, yarn, or pnpm. It’s a good idea to ensure your studio is up to date while you’re at it. AI Assist requires your studio to be v3.26.0 or later to work.

npm install sanity@latest @sanity/assist

Once installed in your project, you must activate the plugin by importing it and adding it to your main studio configuration. In sanity.config.ts, add assist to the plugins array:

import { defineConfig } from 'sanity'
import { assist } from '@sanity/assist'
/* other imports */

export default defineConfig({
  /* other config */
  plugins: [
    /* other plugins */
    assist(),
  ]
})

We’ll look at some configuration options for the assist plugin further on in this article, but for now, this is everything you need to get started.

After installing the AI Assist package and importing and adding the plugin to your studio configuration, you need to create a token to allow the plugin to access the AI Assist API. This needs to be done by a project member with token creation permissions (typically someone with an admin or developer role):

• Start the studio and open any document
• Click the sparkle icon(✨) in the document header near the close document X-button
• Then select Manage instructions

Selecting Manage instructions will open an inspector pane to the right side of the current document with a button prompting you to Enable Sanity AI Assist.

Click the Enable Sanity AI Assist button to create a token and enable AI Assist for everyone accessing the project.

You will find a new API token entry for your project named “Sanity AI” has been created in your project's API settings, which you can examine at sanity.io/manage.

AI Assist will now work for any dataset in your project.

At this point, AI Assist should be operational and ready for your perusal. You might want to take a detour and check out the article linked below for a closer look at where and how you can interact with the assistant in the Studio interface, or keep reading to learn more about how AI Assist works.

By default, unless otherwise specified, AI Assist is enabled for all compatible fields and document types. We will look at how you can selectively exclude fields or document types from being affected by the assistant further on in the article.

AI Assist can use most fields in your schema as a context in an instruction.

These are the field types it can write content to, including custom schema types based on the following:

• String and text
• Objects and the fields within them
• Arrays with inline objects and references
• Portable Text, including formatting and custom blocks
• Image assets (and image fields)
• References (requires additional configuration)

Fields and field sets that are conditionally visible or read-only can have instructions and can be written to by an instruction, as long as the field is non-hidden when the instruction is initiated.

There are some field types that AI Assist can use as context but not write content for:

• Booleans
• Numbers
• Slugs
• URLs
• Date and DateTime
• Geolocation
• Cross Dataset References
• File assets

AI Assist can create assets for images configured with a prompt field.

Image generation can be done directly using the Generate image from prompt command on the prompt field or indirectly whenever an AI Assist instruction modifies the image prompt field.

To enable image generation for an image field, you must:

• Set options.aiAssist.imageInstructionField to a child-path relative to the image
• Have a string or text field that corresponds to the imageInstructionField path

This will add a "Generate image from prompt" instruction to the image prompt field. Executing this instruction will generate an image.

defineType({
  type: 'document',
  name: 'article',
  fields: [
    defineField({
      type: 'image',
      name: 'articleImage',
      fields: [
        defineField({
          type: 'text',
          name: 'promptForImage', 
          title: 'Image prompt',
          rows: 2,
        }),
      ],
      options: {
        aiAssist: {
          imageInstructionField: 'promptForImage',
        },
      },
    })
  ]
})

An image will be generated each time an AI Assist instruction modifies the image prompt field. This modification could come from a document instruction, an instruction for the image field or parent object, or directly on the image prompt field.

In addition to generating images from a prompt field, the assistant can also be set to generate descriptions from image assets. To enable the assistant to auto-generate descriptions that can be used for alt text or captions, supply a valid field path in options.aiAssist.imageDescriptionField.

defineType({
  type: 'document',
  name: 'article',
  fields: [
    defineField({
      type: 'image',
      name: 'articleImage',
      fields: [
        defineField({
          type: 'text',
          name: 'alt',
          title: 'Alternative text',
          rows: 2,
        }),
      ],
      options: {
        aiAssist: {
          imageDescriptionField: 'alt',
        },
      },
    })
  ] 
})

This will add a Generate Caption instruction to the caption field that will produce a description of the image.

To work with a reference field, the AI Assist plugin must consult an embedding index that includes the types it will refer to. To learn about embedding indexes and how to set them up, visit this article.

You can manage your indexes directly in the studio using the Embeddings Index Dashboard plugin. Once you have an index configured, you can enable reference fields for AI Assist by setting options.aiAssist.embeddingsIndex to whatever you named your index.

import { defineField } from 'sanity'

defineField({
  type: 'reference',
  name: 'articleReference',
  title: 'Article reference',
  to: [{ type: 'article' }],
  options: {
    aiAssist: {
      embeddingsIndex: 'all-our-stuff-index'
    },
  },
})

Reference fields with this option set can have instructions attached and will be included when running instructions for object fields and arrays. An example instruction might look like this:

Given <Document field: Title> suggest a related article

AI assist will use the embeddings index, filtered by the types specified in the field declaration, to look up contextually relevant references. One or more references can be added for arrays or portable text fields with references.

AI Assist defaults to inclusivity and will target every supported field it comes across. This may not always be desirable, so it comes with the option to exclude fields and document types selectively by setting the options.aiAssist.exclude option to true.

// disable AI assistance wherever it is used,
// ie: as field, document, array types
defineType({
 name: 'policy',
  type: 'document',
  options: {
    aiAssist: {exclude: true}
 },
  fields: [
    // ...
  ]
})

// disables AI assistance for the specific array member
// if all types in the `of` array are excluded, the array type is also considered excluded
defineType({
 name: 'myArray',
  type: 'array',
  of: [
    defineArrayMember({
      type: 'someType',
      options: {
        aiAssist: {exclude: true}
     }
  })
  ]
})

// disables AI assistance for the specific array member
// if all types in the `of` array are excluded, the array type is also considered excluded
defineType({
 name: 'myArray',
  type: 'array',
  of: [
    defineArrayMember({
      type: 'someType',
      options: {
        aiAssist: {exclude: true}
     }
  })
  ]
})

It’s worth spending a moment examining how AI Assist deals with conditional fields. Any field that has hidden or readOnly set to true when the relevant instruction starts running will be skipped. An important word in that previous statement is “starts.” If a field has its hidden or readOnly value changed while the assistant is doing its thing, the new value will not be considered for that running process, even if the assistant has yet to reach that field.

Fieldsets with hidden and readOnly states are also accounted for.

This plugin adds an AI Context document type.

If your Studio uses Structure Builder to configure the studio structure, you might have to add this document type to your structure.

The document type name can be imported from the plugin:

import {contextDocumentTypeName} from '@sanity/assist'

// put into structure in structure  
S.documentTypeListItem(contextDocumentTypeName)

There are limits to how much text the AI can process when processing an instruction. Under the hood, the AI Assist will add information about your schema, which adds to what's commonly called “the context window.”

If you have a very large schema (many document and field types), it can be necessary to exclude types to limit how much of the context window is used for the schema itself.

We recommend excluding all types that would rarely benefit from automated workflows. A quick win is typically to exclude array types. It can be a good idea to exclude most non-block types from Portable Text arrays. This will ensure that the Sanity Assist outputs mostly formatted text.

This version of the feature uses OpenAI.com as a third-party sub-processor. Sanity's security team has vetted its security posture and approved it for use.