Best practices for creating/updating multiple documents in Sanity using JS client and p-throttle
8 replies
Last updated: Apr 7, 2020
F
Let's say I need to create/update 100+ documents. Is there a best practice? Tried a naive approach to just loop through the data, and do a
client.create()for each entry, but then things quickly starts to fail. Using the JS client.
Apr 7, 2020, 5:22 PM
F
Works with 10-20 requests at a time (hence the
slice(), but starts to fail when increased to about 30 simultaneous requests.
Apr 7, 2020, 5:55 PM
I'd suggest using a throttler such as `p-throttle`:
npm install p-throttle
const pThrottle = require('p-throttle') const persistSpeaker = pThrottle( // Define the function to be called when ready speaker => client.createOrReplace(speaker, {visibility: 'async'}), // Max 20 requests 20, // Within a 1 second window 1000 ) Promise.all( speakers .map(transformSpeaker) .map(persistSpeaker) ) .then(console.log) .catch(console.error)
Apr 7, 2020, 6:51 PM
You could also use a transaction, but if you've got 100+ speakers I'm not sure I would suggest it - one should try to keep the size of the transaction payload below a reasonable size (< 500kB perhaps?)
So kind of depends on the size and number of those documents.
So kind of depends on the size and number of those documents.
Apr 7, 2020, 6:52 PM
// Use a transaction (not great for a large number of documents) speakers .map(transformSpeaker) .reduce( (trx, speaker) => trx.createOrReplace(speaker), client.transaction() ) .commit() .then(console.log) .catch(console.error)
Apr 7, 2020, 6:53 PM
Even better, combine the two approaches, batching the speakers up into groups of a certain size (10 in this case) and doing transactions for them:
const pThrottle = require('p-throttle') const persistSpeakerBatch = pThrottle( // Define the function to be called when ready batch => batch.reduce( (trx, speaker) => trx.createOrReplace(speaker), client.transaction() ), // Max 20 requests 20, // Within a 1 second window 1000 ) let batch = [] for (let i = 0; i < speakers.length; i++) { batch.push(speakers[i]) if (batch.length === 10 || (i === speakers.length - 1 && batch.length > 0)) { persistSpeakerBatch(batch) batch = [] } }
Apr 7, 2020, 7:07 PM
F
Thanks for clarifying! š I did manage to get it to work with the
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
š
throttled-queuelibrary, but
p-throttlelooks a bit nicer, and has more users so think I will replace it. š
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
š
Apr 7, 2020, 7:09 PM
F
I also looked at the
transaction()API to see if I could use that, but didn't figure out how to reduce the list into it, so really cool to see that example. š
Apr 7, 2020, 7:11 PM
Sanityā build remarkable experiences at scale
Sanity is a modern headless CMS that treats content as data to power your digital business. Free to get started, and pay-as-you-go on all plans.