Discussion about caching issues and image optimization in Gatsby and Netlify site, with a mention of search functionality.

Hi everyone. I have a sanity + gatsby + netlify site and the client is constantly complaining that when they update the site, the content doesn’t appear. If I go to the site and hit refesh the content appears. How can I ensure that the content is always fresh? This caching stuff confuses the hell out of me

are you setting custom headers on Netlify?

Nah it’s all default

What is your site again? Perhaps it’s something weird with the header

mmm headers looks fine

do you have gatsby-plugin-offline installed?

I did at one point but I scrapped it in the end for this reason

Are you using the CDN or got that disabled?

It’s been a non stop complaint from them, not an issue I’ve had before with the same set up, I don’t know if their expectations of the site updating instantly are unrealistic

Disabled

gatsby-plugin-offline install workers IIRC — you might have to do something extra to remove installed workers, not just removing the plugin

Might be right. I still have the worker installed from when I helped you out a while back.

When I go to application / service workers, I can see your old gatsby-plugin-offline worker that was installed the first time we talk (3/17/2021)

here’s a handy snippet: https://developer.mozilla.org/en-US/docs/Web/API/ServiceWorkerRegistration/unregister#example

Do you think that could be causing the cache issues then? Like I say, when I visit the page myself I have to refresh the site to see new content

I’ll try de-registering it and see if that works! 🤞

Client from hell this one haha

In chrome you can disable the service worker to check.

There's a bypass toggle.

I think its worth a try. IIRC gatsby offline plugin default setting is to retrieve data from cache first, then network requests. Not sure if that had changes. Since you had already uninstalled the plugin, might as well remove it completely.

Client from hell definitely haha they seem very picky

might also want this

gatsby-plugin-remove-serviceworker

netlify’s default caching setup is very good (must-validate or whatever) and has eliminated most all of our caching madness

Yeah I did some reading into it and found that to be the case, which made my head hurt even more that the client was still having issues

Hoping what and suggested will solve it! Just back from walking the dog so I’ll try it now

could you check again to see if it’s unregistered for you too? https://dscvrd.co/ It’s gone for me

Service worker is gone

What’s the reasoning behind the image proxy? https://images.weserv.nl/?url=https://cdn.sanity.io/images/lylk5ufs/production/79bf8804f548c6d6d125edf0342186b1bf0e6fd1-959x960.jpg?w=1080&h=1080

it’s very slow

Thanks Raffi. I just added a demo article and it literally updated immediately, so I’m hoping that’s sorted now

And serving large pngs https://images.weserv.nl/?url=https://cdn.sanity.io/images/lylk5ufs/production/13f452711244e974bcb5cb9faea79fccc61843d6-1200x1035.png?w=1080&h=1080

The images, they were going like 4x over their bandwidth on Sanity which was charging me, so I tried offloading this to another service

But yeah I need to look at this as they’re crazy slow

nice job. I think these service workers are one of the most dangerous “easy to enable” features of Gatsby.

we did it because… why not? Oh, that’s why.

100% chris

Ah ok. Then you couldn’t use the auto format either.

If there’s any way I can reduce the bandwidth you know, I’m all ears as I need to sort this haha

are you using the sanity image stuff that requests smaller files with different compression settings? I’m selfishly interested as we recently had an editor use a multi MB PNG for a share image and services wouldn’t use them because it went through untouched

Well with sanity you can call auto=format which would make that image https://cdn.sanity.io/images/lylk5ufs/production/13f452711244e974bcb5cb9faea79fccc61843d6-1200x1035.png?w=1080&h=1080&auto=format

I was originally using gatsby image with gatsby-source-sanity which was giving me all the sizes I needed, but I had to refactor the site to grab the data at load time instead of build time so it was faster for the client publishing articles so I had to look at other ways of handling the images

280kb

can I append that to the weserv.nl CDN or would you suggest just serving them straight from sanity?

I sort of assumed images came from sanity directly with their resizing CDN, but the couple sanity-image plugins do a lot of this behind the curtain

You can’t use a proxy as it’ll server webp in most browsers so will break non webp browsers.

we use this, the developer is in this Slack and is super helpful:

gatsby-plugin-sanity-image

had to rebuild the functionality as an spa within gatsby because his client didn’t want to wait for the rebuilds. I think.

Yeah that’s right… it was a total nightmare haha

I’ve built a lot of sites in Gatsby this past year and never had any issues like this, even got the build time down to like 1m 30s

I’ve come close to giving up the freelance life because of this project 😂

Which functionality did you rebuild as a spa ?

I’m not sure what spa stands for exactly, but I guess it means grabbing data at load time? It was the articles themselves, so the index page, the category page and the article data itself

Single page app.

Things like the header banner, the ‘stories’, the ads, etc, they all come at build time still but as they’re not as time sensitive it was a good trade-off

If you followed through with the reach router approach then that's what I'd call it.

Yeah I did, the reach router thing worked really nicely in the end

Wow, thats a bummer

It's unfortunate that your client didn't understand the concept or benefits of your original approach.

Together with offline and image stuff you've probably throw away 75% benefits of Gatsby hahaha

Yeah exactly my thoughts

Wow, glad you got it working. Is it too late for gatsby cloud?

I’ve learned to make sure I document how gatsby/etc actually work before committing to building anything haha

Interestingly I was talking with a colleague today and he's got a whole page in a proposal about the change in mentality for using a headless CMS and the build process.

I think so

It’s a massive deal , non-technical clients who have come from wordpress really don’t understand it

Yeah. Sounds like they were used to a traditional approach & had a hard time adjusting

Yeah remember you saying in our long thread the other week.

Site still looks great.

So the outcome to a user apart from image load speed is really good.

Thank you 👏 it means a lot as I’m really at my wits end with it

separate thing, we need some highly dynamic routes due to rapid data changes, but we still need the opengraph tags generated at request time (for crawlers that probably aren’t rendering the JS). Is there a solution to this that essentially amounts to SSR at request time?

liek I’m in your boat now wishing I jsut had a plain express app or something

Thats solution is called nextjs I think

didn’t gatsby bring out something similar to that recently?

Agree w Raffi, your site looked great dude. Did your client give you as much of a difficult time when you were designing it?

well we’d have this but I don’t think it solves for rendering the meta at request time. I’ve even considered a hacked up function that swaps in the few meta tags and hope I can get it to hydrate gatsby correctly If you’re a regular browser https://www.gatsbyjs.com/docs/data-fetching/#fetching-data-at-client-side-runtime

Generally they were onboard the whole way, in fact they were really happy with them. However they dropped it on me yesterday that they need a search function, but it was never in the designs they signed off, but they said they mentioned it in their first email which they did and I didn’t see it… so it’s like this awkward situation about whether I build it as part of the original cost or not 😕

Something that helps is a sitemap when signing off scope with small clients.

Yeah something I’m learning. I started freelancing just over a year ago and generally clients have been really great, but the odd few have made me a sceptic. Looking at written agreements, signed by both parties before any work starts now

Welp lucky you sanity just release a few groq goodness that would really help w implementing search

Hope you sort that pricing thing out

Thanks it’s all a big learning curve

I watched the sanity seminar about that actually but not sure where I would start looking to get some ideas towards building a search… can you link me to anything? Last favour haha

Good search is hard. You'll get into the arguement about whats good relevancy for listing the results.

Although in this you can probably do text search with a topic filtering perhaps.

Or just settle on text search.

Did the client signoff designs?

It's pretty evident that search is mising and they should have picked up when the site launched.

I posted an example of searching with the new groq functions here https://twitter.com/DerekNguyen10/status/1375329869752455169?s=20

You can use

boost()

to add weight to more important parameters

They did sign off the designs that’s my side of the “argument” — I agreed I’d do the search but it would have to be in my own time (ie. evenings) and now I’m just getting pressured about it non-stop

Thanks I’ll take a look at this

They better pay extra for it as well!

😂 they ought to be!

Deliver search, but don't think too much about it then. Best of luck.

Yeah, my thoughts exactly. Keep it as simple as it can be 🙂 thanks for the help everyone