Hi,

A few weeks ago I ran a study across 517 websites in our network to see what AI crawlers actually read: the Markdown (.md) copies of pages, llms.txt, brand.json and a few others. In the first week (17 to 23 October), AI crawlers asked for the .md files 50,158 times and for llms.txt only 24 times.

This week I checked on it again (2 to 8 November). Here is what changed:

My reading: AI crawlers make one big pass when Markdown copies first appear, then mostly come back for the pages themselves.

What I am taking from it:

  1. Make the actual web page easy to read: a clear answer near the top, real headings, little clutter. That is what AI assistants open when they answer.
  2. Markdown copies help AI discover your content once. They are not a lasting channel.
  3. Keep llms.txt if you like, but do not count on it.

The full study, with the daily chart and every number, is here: rankera.ai

One more thing: I have extended our Diwali offer by 24 hours. You can still lock in 30% off our monthly plans (20% off annual) on Autoblogging, Rankera and CTRTool, and the price stays the same for as long as you keep the plan. It now ends Tuesday 10 November at 6:26 pm UTC (10 Nov, 11:56 pm IST). Your billing page shows the lower prices: dash.autoblogging.ai/billing. If you are already on a plan and want to move up, reply to this email and I will set it up.

Thanks for reading,
Vaibhav

You are getting this because you have an Autoblogging.ai account. Unsubscribe