Autoblogging.ai
Day 28 after signup
I re-ran my AI crawler study three weeks later. Here is what changed
Why it works
- Lead with a timely update to a previous study, then make the new findings easy to scan with specific comparisons and figures.
- Translate the findings into a short list of practical recommendations before introducing the promotional offer.
- Keep the offer distinct from the editorial section so readers can understand both the useful content and the commercial next step.
How it is built
- Personal study introduction
- Updated research findings
- Practical recommendations
Email copy
Hi, A few weeks ago I ran a study across 517 websites in our network to see what AI crawlers actually read: the Markdown (.md) copies of pages, llms.txt, brand.json and a few others. In the first week (17 to 23 October), AI crawlers asked for the .md files 50,158 times and for llms.txt only 24 times. This week I checked on it again (2 to 8 November). Here is what changed: AI crawlers asked for the .md files 64% less: 18,221 times. Amazonbot went from almost 14,000 requests to 225. They did not go away. Their total crawling of the same sites went up 19%. They went back to reading the normal web pages. When ChatGPT, Claude, Perplexity and DuckDuckGo open a page to answer someone, they read the normal page: 55,310 times in one week, and a .md file only 42 times. llms.txt is still ignored by AI: 23 requests out of 5,120. Most came from SEO tools. I also tested a structured "knowledge bundle" on 50 sites against 50 similar sites. AI crawlers found every bundle within a day, but it made no measurable difference to how much they crawled. My reading: AI crawlers make one big pass when Markdown copies first appear, then mostly come back for the pages themselves. What I am taking from it: Make the actual web page easy to read: a clear answer near the top, real headings, little clutter. That is what AI assistants open when they answer. Markdown copies help AI discover your content once. They are not a lasting channel. Keep llms.txt if you like, but do not count on it. The full study, with the daily chart and every number, is here: rankera.ai One more thing: I have extended our Diwali offer by 24 hours. You can still lock in 30% off our monthly plans (20% off annual) on Autoblogging, Rankera and CTRTool, and the price stays the same for as long as you keep the plan. It now ends Tuesday 10 November at 6:26 pm UTC (10 Nov, 11:56 pm IST). Your billing page shows the lower prices: dash.autoblogging.ai/billing. If you are already on a plan and want to move up, reply to this email and I will set it up. Thanks for reading, Vaibhav You are getting this because you have an Autoblogging.ai account. Unsubscribe