← All Guides

Save This

Prep Your Site for the Bot-Reader Era

Bots just passed humans online, and your customers now ask AI instead of visiting your site. The real checklist to make sure AI can find, understand, and recommend your business, plus the robots.txt block to paste and the in-person play AI can't touch.

A free guide by New Money School

Save This

Bots just passed humans online for the first time. Your customers are asking AI instead of visiting your site. Here is how to make sure AI can find you, understand you, and recommend you.

The Source

Cloudflare Radar: live bot vs. human traffic ↗

For the first time in the internet's history, bots have passed humans. Cloudflare's data shows roughly 57% of web traffic is now machines and about 43% is people. A big chunk of that is AI reading the web to answer questions inside tools like ChatGPT, Claude, and Perplexity.

Here is what that means for you: more and more, people do not visit your website. They ask an AI, the AI reads your site for them, and it reports back. So you cannot only build your site for humans anymore. You have to make sure AI can find your business, understand it, and recommend it. This is the checklist to do exactly that.

One Quick Reality Check

"Bots passed humans" is about the number of requests, not attention. People did not leave the web. Your website actually matters more now, because it is the source the AI reads about you.

Step 1Make Sure AI Can Even Reach You

A lot of sites quietly block AI without knowing it. If AI cannot read your site, it cannot recommend you. So check this first.

Go to yourwebsite.com/robots.txt in any browser. This is a little file that tells bots what they are allowed to read. Look for lines that say Disallow: / under names like GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, or Google-Extended. If those are blocked, AI tools may be shut out of your site.

The "let the answer bots in" block

This is the safe starting point. It lets in the bots that answer live questions and can cite you. Hand this to whoever manages your site, or paste it into your robots.txt file.

Copy

User-agent: OAI-SearchBot Allow: /

User-agent: ChatGPT-User Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Claude-SearchBot Allow: /

User-agent: Claude-User Allow: /

The Honest Tradeoff

There are two kinds of AI bots. Answer bots (the ones above) fetch your page to answer someone's question and can send credit back to you. Training bots (like GPTBot and ClaudeBot) vacuum up content to train models with no guaranteed credit. If your goal is to get recommended, let the answer bots in. If you also run Cloudflare, check Security, then Bots, and make sure its one-click "block AI" setting is not silently shutting out the bots you actually want.

Step 2Make Your Site Easy For AI To Understand

AI likes pages that clearly answer real questions. Clever and vague hurts you here. Plain and clear wins.

Do This

Say Exactly What You Do

On your homepage and About page, plainly state what you do, who it is for, and where you are. AI reads this literally. Example: "We are a family-owned bakery in Austin, Texas making custom wedding cakes for couples." That one sentence does more than a page of pretty branding.

Do This

Build A Real FAQ Page

Write out the actual questions customers ask: "Do you ship nationwide?" "How much does it cost?" "Do you offer same day?" Use the question as a heading, then answer it in one to three plain sentences right under it. This is exactly the format AI pulls answers from.

Do This

Use Clear Headings And Add Schema

Name your sections what they are. "Pricing" beats "Investment." Then add schema markup, which is hidden labels that tell AI your business name, address, hours, and product info. You do not have to code it. On Squarespace, Shopify, and Wix a lot is built in, and on WordPress a free plugin like Yoast or RankMath adds it. Ask whoever runs your site to turn on LocalBusiness, FAQ, and Product schema.

Step 3Get AI To Actually Recommend You

AI trusts what the wider web says about you, not just your own claims. So the work is partly off your own site.

  • Make your business info identical everywhere. Name, address, phone, hours, and website must match across Google Business Profile, Yelp, Instagram, Apple Maps, and any directories. Mismatches make AI distrust you and skip you.
  • Get and keep reviews. AI leans hard on third-party reviews and ratings when it decides who to recommend. Google and Yelp reviews are gold here.
  • Get mentioned on sites you do not own. Local press, "best of" roundup lists, partner sites, podcasts. Being named somewhere other than your own page is one of the strongest signals that you are real and worth recommending.

Skip The Hype On This One

You may hear about an llms.txt file as a magic fix. It is a real idea, but as of 2026 most major AI crawlers ignore it and there is no proven traffic from it. Do not waste your time there until you have nailed robots.txt, clear pages, consistent info, and reviews. Those are what actually move the needle.

Step 4Test It This Week

This is the satisfying part. Go check your own AI visibility right now. Open ChatGPT, Claude, Perplexity, and Google's AI and ask the questions a customer would ask:

  • "Best [your category] in [your city]?"
  • "Who does [your service] near me?"
  • "What does [your business name] do, and are they any good?"

Then grade what comes back. Do you show up at all? Is the info correct? Are competitors named and you are not? Fix based on the answer:

  • Not showing up at all? Check that robots.txt is not blocking AI, and go earn more reviews and third-party mentions.
  • Showing up with wrong info? Fix your About page, your Google Business Profile, and your directory listings so the facts match everywhere.
  • Competitors winning? See where they get cited (reviews, roundup lists) and go earn the same.

Re-test every few weeks. One heads up: AI answers change run to run, so do not panic over a single result. Look at the pattern.

The Other HalfBuild What AI Can't Touch

Here is my second take, and honestly the one I am betting on most. As the internet fills up with AI, the things AI cannot copy get more valuable. The biggest one is real, in-person community. People are starving for it. We spend less time with friends than we used to, and most of us feel it.

You can already see brands leaning in. Run clubs are exploding, with brands like Fulton sponsoring city running crews. "Coffee raves," where a coffee shop throws a morning DJ set, are selling out by the thousands. Brands are hosting real-world activations instead of just running more ads.

You do not need a rave. You need one real recurring in-person thing: a workshop, a meetup, a class, a monthly coffee. It builds the loyalty and word of mouth no AI can generate. And here is the bonus: those events create the local mentions and reviews that also boost your AI visibility from Step 3.

The Real Win

Optimize everything online for AI as hard as you can, and at the same time build in-person community that AI cannot touch. That combination is how you stay found and stay loved while the web fills up with machines.

Keep the momentum going

There are 300+ more free guides covering AI for your career, your business, and your life. Find the next one worth your time.

Browse All Guides →