-- Rubik Digital, a UK web design, development and SEO agency, published original research on 27 October 2026 analysing 1,440,971 crawler requests to 17 live business websites it hosts and manages, recorded over 14 days from 28 September to 11 October 2026. The study began after AI crawlers slowed down one of those websites, prompting the agency to measure how AI companies and search engines were accessing the sites it manages.
The analysis found that AI crawlers accounted for 40.4% of all crawler requests, or 582,806 of the 1,440,971 total, compared with 28.5% for traditional search engines combined. AI crawlers made 28.7 times as many requests as one major search engine's crawler across the sample, a figure that fell to 8.7 times once the three busiest sites were removed, and outnumbered that same crawler on almost nine in ten of the 17 sites studied. The full findings are set out in the AI crawler research published by Rubik Digital.
Within the AI crawler traffic, one operator accounted for 57% of requests and another for 38%, with the remaining providers sharing 5%. Two-thirds of AI crawling was for search and answers; one third was for training. Two AI assistants fetched pages live for users about 940 times a day across the sample. Server errors occurred in 1.6% of AI crawler requests. The method counted page requests only, from crawlers that identified themselves by user-agent name, and did not use any visitor data; only combined figures across the 17 sites were published.
Giedrius Kudzinskas, CTO and Head of Digital at Rubik Digital, said: "AI crawlers don't behave like the search crawlers businesses have spent twenty years optimising for. If your important content only appears after JavaScript runs, many AI tools may never see it. That's now a visibility problem, not just a technical one."
Rubik Digital offers AI search visibility services that track business mentions, citations and accuracy in AI-generated answers, with monthly reporting and work to address the findings. More information on the agency's web design, development and SEO work is available at Rubik Digital.
About Rubik Digital
Rubik Digital is a UK web design, development and SEO agency founded in 2010 and based in Milton Keynes. It builds, hosts and grows websites and platforms for businesses in finance, travel, healthcare, professional services and other sectors, with a focus on search and AI visibility.
Contact Info:
Name: Giedrius Kudzinskas
Email: Send Email
Organization: Rubik Digital
Website: https://www.rubikdigital.co.uk/insights/ai-crawlers-vs-googlebot-research
Release ID: 89208217
Google
RSS