Our crawler
Our research crawler: what it reads, and how to keep it off your site.
If OverdeskResearch/1.0 is in your logs, it is Overdesk’s research crawler. It reads the public pages and DNS records of software companies, to learn which help desk, billing and code hosting tools they use.
On WordPress.org, it reads a plugin’s public support threads and release list, to summarise what the plugin’s users are asking about. It runs when someone on our team starts it, not on a schedule, and it reads a handful of pages from each company’s site. For a company our team may write to, it also reads the contact page and, if that lists no shared address, the home page, for a shared address the company publishes on its own domain, such as support@, help@ or info@. It never records a person’s own address.
robots.txt
It reads your robots.txt first, and follows it.
Before it reads a page, it reads your robots.txt. A group for OverdeskResearch applies to it; without one, the group for every crawler (User-agent: *) does.
To keep it off your whole site, add this to robots.txt:
User-agent: OverdeskResearch
Disallow: /
It honours Crawl-delay. Set a Crawl-delay over 30 seconds and it skips your site.
One host is the exception: api.wordpress.org, whose robots.txt disallows all crawlers. The crawler does not apply that robots.txt, because the host is the WordPress.org API that WordPress itself calls to look up plugins.
For its plugin briefs, the crawler sends GET requests to api.wordpress.org/plugins/info/1.2/ and nothing else on that host, at most 60 in a run, for a plugin’s public counters.
It also reads the plugin directory’s list of popular plugins from that address, 250 plugins a request.
Pace
Slow, small, and polite.
Its requests to the sites it reads are at least half a second apart on each host, or your Crawl-delay if that is longer, and it reads at most 5 pages from a host, apart from the WordPress.org pages and API it was built to read.
On WordPress.org it reads up to 15 pages of a plugin’s support forum in a run, and the threads listed on them, as well as plugin and profile pages.
When it opens a page in a headless browser to see which support tools load on it, the page’s own files load the way they do for any visitor.
A site that answers 403, 429 or 503, or with a bot challenge, is recorded as blocked and is not asked again in that run.
It connects to public addresses only, never to a private or internal one.
Questions
How do I stop OverdeskResearch reading my site?
Add a robots.txt group for User-agent: OverdeskResearch with Disallow: /. It reads robots.txt before anything else on your site.
Does it fill in forms or log in?
No. It reads public pages, their robots.txt, public DNS records and WordPress.org’s plugin API.
Does it collect email addresses?
Only a shared address a company publishes on its own contact or home page, such as support@ or info@, for our team to write to. Never a person’s own address.
Something else? Email us.