Our crawler

HomePortfolioBot

HomePortfolioBot is HomePortfolio's crawler. This page says what it fetches, how often, and how to stop it.

How it identifies itself

Every request it makes, robots.txt included, carries this user agent:

HomePortfolioBot/1.0 (+https://homeportfolio.com/bot)

What it fetches

Published information, for these purposes, and nothing else:

  • The brochures, floor plans and specifications that property listings link to: the files themselves (PDFs and floor plan images), so that their text can be read and searched. It asks only for the documents those listings link to.
  • An independent school's published fees: the school's home page, its sitemap, and the fees pages and fee schedules it links to, at most 14 requests a visit and a visit about every 60 days.
  • The Independent Schools Inspectorate's published inspection reports: its sitemap, each school's page on the Inspectorate's site and the reports that page lists.
  • Councils' published school admissions documents: allocation reports and the distance or score of the last place offered, and grammar schools' published qualifying scores.
  • School catchment areas: the map layers councils publish of their catchment boundaries.
  • The Department for Education's published school census, school performance measures and school-level applications and offers, and its register of schools' public download files.
  • Ofsted's published school inspection outcomes and their history.
  • The school registers and school statistics that the Welsh and Scottish Governments and Northern Ireland's Department of Education publish.
  • The Bank of England's published quoted mortgage rates.

It never submits forms or reads anything behind a login.

How often

  • One request to a site at a time, and at most one request to a site every 5 seconds.
  • Listing documents mostly at night, between 23:00 and 06:00 UK time, and also during the day; the schools and rates data on their own daily, weekly, monthly or yearly schedules.
  • A document it has read is checked again about every 30 days, or once a year for an inspection report, with a conditional request, so an unchanged file costs a short "not modified" reply.
  • It runs from HomePortfolio's own servers, never through a proxy.

robots.txt

It reads your robots.txt before it asks for anything on your site, and follows it, with one exception: public documents of a few kinds (property brochures, floor plans, specification sheets, independent school fee pages, council school admissions reports and catchment pages, and school statistics published by government), which it fetches under a recorded HomePortfolio decision although the robots.txt of the site holding them disallows them. Each decision covers one site and a recorded pattern of addresses on it, and names the exact robots.txt rules it overrides, as published on the day of the decision. It lapses as soon as that robots.txt says anything more about those documents: a new rule, or a group naming HomePortfolioBot.

To stop it everywhere on your site, those documents included, add:

User-agent: HomePortfolioBot
Disallow: /

A group in your robots.txt naming HomePortfolioBot always decides what it may fetch, with no exception. If your robots.txt cannot be read because of a server error, it asks for nothing on your site until it has been unreadable for 30 days.

If your site refuses it

An answer of 401, 403 or 429, or a challenge page, stops it asking your site for 7 days, and a school website that refuses it is not asked again for 60 days. It never retries a refusal under another name or from another network, and it never solves or works around a challenge.

Contact

To ask it to stop, to slow it down, or to report a problem, write to [email protected].