How it identifies itself
Every request it makes, robots.txt included, carries this user agent:
HomePortfolioBot/1.0 (+https://homeportfolio.com/bot)What it fetches
The brochures, floor plans and specifications that new-home listings link to: the files themselves (PDFs and floor plan images), so that their text can be read and searched. It asks only for the documents those listings link to. It does not crawl pages, follow links, submit forms or read anything behind a login.
How often
- One request at a time, and at most one request to a site every 5 seconds.
- Only at night, between 23:00 and 06:00 UK time.
- A document it has read is checked again about every 30 days, with a conditional request, so an unchanged file costs a short "not modified" reply.
- It runs from HomePortfolio's own servers, never through a proxy.
robots.txt
It reads your robots.txt before it asks for anything on your site, and obeys it. To stop it everywhere on your site, add:
User-agent: HomePortfolioBot
Disallow: /If your robots.txt cannot be read because of a server error, it asks for nothing on your site until it has been unreadable for 30 days.
If your site refuses it
An answer of 401, 403 or 429, or a challenge page, stops it asking your site for 7 days. It never retries a refusal under another name or from another network, and it never solves or works around a challenge.
Contact
To ask it to stop, to slow it down, or to report a problem, write to [email protected].