OppsightBot
The web crawler that helps Oppsight keep business contact details accurate.
What it does
OppsightBot visits a business’s own website to read the contact details that business has chosen to publish: its main phone number and email, and the names and job titles of the people it lists on its team, about or contact pages. Oppsight members use these details to reach businesses for work purposes.
How it behaves
- It identifies itself as
OppsightBot/1.0 (+https://oppsight.ca/bot). - It follows robots.txt on every page. If your robots.txt disallows it, or disallows all bots, it reads nothing.
- It reads at most seven pages per site: up to four (such as home, about, team and contact), a careers page when there is one, the terms or legal page when the site gives email addresses, and, for a business in a group whose staff are listed on each location’s own website, that location’s team page (only when that site’s robots.txt allows it). One page at a time, and only pages anyone can open without signing in. It never fills in forms.
- If a site says it doesn’t want unsolicited messages, or doesn’t allow its contact details to be used for commercial purposes, it keeps no email addresses from that site, and any already on record are withheld. When a site gives email addresses, it also reads the site’s terms or legal page for such a notice. Oppsight members see “Email not shown” on that business, with a link to the site’s notice.
- It keeps business contact information only. It doesn’t collect home addresses, personal social media or anything behind a login.
- It returns from time to time, so changes on your site reach Oppsight.
How to opt out
Add this to your site’s robots.txt:
User-agent: OppsightBot Disallow: /
To have details we already hold removed or corrected, or for a person to ask that their name and work details not be shown, email [email protected]. We act on requests within 30 days.