MorphBot: The Snake Crawler
Why is MorphBot crawling my page?
MorphBot is the MorphMarket web crawler which visits breeder websites nightly to get updates of their inventory. By crawling your page, MorphMarket displays your latest inventory without you having to manually update anything else besides your website.
MorphBot is a polite bot:
- It respects robots.txt (the way a website owner defines which bots can crawl a site, where and how)
- It crawls sites slowly and during off-peak hours so as to not disturb normal traffic.
- It crawls only select portions of a site and only as necessary. Most nights it only crawls the top level "listing" pages rather than drilling into each of the item pages. This is usually only a dozen or so pages.
If you have any concerns about MorphBot, please contact us.