MorphBot: The Snake Crawler

Why is MorphBot crawling my page?

MorphBot is the MorphMarket web crawler which visits breeder websites nightly to get updates of their inventory. By crawling your page, MorphMarket displays your latest inventory without you having to manually update anything else besides your website.

MorphBot is a polite bot:

  • It respects robots.txt (the way a website owner defines which bots can crawl a site, where and how)
  • It crawls sites slowly and during off-peak hours so as to not disturb normal traffic.
  • It crawls only select portions of a site and only as necessary. Most nights it only crawls the top level "listing" pages rather than drilling into each of the item pages. This is usually only a dozen or so pages.

If you have any concerns about MorphBot, please contact us.

I want MorphBot to crawl my website; how can this happen?

If you are a large reptile breeder who maintains an accurate inventory on your website, MorphBot may be able to crawl your site. Feel free to contact us and ask.