MotoMinerBot

MotoMinerBot is MotoMiner's web crawling bot (also known as a "spider"). Crawling is the primary method we use to add vehicle detail pages to the MotoMiner index. It should be noted that MotoMinerBot primarily crawls auto dealership websites and adds their vehicle detail pages to the MotoMiner index, so shoppers can more easily discover vehicles for sale on the World Wide Web. MotoMinerBot does not crawl vehicle listing/classified sites such as AutoTrader, Cars.com, CarGurus, Craigslist, etc.

Since November 1, 2014, MotoMinerBot has indexed 1,430,389,317 vehicle detail pages on the World Wide Web.

Vehicle detail pages we index are searchable via MotoMiner's search engine. Search results present the user with information about each indexed vehicle and its associated vehicle detail page (VDP). If users want to know more about a vehicle, they can click the vehicle's deep link to navigate to the vehicle's detail page. We believe this process is the right thing to do for the seller and for the shopper.

We use many computers to fetch (or "crawl") millions of web pages. MotoMinerBot has a proprietary process, which it uses to identify sites to crawl, how often, and which pages contain relevant vehicle classified data to add to our index. New sites, changes to existing sites, new vehicle page links, and dead vehicle page links are noted and applied to the MotoMiner index on a periodic basis.

MotoMinerBot is a Respectable Search Engine Crawler

• We play by the rules and follow the robots.txt protocol.
• The crawler is bandwidth-friendly to your website:
  • supports gzip compression
  • does not load scripts
  • does not load unnecessary images
  • crawls pages identified as a vehicle detail page
  • creates a cached thumbnail of a vehicle's primary image
• We take into account the crawl delay specified in your robots.txt file.
• Our user agent carries the name of the web crawler, so you can target it by name.
• Our user agent carries the URL of this web page (motominer.com/Bot).
• On this page, we publish all MotoMiner IP addresses to allow targeting our crawler directly (see below).

Website Providers

Your dealers want to be indexed by MotoMiner, and we understand that our crawler puts extra load on your data center. We want you to know that we have innovative technology called The MotoMiner Traffic Cop in place that can ensure we don't exceed defined traffic thresholds. This technology also throttles and disperses outbound requests.

If you would like to learn more or would like to set up traffic thresholds for your data center, drop us a line via our online contact form.

Controlling How MotoMinerBot Interacts with Your Website

We will not index anything you would like to remain private. If you want to prevent MotoMinerBot from crawling content on your site, you have a number of options.

Before you block MotoMinerBot, please consider the following: our shoppers will not find your vehicle detail pages if you block MotoMinerBot!

One option is to use the robots exclusion standard.

Another option is to use the noindex meta standard. We respect global or bot-specific directives.

MotoMinerBot also takes into account the crawl-delay directive if you want to limit the number of crawl requests we may send to your site per second.

How can I verify MotoMinerBot is really MotoMinerBot?

You can verify that the MotoMinerBot visiting your site is legitimate by checking if the IP address of the bot matches the IP address below. While our IP address may change in the future, our current IP address is the following: