MotoMinerBot is MotoMiner's web crawling bot (also known as a "spider"). Crawling is the primary method we use to add vehicle detail pages to the MotoMiner index. It should be noted that MotoMinerBot primarily crawls auto dealership websites and adds their vehicle detail pages to the MotoMiner index, so shoppers can more easily discover vehicles for sale on the World Wide Web. MotoMinerBot does not crawl vehicle listing/classified sites such as AutoTrader, Cars.com, CarGurus, Craigslist, etc.
Since November 1, 2014, MotoMinerBot has indexed 278,029,068 vehicle detail pages on the World Wide Web.
Vehicle detail pages we index are searchable via MotoMiner's search engine. Search results present the user with information about each indexed vehicle and its associated vehicle detail page (VDP). If users want to know more about a vehicle, they can click the vehicle's deep link to navigate to the vehicle's detail page. We believe this process is the right thing to do for the seller and for the shopper.
We use many computers to fetch (or "crawl") millions of web pages. MotoMinerBot has a proprietary process, which it uses to identify sites to crawl, how often, and which pages contain relevant vehicle classified data to add to our index. New sites, changes to existing sites, new vehicle page links, and dead vehicle page links are noted and applied to the MotoMiner index on a periodic basis.
Your dealers want to be indexed by MotoMiner, and we understand that our crawler puts extra load on your data center. We want you to know that we have innovative technology called The MotoMiner Traffic Cop in place that can ensure we don't exceed defined traffic thresholds. This technology also throttles and disperses outbound requests.
If you would like to learn more or would like to set up traffic thresholds for your data center, drop us a line via our online contact form.
We will not index anything you would like to remain private. If you want to prevent MotoMinerBot from crawling content on your site, you have a number of options.
Before you block MotoMinerBot, please consider the following: our shoppers will not find your vehicle detail pages if you block MotoMinerBot!
One option is to use the robots exclusion standard.
Another option is to use the noindex meta standard. We respect global or bot-specific directives.
MotoMinerBot also takes into account the crawl-delay directive if you want to limit the number of crawl requests we may send to your site per second.
You can verify that the MotoMinerBot visiting your site is legitimate by checking if the IP address of the bot matches one of the IP addresses below. While our IP addresses may change in the future, our current IP addresses are the following: