Also called a web crawler, an internet spider is a form of online bot that trawls the internet for specific information mostly to index the content of websites for search engines.