Study of Web Crawler

Siddharth Bhandari; Simal Sethi; Suyog Gune

A web crawler is a relatively simple automated program, or script that methodically scans or crawls through Internet pages to create an index of the data it's looking for; these programs are usually made to be used only once, but they can be programmed for long-term usage as well. There are several uses for the program, perhaps the most popular being search engines using it to provide webs surfers with relevant websites. Other users include linguists and market researchers, or anyone trying to search information from the Internet in an organized manner. Alternative names for a web crawler include web spider, web robot, crawler and automatic indexer. Crawler programs can be purchased on the Internet, or from many companies that sell computer software, and the programs can be downloaded to most computers.

Select Volume / Issues:

• Volume 4 Issue 3, May. 2013

Year:

2013

Type of Publication:

Article

Keywords:

Crawlers; Frontier; Seed URL; Web Search Engines

Journal:

IJECCE

Volume:

4

Number:

3

Pages:

737-739

Month:

May