OpenWebSpider

Project Information

Platform: Unix (Linux) , Windows
Price: Free, open source, GPL

Features

• Developer tool: requires MySQL server, C/C++ compiler, web server (Apache suggested), PHP on site.
• Spider crawls web sites via HTTP
• Option to follow links on additional hosts
• Indexes full text and can cache text
• Calculates a page's ranking among indexed sites
• Defaults to matching any search term, supports +, -, quotes, left truncation with * and relevancy adjustment with > and <
• Results in relevance order
• Shows match terms in context
• Can display relevance number, page level, rank and cache link

Examples

• Allcoins.org - numismatic portal
• OSLookup.org - directory of the open source community