Infrastructure/Search

From FedoraProject

< Infrastructure
Revision as of 09:13, 13 October 2009 by Huzaifas (Talk | contribs)

Jump to: navigation, search
Infrastructure InfrastructureTeamN1.png


Contents

Points of Contact

Project Sponsor

Name: Mike McGrath
Fedora Account Name: mmcgrath
Group: Infrastructure
Infrastructure Sponsor: mmcgrath

Secondary Contact info

Name: Huzaifa Sidhpurwala
Fedora Account Name: huzaifas
Group: Infrastructure

Name: Allen Kistler
Fedora Account Name: akistler
Group: Infrastructure

Name: Keiran Smith
Fedora Account Name: affix
Group: Infrastructure

Project Info

Project Name: Search Engine Enhancement
Target Audience: All users of Fedora web sites
Expiration/Delivery Date (required): F13

Description/Summary

Fedora needs a search engine[1]

Requirements

  • Crawl the web sites (wiki and non-wiki)
  • Search the web sites (wiki and non-wiki)

Preferences

  • Python-based (no Java)
  • Programmable keywords to have control over what pages get displayed for certain keywords
  • XML or library interface so other applications can use it

Project Plan

  1. Investigate and evaluate existing open source search engines
  2. Select candidate software
  3. Create public test instance of candidate software
  4. Test for functionality, performance, and impact (re-evaluate, if necessary)
  5. Create capacity and deployment plans
  6. Deploy

Resources Needed

  • Public Test for testing candidate software
  • Permanent home(s) for deployment
    • Web server(s)
    • Database server(s)

Software Investigation and Evaluation

In Progress

  • KinoSearch [2]
Perl port of Lucene
written in Perl
  • written in Perl or TCL on top of PostgreSQL
  • Python interface available
  • not actively maintained
  • Perl port of Lucene
  • not actively maintained
  • SphinxSearch [6] Huzaifa investigating

Not Suitable

  • DataparkSearch [7]
written in C
written in Java
written in C#
  • written in Java
  • archives content rather than simply indexing it
  • written in C++
  • not actively maintained
written in C/C++
written in C++
written in Java, but ported to others [15]
written in C
  • Requires EzMwLucene (Java, not desirable) to be running on the servers to be searched
  • EzMwLucene is wiki-only, therefore MWSearch is wiki-only
  • written in Java
  • based on Lucene
  • RigorousSearch [19]
Crawls the MediaWiki database, not the web site
Doesn't work for non-MediaWiki web sites, including any non-wiki web site
written in C++
written in C
Swish++ is a rewrite in C++
  • Terrier (TERabyte RetrIEveR) [22]
written in Java
written in C++
written in C
written in C

Public Testing

<tbd>

Deployment Plan

<tbd>

References

  1. "Fedora Search Engine". Infrastructure/Tickets. https://fedorahosted.org/fedora-infrastructure/ticket/1055. 
  2. "KinoSearch". Rectangular Research. http://www.rectangular.com/kinosearch/. 
  3. "Namazu". Namazu Project. http://www.namazu.org/. 
  4. "OpenFTS". SourceForge. http://openfts.sourceforge.net/. 
  5. "Plucene". CPAN. http://search.cpan.org/~tmtm/Plucene-1.25. 
  6. "SphinxSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:SphinxSearch. 
  7. "DataparkSearch". DataparkSearch. http://www.dataparksearch.org/. 
  8. "Egothor". Egothor. http://www.egothor.org/. 
  9. "Grub". Wikia, Inc.. http://grub.org/. 
  10. "Heritrix". Internet Archive. http://crawler.archive.org/. 
  11. "ht://Dig". The ht://Dig Group. http://www.htdig.org/. 
  12. "Indri". The Lemur Project. http://www.lemurproject.org/indri/. 
  13. "Isearch". Isite. http://isite.awcubed.com/. 
  14. "Lucene". Apache Software Foundation. http://lucene.apache.org/. 
  15. "Lucene Implementations". Apache Software Foundation. http://wiki.apache.org/lucene-java/LuceneImplementations. 
  16. "mnoGoSearch". LavTech. http://www.mnogosearch.org/. 
  17. "MWSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:MWSearch. 
  18. "Nutch". Apache Software Foundation. http://lucene.apache.org/nutch/. 
  19. "RigorousSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:RigorousSearch. 
  20. "Sphinx". Sphinx Technologies. http://sphinxsearch.com/. 
  21. "Swish-e". Swish-e. http://swish-e.org/. 
  22. "Terrier". Terrier Project. http://ir.dcs.gla.ac.uk/terrier/. 
  23. "Xapian". Xapian Project. http://xapian.org/. 
  24. "YaCy". Karlsruhe Institute of Technology. http://yacy.net/. 
  25. "Zettair". Search Engine Group, Royal Melbourne Institute of Technology. http://www.seg.rmit.edu.au/zettair/.