From Fedora Project Wiki

< Infrastructure

Revision as of 19:56, 5 November 2009 by Akistler (talk | contribs) (→‎In Progress: Added info on rpms already in Fedora; Added CLucene)


Points of Contact

Project Sponsor

Name: Mike McGrath
Fedora Account Name: mmcgrath
Group: Infrastructure
Infrastructure Sponsor: mmcgrath

Secondary Contact info

Name: Huzaifa Sidhpurwala
Fedora Account Name: huzaifas
Group: Infrastructure

Name: Allen Kistler
Fedora Account Name: akistler
Group: Infrastructure

Project Info

Project Name: Search Engine Enhancement
Target Audience: All users of Fedora web sites
Expiration/Delivery Date (required): F13

Description/Summary

Fedora needs a search engine[1]

Requirements

  • Crawl the web sites (wiki and non-wiki)
  • Search the web sites (wiki and non-wiki)

Preferences

  • Python-based
Note: Other languages are permitted, but Java must be the GCJ/OpenJDK versions in RHEL5. Sun/IBM/BEA Java is not acceptable.
  • Programmable keywords to have control over what pages get displayed for certain keywords
  • XML or library interface so other applications can use it

Project Plan

  1. Investigate and evaluate existing open source search engines
  2. Select candidate software
  3. Create public test instance of candidate software
  4. Test for functionality, performance, and impact (re-evaluate, if necessary)
  5. Create capacity and deployment plans
  6. Deploy

Resources Needed

  • Public Test for testing candidate software
  • Permanent home(s) for deployment
    • Web server(s)
    • Database server(s)

Software Investigation and Evaluation

In Progress

C++ port of Lucene
in Fedora already
described as beta by the developers
  • DataparkSearch [3]
written in C
written in Java
Ruby port of Lucene
KinoSearch and Ferret intend to merge as Lucy [6]
written in C/C++
written in C++
  • KinoSearch [9] - akistler examined
  • Description
Perl/C port of Lucene
in Fedora already
maintainer Rectangular Research appears to be just one guy, who considers KinoSearch to be alpha software
KinoSearch and Ferret intend to merge as Lucy [6]
  • Evaluation
Search engine library with sample indexer and search page rather than fully-functional application. Stores indices in Berkeley DB files with JSON interfaces. Allows custom-designed indices, including categories (exact match) to fulfill "programmable keywords" requirement. Each document index on each document source is a single write-once file collection (BDB and JSON) in a unique directory. Rerunning the indexer creates a new directory, obsoleting the old directory if all the old documents are included. The old directory then needs to be cleaned up. Postings can, however, be deleted from an index. Additionally, only the new documents can be indexed, but that's not efficient.
  • Requirements
buildrequires
  • gcc
  • (EPEL) perl-Module-Build
requires
  • (EPEL) perl-JSON-XS
Problem: Desires 1.53, but EPEL has 1.43
Note: http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/
Note: works with 1.43, anyway
  • (EPEL) perl-Lingua-Stem-Snowball
  • (EPEL) perl-Lingua-StopWords
  • (EPEL) perl-Parse-RecDescent
sample indexer reads files from the file system and requires
  • (EPEL) perl-HTML-Tree
sample cgi search script requires
  • (CPAN) Data::Pageset (which requires Data::Page)
  • (EPEL) perl-Test-Exception
  • (EPEL) perl-Class-Accessor-Chained
written in Java, but ported to others [11]
in Fedora already
PyLucene [12] is a Python wrapper around Java Lucene
written in C
  • Namazu [14] - Huzaifa investigating
written in Perl
in Fedora already
  • written in Java
  • based on Lucene
  • Swish-e [16] - Allen investigating
  • Description
written in C
Swish++ is a rewrite in C++
  • Evaluation
Search engine with a built-in web crawler, a built-in file system crawler, and an interface for an external crawler. The distribution includes sample search pages which use the Perl API. There is also a C API. The index is not customizable, but does include a facility for including metawords and the path in the index for each document. The documentation acknowledges that the software only supports ASCII, but some MBCS may also work.
  • Requirements
<in progress>
  • Terrier (TERabyte RetrIEveR) [17]
written in Java
written in C++
in Fedora already
bindings to Python, Perl, PHP, Ruby, Java, and more
Omega (builtin) provides a more complete search engine experience on top of core Xapian
built-in web crawler is a script that interfaces with ht://Dig
Flax [19] is another search engine built on top of Xapian and CherryPy
written in C

Not Suitable

  • written in Ruby
  • specializes in source code search
  • not actively maintained
written in C#
  • written in Java
  • archives content rather than indexing it
  • written in C++
  • not actively maintained
It's just a MediaWiki plugin, not suitable for searching non-wiki sites
  • Requires EzMwLucene (Java, not desirable) to be running on the servers to be searched
  • EzMwLucene is wiki-only, therefore MWSearch is wiki-only
  • written in Perl or TCL on top of PostgreSQL
  • Python interface available
  • not actively maintained
  • Perl port of Lucene
  • not actively maintained
Crawls the MediaWiki database, not the web site
Doesn't work for non-MediaWiki web sites, including any non-wiki web site
written in C++
designed to index SQL tables, not web pages
Written in C++
MediaWiki plug-in, so it's wiki-only
  • YaCy [32] - Examined by Huzaifa
  • written in Java, but requires Sun Java
  • Well maintained
  • Support for peer search engine database exchanges
  • Customized search parameters
  • Fast indexing and web interface for querying the back end db

Comparison by Requirements

Engine Name Source Language Integrated Web Crawler Integrated Web Front-End Programmable Categories Application Interface
DataparkSearch C
Egothor Java
Ferret Ruby
Indri C/C++
Isearch C++
KinoSearch Perl/C No
(sample file crawler included) 		No
(sample included) 		Yes Yes
(BDB/JSON)
Lucene Java
mnoGoSearch C
Namazu Perl
Nutch Java
Swish-e C Yes No
(sample included) 		Yes
(part of index config) 		Yes
(Perl and C APIs)
Terrier Java
Xapian C++
Zettair C
Engine Name Source Language Integrated Web Crawler Integrated Web Front-End Programmable Categories Application Interface

Public Testing

<tbd>

Deployment Plan

<tbd>

References

  1. ↑ "Fedora Search Engine". Fedora Trac. https://fedorahosted.org/fedora-infrastructure/ticket/1055. 
  2. ↑ "CLucene". CLucene Project. http://sourceforge.net/projects/clucene/. 
  3. ↑ "DataparkSearch". DataparkSearch. http://www.dataparksearch.org/. 
  4. ↑ "Egothor". Egothor. http://www.egothor.org/. 
  5. ↑ "Ferret". David Balmain. http://ferret.davebalmain.com/. 
  6. ↑ 6.0 6.1 "Lucy". Apache Software Foundation. http://lucene.apache.org/lucy/. 
  7. ↑ "Indri". The Lemur Project. http://www.lemurproject.org/indri/. 
  8. ↑ "Isearch". Isite. http://isite.awcubed.com/. 
  9. ↑ "KinoSearch". Rectangular Research. http://www.rectangular.com/kinosearch/. 
  10. ↑ "Lucene". Apache Software Foundation. http://lucene.apache.org/. 
  11. ↑ "Lucene Implementations". Apache Software Foundation. http://wiki.apache.org/lucene-java/LuceneImplementations. 
  12. ↑ "PyLucene". Apache Software Foundation. http://lucene.apache.org/pylucene/. 
  13. ↑ "mnoGoSearch". LavTech. http://www.mnogosearch.org/. 
  14. ↑ "Namazu". Namazu Project. http://www.namazu.org/. 
  15. ↑ "Nutch". Apache Software Foundation. http://lucene.apache.org/nutch/. 
  16. ↑ "Swish-e". Swish-e. http://swish-e.org/. 
  17. ↑ "Terrier". Terrier Project. http://ir.dcs.gla.ac.uk/terrier/. 
  18. ↑ "Xapian". Xapian Project. http://xapian.org/. 
  19. ↑ "Flax". Flax. http://www.flax.co.uk/products.shtml. 
  20. ↑ "Zettair". Search Engine Group, Royal Melbourne Institute of Technology. http://www.seg.rmit.edu.au/zettair/. 
  21. ↑ "Gonzui". SourceForge. http://gonzui.sourceforge.net/. 
  22. ↑ "Grub". Wikia, Inc.. http://grub.org/. 
  23. ↑ "Heritrix". Internet Archive. http://crawler.archive.org/. 
  24. ↑ "ht://Dig". The ht://Dig Group. http://www.htdig.org/. 
  25. ↑ "HtdigSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:HtdigSearch. 
  26. ↑ "MWSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:MWSearch. 
  27. ↑ "OpenFTS". XWare. http://www.astronet.ru/xware/#fts. 
  28. ↑ "Plucene". CPAN. http://search.cpan.org/~tmtm/Plucene-1.25. 
  29. ↑ "RigorousSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:RigorousSearch. 
  30. ↑ "Sphinx". Sphinx Technologies. http://sphinxsearch.com/. 
  31. ↑ "SphinxSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:SphinxSearch. 
  32. ↑ "YaCy". Karlsruhe Institute of Technology. http://yacy.net/. 
Retrieved from "https://fedoraproject.org/w/index.php?title=Infrastructure/Search&oldid=134337"