Most commercial search engines include a more or less advanced document processing pipeline for transforming raw input into something that can be indexed. The process involves normalization, entity extraction, linguistic processing, annotation, data cleansing etc. When it comes to Open Source search engines, they start getting pretty good at the core of indexing and search,
OpenPipeline – an open-source document processing pipeline
Most commercial search engines include a more or less advanced document processing pipeline for transforming raw input into something that can be indexed. The process involves normalization, entity extraction, linguistic processing, annotation, data cleansing etc. When it comes to Open Source search engines, they start getting pretty good at the core of indexing and search,
Read More