In DefaultSimilarity, lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be 1 / (numTerms in field), all fields will be treated 'fairly'.

Changing Length Normalization - By overriding State), it is possible to discount how the length of a field contributes to a score.

In these cases people have overridden Similarity to return 1 from the Tf() method.

Overriding Tf - In some applications, it doesn't matter what the score of a document is as long as a matching term occurs.

where you think the frequency of terms is more significant.

The SweetSpotSimilarity gives small increases as the frequency increases a small amount and then greater increases when you hit the 'sweet spot', i.e.If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at Overriding Similarity. Instead of subclassing the Similarity, one can simply introduce a new basic model and tell DFRSimilarity to use it. DFRSimilarity delegates computation of the three parts of its formula to the classes BasicModel, AfterEffect and Normalization. These Similarities are implemented modularly, e.g. Subclasses are only required to implement the Float) and #toString() methods.Īnother option is to extend one of the frameworks based on SimilarityBase. The easiest way to quickly implement a new ranking method is to extend SimilarityBase, which provides basic implementations for the low level. To make this change, implement your own Similarity (likely you'll want to simply subclass an existing method, be it DefaultSimilarity or a descendant of SimilarityBase), and then register the new class by setting IndexWriterConfig.Similarity before indexing and IndexSearcher.Similarity before searching. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen. To change Similarity, one must do so for both indexing and searching, and the changes must happen before either of these actions take place. For instance, some applications do not need to distinguish between shorter and longer documents (see a 'fair' similarity). However, in some applications it may be necessary to customize your Similarity implementation. Changing SimilarityĬhances are the available Similarities are sufficient for all your searching needs. However, optimizations can always be implemented in subclasses see below. Since SimilarityBase is not optimized to the same extent as DefaultSimilarity and BM25Similarity, a difference in performance is to be expected when using the methods listed above. The implementation of two language models from Zhai and Lafferty's paper.Clinchant and Gaussier's Information-based models for IR.Lucene ships the following methods built on SimilarityBase: SimilarityBase provides a basic implementation of the Similarity contract and exposes a highly simplified interface, which makes it an ideal starting point for new ranking functions. For more information, see TFIDFSimilarity.īM25Similarity is an optimized implementation of the successful Okapi BM25 model. It is based on a highly optimized Vector Space Model. For searching, users can employ the modelsĪlready implemented or create their own by extending one of the classes in thisĭefaultSimilarity is the original Lucene scoring function. This package contains the various ranking models that can be used in Lucene.

Apache lucene relevance models