Problem
Unit six looks at how to evaluate the effectiveness of an information retrieval system. Precision, recall, accuracy, and the F measure are all discussed as metrics that can be used to measure the effectiveness of results retrieved from an IR system. In chapter 8 of the text, we learn that there are a number of document collections (corpus) that are used for this purpose. These document collections are used along with well known queries (keep in mind that a query is the terms that are used to search the collection). An IR system to be evaluated first indexes the corpus and then the queries are used to test the results that the IR system returns. What is important about this process is that for these queries there are known metrics such as the number of documents in the collection that SHOULD be relevant.
These measures of effectiveness are calculated based upon such known information and the results returned from a query submitted to an IR system.
For example, consider our CS3308 corpus. We know that it contains 2,476 documents which are all news briefs reported by the Reuters news service. Suppose that we have known metrics such as the fact that there are a total of 20 documents in this collection that are relevant to a query for the terms 'home' and 'mortgage'.
Assume that the IR system that we have developed returns 8 relevant documents and 10 documents that are not relevant. Using this information and the formulas for Precision, Recall, F-Measure, and Accuracy, calculate what each of these measures would be for the example presented above. When you have determined the metric for each post a response that includes:
i. The Precision, Recall, F-Measure, and Accuracy effectiveness metrics which you will calculate using the metrics provided above.
ii. Discuss which approach provides the most valid measure of the effectiveness of the IR system and why.
Keep in mind that Precision and Recall are used together a measure of effectiveness, the F-Measure provides a single measure that balances Precision and Recall metrics and Accuracy provides a measure of the accuracy of classifications in the collection.