JISC

This report documents a six month investigation into content-based image retrieval (CBIR) software.  The study forms part of a joint venture between Manchester Visualization Centre and the Institute for Image Data Research, which aims to investigate the feasibility of content-based image retrieval for the UK Higher Education Community. 

Executive Summary

The project is funded through a JISC Technology Applications Programme (JTAP) award.  The primary aim of this phase of the project was to undertake a review of currently available CBIR software in order to make informed recommendations for Phase III of the project.  The report compliments the review of CBIR, Phase I, conducted by the Institute for Image Data Research.  The study involved the identification, acquisition, installation, analysis and testing of a number of available CBIR applications.  The original intention was to systematically analyse each system within a laboratory setting to establish its functionality, effectiveness and usability.  Due to a number of factors this was not possible for all the systems identified.  This was primarily a direct result of the application type and associated development time required to implement each system.  The project was limited to content-based image retrieval systems that retrieved static image data.

Over the course of the investigation, 74 systems were identified, which included systems both past and present.  These were a combination of prototype research systems, database management systems (DBMS), software development kits (SDK), ‘turnkey’ systems, and World Wide Web (WWW) image search engines.  While the identification of CBIR systems was very encouraging, attempts at acquisition proved disappointing.  Of the identified systems, 9 in total were acquired:

• ARTISAN
• Excalibur Visual RetrievalWare SDK
• ImageFinder
• IMatch
• Informix Internet Foundation 2000 Server & the Excalibur Image Datablade
• Oracle 8i Enterprise Server and the Virage VIR Image Data Cartridge
• Photobook
• QBIC Development Kit
• Virage VIR Image Engine SDK and Image Read/Write Toolkit

The majority of the identified systems were research prototypes.  Prototype research systems exist primarily to test the feature matching algorithms being developed by the research community.  These systems are generally not available for public use or were not shipped in a suitable form to have been included in this review without significant development work.  As a result, the functionality of the research prototypes is outlined.  No attempt was made to assess their retrieval effectiveness or usability.  Several of the prototype research systems are available over the web and URLs are listed for the web-based demonstrator. 
 
The report provides a summary of the functionality for the following applications:

• Excalibur Visual RetrievalWare SDK, Excalibur Corp.
• ImageFinder, Attrasoft
• IMatch, MWLabs
• QBIC Development Kit, IBM Corp.
• Virage VIR Image Engine SDK and Image Read/Write Toolkit, Virage Inc.

General system information and matching features are described, and are collated in a matrix, Appendix A and Appendix B, for all identified systems.  The amount of publicly accessible information varied significantly between systems and this is reflected in the information collated.  The results of a small-scale feature matching and retrieval experiment for three applications are also documented: ImageFinder, IMatch and QBIC.  The purpose of the test was to provide a general indication as to the initial effectiveness of the systems matching features and retrieval capabilities.  The tests were not rigorous or scientifically controlled retrieval experiments and must not be regarded as an indicator as to the systems overall effectiveness.  The experiments were specifically designed to test whether the matching features in the application achieved their intended purpose by identifying similar images to the query image based on the matching feature employed.  Due to the widely recognised difficulties of assessing the effectiveness of retrieval results readers are left to draw their own conclusions.  A sample of the retrieval output is documented in Appendix C.  The image dataset used in the retrieval experiment was comprised of several distributed image datasets, which are freely available to the academic community, and image datasets bundled with ImageFinder and QBIC.  The datasets were merged and converted to produce a set of 24-bit colour and greyscale JPEG files with no additional compression.  The retrieval test verified that generally the matching features behaved predictably in terms of their functionality.  Comparison of the retrieval results suggests that there are varying degrees of retrieval precision and recall between the CBIR applications.  Detailed knowledge of the data set used in the retrieval experiment indicates that the degree of retrieval precision and recall, and the reliability of the system are in general highly questionable.

Two systems were subject to a heuristic evaluation, ImageFinder and IMatch.  The evaluation involved the examination of the user interface to judge its compliance with recognized and widely accepted usability principles.  The results suggest that there is considerable scope for improvement for both systems.  Content-based image retrieval potentially provides new opportunities to extend and enhance the constraints and limitations imposed by the traditional information retrieval paradigm on image collections.  The number of CBIR systems is extremely encouraging.  Nevertheless, there are still a significant number of open research issues to be addressed if this technique is to prove fruitful.  The current impasse with regards to the efficacy of the retrieval techniques being developed and the need to develop suitable evaluation frameworks and benchmarks is now critical.

Read the final report below