Image Similarity Search Demo

The Austrian Institute of Technology (AIT), Austria's largest non-university research institute, specialises in the key infrastructure issues of the future. Within the Europeana Creative project, AIT contributes to the development of the tourism and design pilot applications. AIT also coordinates the implementation of back-end services and APIs, develops image similarity and geographic mapping services and manages user-generated content services.

Developing an Image Similarity Search

AIT recently presented a demo of the image similarity search service at SUEDL2013, the 2nd International Workshop on Supporting Users Exploration of Digital Libraries. The current implementation supports users who are exploring the content of Europeana and looking for images that share similarities.

Image Similarity Search Process

The search service demonstrated within this paper was developed in the context of the ASSETS project by ISTI-CNR to improve the user experience when exploring Europeana datasets. It is re-used and enhanced within the Europeana Creative project, with the goal of supporting discovery of content suitable for inspiring the creativity of professional designers.

In the context of the Europeana Creative project, ongoing research activities are related to the re-use of cultural heritage content for inspiring creative design. Several categories of cultural heritage objects have been identified for use in this context. The current version of the search algorithm provides a good overall retrieval accuracy on the given dataset, but there are certain types of images for which search performance doesn't meet the expectations of professional users.

Screenshot of the demo

Currently AIT is working on increasing the size of the dataset that will be re-used for the implementation of the design pilot application and optimising the effectiveness of the retrieval algorithms for particular classes of objects. Additionally they aim to integrate the service within the Europeana APIs, making it available for any stakeholders building domain-specific applications.

Working with Cultural and Creative Industries

AIT's staff are from very technical backgrounds; one challenge they experienced when working with cultural heritage institutions and creative industry hubs is that AIT focus more on complexity (back-end) while domain experts concentrate on usability (front-end). Domain experts are often not aware of the power of particular technologies, or may not understand what they are good for (something called 'knowledge acquisition bottleneck'). Aligning the terminologies and the goals of the two communities is the key for successful collaboration.

Working with domain experts is both a challenge and an opportunity. For AIT, it is a challenge to be able to fulfil their expectations with the developed applications, and an opportunity to learn new concepts and new ways to make use of technology and to face new ways of perceiving developments such as image similarity. Consequently, this will lead to identification of new research questions and possible implementations.

How It Works

The implementation is based on open source technologies. The MPEG-7 descriptors are computed by using the LIRE feature extractors and the approximate similarity search algorithm uses the Melampo library. The core of the retrieval process is represented by the computation of the similarities between the query and result images. This is performed by employing the cosine similarity computed on colour and shape features extracted from the original images, namely the MPEG-7 Sclable Color, Color Layout and Edge Histogram descriptors.

The current search algorithm has a retrieval accuracy able to satisfy the expectations of public users navigating through the Europeana repository, and it may satisfy the expectations of professional designers searching for certain types of content like paintings, drawings, glass or porcelain objects. For other categories of objects, shape-based descriptors and algorithms may perform better.

Future work will focus on identifying algorithms that are able to maximise the retrieval performance on each of the selected classes of images. We plan to evaluate additional image descriptors (e.g. local and shape-based descriptors) and search algorithms to better satisfy the expectations of design communities.