The PAD-IR project will extend the notion of paper-digital retrieval systems beyond that of a paper-based interface to an information retrieval (IR) system in order to truly bridge the paper-digital divide by allowing retrieval across different forms of media, including handwritten notes and sketches. Thus, it will not simply be a case of formulating queries on paper, but also being able to digitally capture information on paper and link it to various forms of digital media based on the semantics of that information and also the context in which it was captured. Objects need to be managed together with their metadata including links between objects, the context of their acquisition and content features. Retrieval may then be based on queries specified digitally or on paper, or even some combination of both. Since queries might encompass several media types and the additional object meta data, dedicated algorithms to effectively search in these media types have to be available as basic building blocks. Query processing will consist of the automatic, individual composition of the necessary building blocks, in a way which is completely transparent to the user.

The application settings that PAD-IR will explore will include various kinds of meeting scenarios as well as post-meeting retrieval of information. During meetings, users often work with several paper and digital documents including handwritten notes taken by individual participants, sketches used as part of collaborative design processes and presentation tools. Although there are existing tools to help record meeting sessions, they tend to focus either  on digital recordings such as a combination of audio, video and presentations or solely on the recording of handwritten notes synchronised with audio recordings. Our goal is to allow participants in a meeting to work with a combination of paper documents and digital media, recording activities across all media in such a way that users can later retrieve information based on keyword search, similarity search (e.g. by using handwritten sketches), timeline or association.