As more digital multimedia archives are developed, they require specialized search engines can index and search these formats. Video and audio are hard to browse, so search engines can save significant time and effort in locating useful content.

Indexing multimedia is much more complex than indexing text. In some cases the media can be converted to text: broadcast television often includes digital text as closed-captions for the hearing impaired, and scene titles and captions within a video can be converted to text using OCR. Speech-recognition technology can digitize words spoken on audio tracks. Continuous media, such as video, also can be broken up into chunks by transitional effects, for better precision in results. Some groups are also working on form and shape recognition, which could allow searchers to draw a shape, such as a bridge or a tumor; or select an example picture and find others like it.

Articles

• About F

• Finding the Right Stuff KMWorld Magazine, May 2003 by Judith Lamont of Zentek
  Covers a project at Sandia National Lab to digitize and search training materials with Convera Screening Room. This allows users to view the videos from their desktops and find content by keywords and key frame similarity.
  
  
• Visual Search Engines WebReference Newsletter, June 2001 by Tony DeYoung
  Describes the value of finding similar pictures, such as stock photo libraries, e-commerce sites and auctions. Points out that pattern recognition has some value but tends to work better for simple images and small databases. Reports on the eVision image search technology, which separates each image into a collection of objects rather than a single thing and provides good user controls for defining the search. Image analysis is slow but search results are very fast.
  
  
• Upstream: Video Searching Technology Review, July/August 2001 by David Voss.
  After some background on video searching, covers current academic research. Includes the work done at Columbia in image matching, such as medical ultrasound videos. Carnegie Mellon's Informedia Digital Video Library uses speech recognition, natural language processing and image analysis together.
  
  
• WebMARS: A Multimedia Search Engine for Full Document Retrieval and Cross Media Browsing Multimedia Information Systems Workshop, October 2000, by Michael Ortega-Binderberger, Sharad Mehrotra, Kaushik Chakrabarti, and Kriengkrai PorkaewI
  An academic report of a research project in multimedia retrieval.
  
  
• An Indexing, Browsing, Search and Retrieval System for Audiovisual Libraries ECDL '99, Paris by J. Hunter and J. Newmarsh
  This system uses a combination of existing metadata and simple tools to generate searchable content.
  
  
• Trends in Image and Video Search 1998 Infonortics Search Engines Meeting by Dragun Petkovic
  Good overview of the growing need for multimedia storage and re-use. Recommends integration with text and database search, especially for e-commerce sites, to make best use of color and shape similarity. But keywords and metadata are not enough for many applications, including brainstorming and multilingual sites.
  
  
• Indexing the Video Frontier Wired News, February 5, 1998 by Chris Oakes
  Describes the issues of indexing and searching video data, which is much harder than text. Algorithms must find meta patterns, such as scene changes, to break down the data into manageable parts -- and these are different in different genres of video, such as documentaries or sports. Products make good use of closed-captioned text, text within the video and analysis of audio tracks to create storyboards and previews. More subtle indexing will require image analysis and pattern recognition. Products also work with Microsoft's NetShow streaming audio and video products. Includes information about Excalibur, Virage and Magnifi.
  
  
• Finding people and animals using body plans and action plans 1997 David Forsyth and Margaret Fleck, various publications
  Uses computer vision algorithms to recognize objects in multiple situations.
  
  
• Storage and Retrieval for Still Image and Video Databases V 1996; Proceedings of the SPIE Volume 2670, edited by Iswar K. Sethi, Ramesh C. Jain.
  Good coverage of the academic and commercial aspects of multimedia searching.
  
  
• Finding the Cut of the Wrong Trousers: Fast Video Search Using Automatic Storyboard Generation CHI 96, by Peter J. Macer, Peter J. Thomas, Nouhman Chalabi, John F. Meech.
  Academic paper describing the process of reducing a video to a storyboard (set of scenes) by detecting transitions and selecting a representative frame. These are available for browsing or visual query matching.

Resources

• Bibliography of Moving Image Indexing - Winter 2000, annotated description of major academic projects
  
  
• Content-Based Image Retrieval - listings of academic projects
  
  
• Informedia Project, Carnegie-Mellon University - digital video library research in automatic video and audio indexing, search and retrieval, and navigation.
  
  
• Multimedia Information Processing Group - group at Institut Eurécom working on Multimedia Indexing, Filtering and Retrieval.
  
  
• Selected Projects in Digital Video/Multimedia Research, Columbia University (USA), Professor Shih-Fu Chang
  
  
• Visual Information Category - Open Directory Project

Products

• Autonomy
• Convera / Excalibur Retrievalware
• DrawingSearcher - AutoCAD search engine based on dtSearch
• eVision - allows search by similarity to another image, color, shape, texture, object
• FAST Search
• IBM CueVideo
• IBM QBIC (Query by Image Content), includes examples
• LTUtech image analysis
• MATE recognition engines
• Oracle interMedia
• Virage (bought by Autonomy, summer 2003)

Page Updated 2003-07-14