83 e have recently developed a video database system, BilVideo, which provides integrated support for spatiotemporal, semantic, and low-level feature queries. 1 As a further development for this system, we present a natural language-based interface for query specification. This natural language processing (NLP)-based interface lets users formulate queries as sentences in English by using a partof-speech (POS) tagging algorithm. The system then groups the specified queries as object-appearance, spatial, and similarity-based object trajectory queries by using POS tagging information. It sends the queries constructed in the form of Prolog facts to the query processing engine, which interacts with both the knowledge base and object-relational database to respond to user queries that contain a combination of spatiotemporal, semantic, color, shape, and texture video queries. The query processor seamlessly integrates the intermediate query results returned from these two system components. The system sends the final results to Web clients. What motivates our work is the need for a convenient and flexible natural language-based interface to complement the text-based query interface and the visual query interface of the BilVideo system, because specification of spatial queries using text or visual interfaces is not very easy for novice users. (For examples of how others have attempted to handle these issues, see the "Related Work" sidebar, next page.) Thus, we developed a natural language-based query interface that's convenient and offers greater flexibility when specifying queries. The POS-based pattern-matching approach we use in identifying queries helps users specify queries without conforming to strict rules. This approach also lets us adjust our query interface easily as we add new query types to the BilVideo system. BilVideo's spatiotemporal queries contain any combination of spatial, temporal, object-appearance, external-predicate, trajectory-projection, and similarity-based object trajectory conditions. A rule-based system built on a knowledge base processes these types of queries. A feature database maintained by an object-relational database also responds to semantic (keyword, event/activity, and category-based) and low-level feature queries (color, shape, and texture). BilVideo has the following parts: 1 ❚ an object extractor, which extracts salient objects from video keyframes; ❚ a fact extractor, which extracts spatiotemporal relations between video objects and store them in the knowledge base as facts; ❚ a video annotator, which extracts semantic data from video clips and stores them in the feature database to query video data for its semantic content; ❚ a Web-based visual query interface, which specifies queries by using visual sketches and displays the results; and ❚ a textual query language, which specifies queries using an extended Structured Query Language (SQL). Currently BilVideo uses a Web-based visual query interface for query specification (available at http://www.cs.bilkent.edu.tr/~bilmdg/bilvideo). The interface uses specification windows for spatial and trajectory queries. The specification and formation of these queries vary significantly; therefore, the interface creates specific windows to handle each type. Users can query the temporal content of videos by combining these two types of primitive queries with temporal predicates (before, during, and so on). Users can also
Kucuktunc et al. REF proposed a pattern matching approach based on Part-of-Speech (POS) tags.
A Natural Language-Based Interface for Querying a Video Database
IEEE MultiMedia
IEEE MultiMedia
Computer Science