Aggregate views are commonly used for summarizing information held in very large databases such as those encountered in data warehousing, large scale transaction management, and statistical databases. Such applications often involve distributed databases that have developed independently and therefore may exhibit incompatibility, heterogeneity, and data inconsistency. We are here concerned with the integration of aggregates that have heterogeneous classification schemes where local ontologies, in the form of such classification schemes, may be mapped onto a common ontology. In previous work, we have developed a method for the integration of such aggregates; the method previously developed is efficient, but cannot handle innate data inconsistencies that are likely to arise when a large number of databases are being integrated. In this paper, we develop an approach that can handle data inconsistencies and is thus inherently much more scalable. In our new approach, we first construct a dynamic shared ontology by analyzing the correspondence graph that relates the heterogeneous classification schemes; the aggregates are then derived by minimization of the Kullback-Leibler information divergence using the EM (Expectation-Maximization) algorithm. Thus, we may assess whether global queries on such aggregates are answerable, partially answerable, or unanswerable in advance of computing the aggregates themselves.
|Journal||IEEE Transactions on Knowledge and Data Engineering|
|Publication status||Published (in print/issue) - 1 Jan 2003|
Bibliographical noteOther Details
This paper describes a scalable and efficient methodology for integrating aggregates from heterogeneous databases distributed over the Internet. The focus is on computing a dynamic shared ontology that is used as a framework for integration using mobile agents. The approach was developed and implemented as part of the EU-IST MISSION project and is ongoing work within the Information and Software Engineering research group in the area of distributed database management and knowledge discovery. The distributed data management concepts developed in this paper are currently being used in the SAP funded PERSERVE project, which is concerned with Service Oriented Architectures.