Schema Matching with Valentine (pyJedAI intregrated)¶

In this notebook we present the pyJedAI schema matching functionality. In general Schema Matching looks for semantic correspondences between structures or models, such as database schemas, XML message formats, and ontologies, identifying different attribute names that describe the same feature (e.g., “profession” and “job” are semantically equivalent)

TU Delft tool for Schema Matching¶

Website: https://delftdata.github.io/valentine

Valentine is an extensible open-source product to execute and organize large-scale automated matching processes on tabular data either for experimentation or deployment in real world data. Valentine includes implementations of seminal schema matching methods that we either implemented from scratch (due to absence of open source code) or imported from open repositories. To enable proper evaluation, Valentine offers a fabricator for creating evaluation dataset pairs that respect specific semantics.

pyJedAI functionalities with Valentine¶

Similarity Flooding constructs a graph representation of the schemas or ontologies to be matched, casting schema matching as a graph matching task. The nodes of the graph represent schema elements, such as attributes, or types, while the edges capture the relationships between these elements. The algorithm starts by initializing similarity scores based on initial mappings or heuristics. It then iteratively propagates similarities through the graph by 'flooding' the network with these scores: each node's similarity score is updated based on the scores of its neighbors, thus capturing the notion that similar nodes are often connected to other similar nodes. This process continues until the similarity scores converge or a specified number of iterations is reached.
Cupid is a comprehensive approach that involves the four steps: [Coma matcher it is required to have java (jre) installed]
- preprocessing, which normalizes schema elements by expanding abbreviations and removing special characters to ensure that linguistic comparisons are effective,
- linguistic matching, which uses a thesaurus or lexical database, like WordNet, to find semantic relationships (synonyms, hypernyms, etc.) between names and descriptions of schema elements, assigning linguistic similarity scores.
- structural matching, which infers the context and relatedness of elements from the hierarchical structure of the schema (e.g., parent-child relationships),
- similarity aggregation, which combines linguistic and structural similarities using a weighted average that can be adjusted to prioritize one aspect over the other depending on the context.
COMA constitutes a flexible approach for combining several complementary techniques, such as linguistic matching based on names and descriptions, structural matching based on schema hierarchies, and constraint-based matching considering keys and cardinalities. These matchers can be executed in parallel or sequentially, and their results can be aggregated using different strategies, such as weighted averages or maximum selection. The end result is a consolidated similarity matrix that reflects the combined evidence from all applied matchers.
EmbDI uses a compact graph-based representation that captures the relationships between the elements of the given schema, facilitating the extraction of sentences that effectively describe the similarity between elements. The resulting sentences are used to learn embeddings that are crafted for the data at hand, using optimizations to improve their quality.
SemProp is composite approach that combines syntactic with semantic similarity. First, it applies the Semantic Matcher, which leverages the semantic similarity between schema elements, as it is determined by pre-trained word embeddings. This similarity is used to yield coherent groups, i.e., clusters of words for which the average cosine similarity of their embedding vectors is higher than a specific threshold. Pairs of attributes that cannot be matched in this way, are associated with syntactic similarities based on the textual similarity of their instances (i.e., attribute values) and attribute names.

In [ ]:

!pip install pyjedai -U

In [1]:

!python --version

Python 3.9.16

Libraries injection¶

In [3]:

import os
import sys
import pandas as pd

from pyjedai.utils import (
    text_cleaning_method,
    print_clusters,
    print_blocks,
    print_candidate_pairs
)
from pyjedai.evaluation import Evaluation
from pyjedai.datamodel import Data

Data reading and pyJedAI-formating¶

In [ ]:

d1 = pd.read_csv(r"C:\Users\nikol\Desktop\GitHub\pyJedAI-Dev\data\ccer\schema_matching\authors\authors1.csv")
d2 = pd.read_csv(r"C:\Users\nikol\Desktop\GitHub\pyJedAI-Dev\data\ccer\schema_matching\authors\authors2.csv")
gt = pd.read_csv(r"C:\Users\nikol\Desktop\GitHub\pyJedAI-Dev\data\ccer\schema_matching\authors\pairs.csv")


data = Data(
    dataset_1=d1,
    attributes_1=['EID','Authors','Cited by','Title','Year','Source tittle','DOI'],
    id_column_name_1='EID',
    dataset_2=d2,
    attributes_2=['EID','Authors','Cited by','Country','Document Type','City','Access Type','aggregationType'],
    id_column_name_2='EID',
    ground_truth=gt,
)

Call pyJedAI methods using Valentine¶

In [ ]:

from pyjedai.schema.matching import ValentineMethodBuilder, ValentineSchemaMatching

sm = ValentineSchemaMatching(ValentineMethodBuilder.cupid_matcher())
sm.process(data)

Get matches¶

In [ ]:

matches = sm.get_matches()

Print matches¶

In [ ]:

sm.print_matches()

Evaluation¶

In [ ]:

sm.evaluate()