In this notebook, we continue from where left off in 7.Entity_Resolution
notebook.
! pip install -q johnsnowlabs
Using my.johnsnowlabs.com SSO
from johnsnowlabs import nlp, finance
# nlp.install(force_browser=True)
If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.
from google.colab import files
print('Please Upload your John Snow Labs License using the button below')
license_keys = files.upload()
nlp.install()
spark = nlp.start()
📜Entity resolution is an important task in natural language processing and information extraction, as it allows for more accurate analysis and understanding of financial texts. For example, in a news article discussing the performance of a company's stock, accurately identifying and disambiguating the company's name is crucial for accurately tracking the stock's performance.
📜An NLP use case in financial or legal applications is identifying financial entities' presence in a given text. One of those entities could be Company Name
. We can carry out NER to extract different chunks of information, but in real financial and legal use cases, the company name is usually not useful as it is mentioned in the text. Sometimes we need the official name of the company (instead of Amazon
, Amazon.com INC
, as registered in Edgar, Crunchbase and Nasdaq). We have pre-trained sentence entity resolver models for these purposes shown below with the examples.
Here are the list of pretrained Entity Resolution models:
📚Other than providing the code in the "result" field it provides more metadata about the matching process:
We will use following Generic Function For Getting the Codes and Relation Pairs
import pandas as pd
pd.set_option('display.max_colwidth', 0)
def get_codes (lp, text, vocab='company_name', hcc=False):
"""Returns LightPipeline resolution results"""
full_light_result = lp.fullAnnotate(text)
chunks = []
codes = []
begin = []
end = []
resolutions=[]
all_distances =[]
all_codes=[]
all_cosines = []
all_k_aux_labels=[]
for i in range(len(full_light_result)):
for chunk, code in zip(full_light_result[i]['ner_chunk'], full_light_result[i][vocab]):
begin.append(chunk.begin)
end.append(chunk.end)
chunks.append(chunk.result)
codes.append(code.result)
all_codes.append(code.metadata['all_k_results'].split(':::'))
resolutions.append(code.metadata['all_k_resolutions'].split(':::'))
all_distances.append(code.metadata['all_k_distances'].split(':::'))
all_cosines.append(code.metadata['all_k_cosine_distances'].split(':::'))
if hcc:
try:
all_k_aux_labels.append(code.metadata['all_k_aux_labels'].split(':::'))
except:
all_k_aux_labels.append([])
else:
all_k_aux_labels.append([])
df = pd.DataFrame({'chunks':chunks, 'begin': begin, 'end':end, 'code':codes, 'all_codes':all_codes,
'resolutions':resolutions, 'all_k_aux_labels':all_k_aux_labels,'all_distances':all_cosines})
return df
The Nasdaq Stock Market
(National Association of Securities Dealers Automated Quotations Stock Market) is an American stock exchange based in New York City. It is ranked second on the list of stock exchanges by market capitalization of shares traded, behind the New York Stock Exchange
Let's suppose we get this text from scrapping the Internet, or from Twitter. Firstly, we get company name from sample text with finner_orgs_prods_alias
model.
text = "Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing."
document_assembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
text_splitter = finance.TextSplitter()\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = nlp.Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")
embeddings = nlp.BertEmbeddings.pretrained("bert_embeddings_sec_bert_base","en") \
.setInputCols(["sentence", "token"])\
.setOutputCol("embeddings")
ner_model = finance.NerModel.pretrained("finner_orgs_prods_alias", "en", "finance/models")\
.setInputCols(["sentence", "token", "embeddings"])\
.setOutputCol("ner")
ner_converter = nlp.NerConverter()\
.setInputCols(["sentence","token","ner"])\
.setOutputCol("ner_chunk")
nlpPipeline = nlp.Pipeline(stages=[
document_assembler,
text_splitter,
tokenizer,
embeddings,
ner_model,
ner_converter,
])
empty_data = spark.createDataFrame([[""]]).toDF("text")
model = nlpPipeline.fit(empty_data)
lp_ner = nlp.LightPipeline(model)
bert_embeddings_sec_bert_base download started this may take some time. Approximate size to download 390.4 MB [OK!] finner_orgs_prods_alias download started this may take some time. [OK!]
ner_result = lp_ner.annotate(text)
ner_result
{'document': ['Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing.'], 'ner_chunk': ['Aspect Development'], 'token': ['Aspect', 'Development', 'provides', 'component', 'and', 'supplier', 'management', '(', 'CSM', ')', 'technology', 'to', 'improve', 'the', 'product-development', 'cycle', 'of', 'manufacturing', '.'], 'ner': ['B-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], 'embeddings': ['Aspect', 'Development', 'provides', 'component', 'and', 'supplier', 'management', '(', 'CSM', ')', 'technology', 'to', 'improve', 'the', 'product-development', 'cycle', 'of', 'manufacturing', '.'], 'sentence': ['Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing.']}
ner_result["ner_chunk"]
['Aspect Development']
Aspect Development
company using finel_nasdaq_data_company_name
model.¶document_assembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("ner_chunk")
embeddings = nlp.UniversalSentenceEncoder.pretrained("tfhub_use_lg", "en") \
.setInputCols("ner_chunk") \
.setOutputCol("sentence_embeddings")
resolver = finance.SentenceEntityResolverModel.pretrained("finel_nasdaq_data_company_name", "en", "finance/models")\
.setInputCols(["sentence_embeddings"]) \
.setOutputCol("name")\
.setDistanceFunction("EUCLIDIAN")
pipeline = nlp.Pipeline(
stages = [
document_assembler,
embeddings,
resolver])
empty_data = spark.createDataFrame([[""]]).toDF("text")
model = pipeline.fit(empty_data)
lp = nlp.LightPipeline(model)
tfhub_use_lg download started this may take some time. Approximate size to download 753.3 MB [OK!] finel_nasdaq_data_company_name download started this may take some time. [OK!]
ner_result["ner_chunk"]
['Aspect Development']
%time
get_codes (lp, ner_result["ner_chunk"], vocab='name')
CPU times: user 4 µs, sys: 2 µs, total: 6 µs Wall time: 12.4 µs
chunks | begin | end | code | all_codes | resolutions | all_k_aux_labels | all_distances | |
---|---|---|---|---|---|---|---|---|
0 | Aspect Development | 0 | 17 | ASPECT DEVELOPMENT INC | [ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS & TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC] | [ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS & TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC] | [] | [0.0000, 0.1882, 0.1944, 0.1957, 0.2052, 0.2220, 0.2248, 0.2397, 0.2410, 0.2488, 0.2500, 0.2539, 0.2544, 0.2546, 0.2546, 0.2559, 0.2560, 0.2605, 0.2608, 0.2620] |
In NASDAQ, the company official is different!
Aspect Development
ASPECT DEVELOPMENT INC