🔎 Understanding Financial Entities In Context¶
📜 Assertion Status, or Understanding Financial Entities in Context, is an NLP atsk in carge of analyzing NER entities, extracted with:
- NER models;
- ContextualParser;
📜 and their surroundings (usually a sentence, but it could take bigger spans too) to assert different conditions / status on the entities, as:
- If an entity is negated in the context;
- If the context talks about past, present, or future;
- If the entity is said to be hypothetical / possible, or certaing;
- etc.
📜 The exposed above are just some examples, since the applications of Assertion DL models can be expanded to whatever to many other scenarios where you need to:
- Disambiguate entities from the context.
- Subclassify or specify an entity depending on context:
🚀Examples:
- Is an ORG mentioned to be a COMPETITOR or part of the SUPPLY_CHAIN (or none of them)?
- Is an ACQUIRED_COMPANY mentioned to be acquired TOTALLY or PARTIALLY acquired in the context?
- etc
Let's see which pretrained models we have and how to train custom ones!
🎬 Installation¶
In [ ]:
! pip install -q johnsnowlabs
🔗 Automatic Installation¶
Using my.johnsnowlabs.com SSO
In [ ]:
from johnsnowlabs import *
# nlp.install(force_browser=True)
🔗 Manual downloading¶
If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.
- Go to my.johnsnowlabs.com
- Download your license
- Upload it using the following command
In [ ]:
from google.colab import files
print('Please Upload your John Snow Labs License using the button below')
license_keys = files.upload()
- Install it
In [ ]:
nlp.install()
📌 Starting¶
In [ ]:
from johnsnowlabs import nlp, finance
# Automatically load license data and start a session with all jars user has access to
spark = nlp.start()
In [ ]:
from pyspark.sql import DataFrame
import pyspark.sql.functions as F
import pyspark.sql.types as T
import pyspark.sql as SQL
from pyspark import keyword_only
📚 Understanding time from context¶
✔️ Past Experiences of a Role (C-level management)¶
Let's start with a small example: analyzing whether a ROLE of a person in a company is mentioned to be a past or present.
📜For that, we need:
- An NER model. We will use
finner_bert_roleswhich usesbert_embeddings_sec_bert_baseembeddings to extractROLEentities; - An Assertion Model which detects time. We will use
finassertiondl_past_roleswhich is a very specific one, to detect time inROLEentities.
In [ ]:
document_assembler = nlp.DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
tokenizer = nlp.Tokenizer()\
.setInputCols(["document"])\
.setOutputCol("token")
embeddings = nlp.BertEmbeddings.pretrained("bert_embeddings_sec_bert_base","en") \
.setInputCols(["document", "token"]) \
.setOutputCol("embeddings")
tokenClassifier = finance.BertForTokenClassification.pretrained("finner_bert_roles","en","finance/models")\
.setInputCols("token", "document")\
.setOutputCol("ner")\
.setCaseSensitive(True)
ner_converter = finance.NerConverterInternal() \
.setInputCols(["document", "token", "ner"]) \
.setOutputCol("ner_chunk")\
.setWhiteList(["ROLE"])
assertion = finance.AssertionDLModel.pretrained("finassertiondl_past_roles", "en", "finance/models")\
.setInputCols(["document", "ner_chunk", "embeddings"]) \
.setOutputCol("assertion")
nlpPipeline = nlp.Pipeline(stages=[
document_assembler,
tokenizer,
embeddings,
tokenClassifier,
ner_converter,
assertion
])
empty_data = spark.createDataFrame([[""]]).toDF("text")
model = nlpPipeline.fit(empty_data)
light_model = nlp.LightPipeline(model)
✔️ Example sentences extracted from 10K filings¶
In [ ]:
sample_texts = ["""From January 2009 to November 2017, Mr. Tan worked as the Managing Director of Cadence""",
"""Jane S. Smith works as a Computer Engineer and Product Lead at Globalize Cloud Services""",
"""Mrs. Johansson has been apointed CEO and President of Mileways""",
"""Tom Martin worked as Cadence's CTO until 2010""",
"""Mrs. Charles was before Managing Director at a big consultancy company""",
"""We are happy to announce that Mary Leigh joins Elephant as Web Designer and UX/UI Developer"""]
✔️ We extract with LightPipelines¶
In [ ]:
import pandas as pd
chunks=[]
entities=[]
status=[]
for i in range(len(sample_texts)):
light_result = light_model.fullAnnotate(sample_texts[i])[0]
for n,m in zip(light_result['ner_chunk'],light_result['assertion']):
chunks.append(n.result)
entities.append(n.metadata['entity'])
status.append(m.result)
df = pd.DataFrame({'chunks':chunks, 'entities':entities, 'assertion':status})
In [ ]:
df
Out[ ]:
| chunks | entities | assertion | |
|---|---|---|---|
| 0 | Director | ROLE | PAST |
| 1 | Computer Engineer | ROLE | NO_PAST |
| 2 | Product Lead | ROLE | NO_PAST |
| 3 | CEO | ROLE | NO_PAST |
| 4 | President | ROLE | NO_PAST |
| 5 | Cadence's CTO | ROLE | PAST |
| 6 | Managing Director | ROLE | PAST |
| 7 | Web Designer | ROLE | NO_PAST |
| 8 | UX/UI Developer | ROLE | NO_PAST |
✔️ Visualization of Assertion Status¶
In [ ]:
for i in range(len(sample_texts)):
light_result = light_model.fullAnnotate(sample_texts[i])[0]
vis = nlp.viz.AssertionVisualizer()
vis.display(light_result, 'ner_chunk', 'assertion')
From January 2009 to November 2017, Mr. Tan worked as the Managing Director ROLEPAST of Cadence
Jane S. Smith works as a Computer Engineer ROLENO_PAST and Product Lead ROLENO_PAST at Globalize Cloud Services
Mrs. Johansson has been apointed CEO ROLENO_PAST and President ROLENO_PAST of Mileways
Tom Martin worked as Cadence's CTO ROLEPAST until 2010