In this notebook, you will learn how to use Spark NLP and Legal NLP to identify relevant entities in legal texts using our state-of-the-art Named-Entity Recognition (NER) models and the recent Zero-Shot models.
We will cover the full analysis cycle, from reading a document in PDF formar, extracting its text contents, classifying its sections and applying NER models on specific sections.
Let`s dive in!
📚For the text classification tasks, we will use two annotators:
ClassifierDL
: uses the state-of-the-art Universal Senten- ClassifierDL
: uses the state-of-the-art Universal Sentence Encoder as an input for text classifications. Then, a deep learning model (DNNs) built with TensorFlow that supports Binary Classification
and Multiclass Classification
(up to 100 classes).MultiClassifierDL
: Multilabel Classification
(can predict more than one class for each text) using a Bidirectional GRU with Convolution architecture built with TensorFlow that supports up to 100 classes. The inputs are Sentence Embeddings such as state-of-the-art UniversalSentenceEncoder, BertSentenceEmbeddings or SentenceEmbeddings.In Legal NLP, since the number of classes can be very high (over 250) and the texts could belong to more than one topic at the same time (multilabel problem), we pretrained several binary classifiers (yes / no) for many clause types in legal documents that can be used independently.
You can select the topics you are interested in (for example, looking for loans and fiscal-year clauses) and create a pipeline with both of them to detect for those types of clauses in your paragraphs.
As a reminder, since the models are independent and the task is multilabel, you may get some times positive results for more than one class (i.e, a paragraph talks about loans and fiscal year at the same time).
As an alternative, we also have MultiClassifierDL
that predicts many clause types in one model. The choice between using binary classifiers or the multilabel model will depend on the document types, and experimentations should be made to verify the accuracy of the models in texts that are too different from the trianed data (CUAD dataset, SEC sample documents, etc.).
📚Example Classification models:
title | language | predicted_entities | compatible_editions |
---|---|---|---|
Human Rights Articles Classification | en | ['Artículo 1. Obligación de Respetar los Derechos', 'Artículo 2. Deber de Adoptar Disposiciones de Derecho Interno', 'Artículo 3. Derecho al Reconocimiento de la Personalidad Jurídica', 'Artículo 4. Derecho a la Vida', 'Artículo 5. Derecho a la Integridad Personal', 'Artículo 6. Prohibición de la Esclavitud y Servidumbre', 'Artículo 7. Derecho a la Libertad Personal', 'Artículo 8. Garantías Judiciales', 'Artículo 9. Principio de Legalidad y de Retroactividad', 'Artículo 11. Protección de la Honra y de la Dignidad', 'Artículo 12. Libertad de Conciencia y de Religión', 'Artículo 13. Libertad de Pensamiento y de Expresión', 'Artículo 14. Derecho de Rectificación o Respuesta', 'Artículo 15. Derecho de Reunión', 'Artículo 16. Libertad de Asociación', 'Artículo 17. Protección a la Familia', 'Artículo 18. Derecho al Nombre', 'Artículo 19. Derechos del Niño', 'Artículo 20. Derecho a la Nacionalidad', 'Artículo 22. Derecho de Circulación y de Residencia', 'Artículo 23. Derechos Políticos', 'Artículo 24. Igualdad ante la Ley', 'Artículo 25. Protección Judicial', 'Artículo 26. Desarrollo Progresivo', 'Artículo 27. Suspensión de Garantías', 'Artículo 28. Cláusula Federal', 'Artículo 21. Derecho a la Propiedad Privada', 'Artículo 29. Normas de Interpretación', 'Artículo 30. Alcance de las Restricciones', 'Artículo 63.1 Reparaciones'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Absence of certain changes Clause Binary Classifier | en | ['other', 'absence-of-certain-changes'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Acceleration Clause Binary Classifier | en | ['other', 'acceleration'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Access Clause Binary Classifier | en | ['other', 'access'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Accounting terms Clause Binary Classifier | en | ['other', 'accounting-terms'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Adjustments Clause Binary Classifier | en | ['other', 'adjustments'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Agreements Clause Binary Classifier | en | ['other', 'agreements'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Amendments Clause Binary Classifier | en | ['other', 'amendments'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Application of proceeds Clause Binary Classifier | en | ['other', 'application-of-proceeds'] | ['Legal NLP 1.0', 'Legal NLP'] |
Conventions Classification | es | ['Convención sobre la Eliminación de todas las formas de Discriminación contra la Mujer', 'Convención sobre los Derechos de las Personas con Discapacidad', 'Convención Internacional Sobre la Eliminación de Todas las Formas de Discriminación Racial', 'Convención Internacional sobre la Protección de los Derechos de todos los Trabajadores Migratorios y de sus Familias', 'Convención de los Derechos del Niño', 'Pacto Internacional de Derechos Civiles y Políticos'] | ['Legal NLP 1.0', 'Legal NLP'] |
For a complete list, check NLP Models Hub
Named-Entity Recognition (NER) is the capability to automatically identify relevant entities in the text. For example, person names, company names, public companies trading code, quantities, etc. There are many ways to implement NER, but nowadays the most efficient one is to use models based on deep learning.
The deep neural network architecture for NER model in Spark NLP is BiLSTM-CNN-Char framework. a slightly modified version of the architecture proposed by Jason PC Chiu and Eric Nichols (Named Entity Recognition with Bidirectional LSTM-CNNs). It is a neural network architecture that automatically detects word and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering steps. This model is implemented in our NerDL
/NerModel
annotators that we will experiment with in this section.
At John Snow Labs, we are proud to have a library of state-of-the-art pretrained, out-of-the-box, NLP models. With our newer package Legal NLP it is no different, and we currently support more than 580 models fine tuned for the legal domain. Specifically to NER, we currently have more than 40 models that can identify entities for different business needs.
📚Example NER models:
title | language | predicted_entities | compatible_editions |
---|---|---|---|
NER on Legal Texts (CUAD, Silver corpus) | en | ['PERSON', 'LAW', 'PARTY', 'EFFDATE', 'LOC', 'DATE', 'DOC', 'ORDINAL', 'ROLE', 'PERCENT', 'ORG'] | ['Legal NLP 1.0', 'Legal NLP'] |
Generic Deidentification NER | en | ['AGE', 'CITY', 'COUNTRY', 'DATE', 'EMAIL', 'FAX', 'LOCATION-OTHER', 'ORG', 'PERSON', 'PHONE', 'PROFESSION', 'STATE', 'STREET', 'URL', 'ZIP'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER - License / Permission Clauses (Bert, sm) | en | ['PERMISSION', 'PERMISSION_SUBJECT', 'PERMISSION_OBJECT', 'PERMISSION_INDIRECT_OBJECT'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER (Headers / Subheaders) | en | ['HEADER', 'SUBHEADER'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER - Whereas Clauses (sm) | en | ['WHEREAS_SUBJECT', 'WHEREAS_OBJECT', 'WHEREAS_ACTION'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER (Parties, Dates, Document Type - sm) | en | ['PARTY', 'EFFDATE', 'DOC', 'ALIAS'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER (Headers / Subheaders) | en | ['SIGNING_TITLE', 'SIGNING_PERSON', 'PARTY'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal ORG, PRODUCT and ALIAS NER (small) | en | ['ORG', 'PROD', 'ALIAS'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal NER Obligations on Agreements | en | ['OBLIGATION_SUBJECT', 'OBLIGATION_ACTION', 'OBLIGATION', 'OBLIGATION_INDIRECT_OBJECT'] | ['Legal NLP 1.0', 'Legal NLP'] |
Legal Zero-shot NER | en | [] | ['Legal NLP 1.0', 'Legal NLP'] |
For the complete list, check the NLP Models Hub.
! pip install -q johnsnowlabs
Using my.johnsnowlabs.com SSO
from johnsnowlabs import nlp, legal
# nlp.install(force_browser=True)
If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.
from google.colab import files
print('Please Upload your John Snow Labs License using the button below')
license_keys = files.upload()
nlp.install()
from johnsnowlabs import nlp, legal, viz
# Automatically load license data and start a session with all jars user has access to
spark = nlp.start()
👌 Detected license file /content/spark_nlp_for_healthcare_spark_ocr_7187 (3).json
👌 Launched cpu optimized session with with: 🚀Spark-NLP==4.3.0, 💊Spark-Healthcare==4.3.0, running on ⚡ PySpark==3.1.2
from pyspark.sql import DataFrame
import pyspark.sql.functions as F
import pyspark.sql.types as T
import pyspark.sql as SQL
from pyspark import keyword_only
Getting an example agreement document, which we will use throughout this notebook to exemplify the real-world usage of our models.
! wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/legal-nlp/data/credit_agreement.txt
credit_agreement = open("credit_agreement.txt", "r", encoding="utf8").read()
# First page - note the "-i-" at the end
print(credit_agreement[:4650])
Exhibit 10.1 EXECUTION COPY $225,000,000.00 REVOLVING CREDIT FACILITY CREDIT AGREEMENT by and among P.H. GLATFELTER COMPANY and Certain of its Subsidiaries, as Borrowers and THE BANKS PARTY HERETO, as Lenders and PNC BANK, NATIONAL ASSOCIATION, as Administrative Agent with PNC CAPITAL MARKETS LLC and CITIZENS BANK OF PENNSYLVANIA, as Joint Lead Arrangers and Joint Bookrunners and CITIZENS BANK OF PENNSYLVANIA, as Syndication Agent Dated as of April 29, 2010 TABLE OF CONTENTS Section Page ------- ---- 1. CERTAIN DEFINITIONS.................................................. 1 1.1 Certain Definitions............................................ 1 1.2 Construction................................................... 28 1.2.1 Number; Inclusion...................................... 28 1.2.2 Determination.......................................... 28 1.2.3 Administrative Agent's Discretion and Consent.......... 28 1.2.4 Documents Taken as a Whole............................. 28 1.2.5 Headings............................................... 29 1.2.6 Implied References to this Agreement................... 29 1.2.7 Persons................................................ 29 1.2.8 Modifications to Documents............................. 29 1.2.9 From, To and Through................................... 29 1.2.10 Shall; Will............................................ 29 1.2.11 Quebec Matters......................................... 29 1.3 Accounting Principles.......................................... 30 2. REVOLVING CREDIT AND SWING LOAN FACILITIES........................... 31 2.1 Revolving Credit Commitments................................... 31 2.1.1 Revolving Credit Loans................................. 31 2.1.2 Swing Loan Commitment.................................. 33 2.2 Nature of Lenders' Obligations with Respect to Revolving Credit Loans.......................................................... 33 2.3 Commitment Fees................................................ 33 2.4 Revolving Credit Loan Requests................................. 34 2.4.1 Revolving Credit Loan Requests......................... 34 2.4.2 Swing Loan Requests.................................... 34 2.5 Making Revolving Credit Loans and Swing Loans; Revolving Credit Notes and Swing Notes.......................................... 35 2.5.1 Making Revolving Credit Loans.......................... 35 2.5.2 Making Swing Loans..................................... 35 2.6 Revolving Credit Notes......................................... 35 2.7 Swing Loan Note................................................ 35 2.8 Borrowings to Repay Swing Loans................................ 36 2.9 Utilization of Commitments in Optional Currencies.............. 36 2.9.1 Periodic Computations of Dollar Equivalent Amounts of Revolving Credit Loans and Letters of Credit Outstanding............................................ 36 2.9.2 Notices From Lenders That Optional Currencies Are Unavailable to Fund New Loans.......................... 36 2.9.3 Notices From Lenders That Optional Currencies Are Unavailable to Fund Renewals of the Euro-Rate Option... 37 2.9.4 European Monetary Union................................ 37 -i-
Sometimes, pages have patterns which tell you how to split them. In our case, the page number
was present in the bottom of our documents.
📚Feel free to always analyze for signals when trying to detect pages boundaries. Patterns you can usually find in the bottom of a page:
document_assembler = (
nlp.DocumentAssembler().setInputCol("text").setOutputCol("document")
)
text_splitter = (
legal.TextSplitter()
.setInputCols(["document"])
.setOutputCol("pages")
.setCustomBounds(["\n+\s*[0-9]+\s*\n+", "[-][iv]+[-]"])
.setUseCustomBoundsOnly(True)
.setExplodeSentences(True)
)
page_splitting_pipeline = nlp.Pipeline(stages=[document_assembler, text_splitter])
📜Explanation:
.setCustomBounds(["\n+\s*[0-9]+\s*\n+", "[-][iv]+[-]"])
sets an array of regular expression(s) to tell the annotator how to split the document. The first regular expression identifies the page numbers of the document, and the second regular expression identifies the initial numbers in roman numerals enclosed by dash (-i-, -ii-, etc.) - only up to -viii-
, which was manually checked, we could add more roman numerals identifiers if needed..setUseCustomBoundsOnly(True)
the default behaviour of TextSplitter is Text Splitting, so we set to ignore the default regex ('\n', ...)..setExplodeSentences(True)
creates one new row in the dataframe per split, instead of an array containing the splits.sdf = spark.createDataFrame([[ credit_agreement ]]).toDF("text")
page_splitter_model = page_splitting_pipeline.fit(sdf)
%%time
#transforms: executes inference on a fit pipeline
res = page_splitter_model.transform(sdf)
# by selecting/showing/collecting the operations are performed
res.select('pages').show()
+--------------------+ | pages| +--------------------+ |[{document, 70, 4...| |[{document, 4678,...| |[{document, 8160,...| |[{document, 11801...| |[{document, 15298...| |[{document, 18938...| |[{document, 22461...| |[{document, 25295...| |[{document, 26399...| |[{document, 31821...| |[{document, 34560...| |[{document, 37458...| |[{document, 40734...| |[{document, 43945...| |[{document, 46511...| |[{document, 49420...| |[{document, 53065...| |[{document, 56539...| |[{document, 59694...| |[{document, 62708...| +--------------------+ only showing top 20 rows CPU times: user 101 ms, sys: 6.2 ms, total: 107 ms Wall time: 11.9 s
# Checking the first page
print(res.select('pages.result').take(1)[0].result[0])
Exhibit 10.1 EXECUTION COPY $225,000,000.00 REVOLVING CREDIT FACILITY CREDIT AGREEMENT by and among P.H. GLATFELTER COMPANY and Certain of its Subsidiaries, as Borrowers and THE BANKS PARTY HERETO, as Lenders and PNC BANK, NATIONAL ASSOCIATION, as Administrative Agent with PNC CAPITAL MARKETS LLC and CITIZENS BANK OF PENNSYLVANIA, as Joint Lead Arrangers and Joint Bookrunners and CITIZENS BANK OF PENNSYLVANIA, as Syndication Agent Dated as of April 29, 2010 TABLE OF CONTENTS Section Page ------- ---- 1. CERTAIN DEFINITIONS.................................................. 1 1.1 Certain Definitions............................................ 1 1.2 Construction................................................... 28 1.2.1 Number; Inclusion...................................... 28 1.2.2 Determination.......................................... 28 1.2.3 Administrative Agent's Discretion and Consent.......... 28 1.2.4 Documents Taken as a Whole............................. 28 1.2.5 Headings............................................... 29 1.2.6 Implied References to this Agreement................... 29 1.2.7 Persons................................................ 29 1.2.8 Modifications to Documents............................. 29 1.2.9 From, To and Through................................... 29 1.2.10 Shall; Will............................................ 29 1.2.11 Quebec Matters......................................... 29 1.3 Accounting Principles.......................................... 30 2. REVOLVING CREDIT AND SWING LOAN FACILITIES........................... 31 2.1 Revolving Credit Commitments................................... 31 2.1.1 Revolving Credit Loans................................. 31 2.1.2 Swing Loan Commitment.................................. 33 2.2 Nature of Lenders' Obligations with Respect to Revolving Credit Loans.......................................................... 33 2.3 Commitment Fees................................................ 33 2.4 Revolving Credit Loan Requests................................. 34 2.4.1 Revolving Credit Loan Requests......................... 34 2.4.2 Swing Loan Requests.................................... 34 2.5 Making Revolving Credit Loans and Swing Loans; Revolving Credit Notes and Swing Notes.......................................... 35 2.5.1 Making Revolving Credit Loans.......................... 35 2.5.2 Making Swing Loans..................................... 35 2.6 Revolving Credit Notes......................................... 35 2.7 Swing Loan Note................................................ 35 2.8 Borrowings to Repay Swing Loans................................ 36 2.9 Utilization of Commitments in Optional Currencies.............. 36 2.9.1 Periodic Computations of Dollar Equivalent Amounts of Revolving Credit Loans and Letters of Credit Outstanding............................................ 36 2.9.2 Notices From Lenders That Optional Currencies Are Unavailable to Fund New Loans.......................... 36 2.9.3 Notices From Lenders That Optional Currencies Are Unavailable to Fund Renewals of the Euro-Rate Option... 37 2.9.4 European Monetary Union................................ 37
Let's keep the pages in a new data frame.
pages = res.select(F.expr("pages.result[0] as page"))
pages.show()
+--------------------+ | page| +--------------------+ |Exhibit 10.1 ...| |TABLE OF CONTENTS...| |TABLE OF CONTENTS...| |TABLE OF CONTENTS...| |TABLE OF CONTENTS...| |TABLE OF CONTENTS...| |TABLE OF CONTENTS...| |LIST OF SCHEDULES...| |CREDIT AGREEMENT ...| |AUGMENTING LENDER...| |BUSINESS DAY shal...| |COMPLIANCE CERTIF...| |immediately prece...| |DECLINED SHARE sh...| |or directives iss...| |EURO-RATE shall m...| |Administrative Ag...| |EURO-RATE OPTION ...| |rate as quoted by...| |XXXXXXX NOTE shal...| +--------------------+ only showing top 20 rows
pages.limit(1).show(truncate=False)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |page | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |Exhibit 10.1 EXECUTION COPY $225,000,000.00 REVOLVING CREDIT FACILITY CREDIT AGREEMENT by and among P.H. GLATFELTER COMPANY and Certain of its Subsidiaries, as Borrowers and THE BANKS PARTY HERETO, as Lenders and PNC BANK, NATIONAL ASSOCIATION, as Administrative Agent with PNC CAPITAL MARKETS LLC and CITIZENS BANK OF PENNSYLVANIA, as Joint Lead Arrangers and Joint Bookrunners and CITIZENS BANK OF PENNSYLVANIA, as Syndication Agent Dated as of April 29, 2010 TABLE OF CONTENTS Section Page ------- ---- 1. CERTAIN DEFINITIONS.................................................. 1 1.1 Certain Definitions............................................ 1 1.2 Construction................................................... 28 1.2.1 Number; Inclusion...................................... 28 1.2.2 Determination.......................................... 28 1.2.3 Administrative Agent's Discretion and Consent.......... 28 1.2.4 Documents Taken as a Whole............................. 28 1.2.5 Headings............................................... 29 1.2.6 Implied References to this Agreement................... 29 1.2.7 Persons................................................ 29 1.2.8 Modifications to Documents............................. 29 1.2.9 From, To and Through................................... 29 1.2.10 Shall; Will............................................ 29 1.2.11 Quebec Matters......................................... 29 1.3 Accounting Principles.......................................... 30 2. REVOLVING CREDIT AND SWING LOAN FACILITIES........................... 31 2.1 Revolving Credit Commitments................................... 31 2.1.1 Revolving Credit Loans................................. 31 2.1.2 Swing Loan Commitment.................................. 33 2.2 Nature of Lenders' Obligations with Respect to Revolving Credit Loans.......................................................... 33 2.3 Commitment Fees................................................ 33 2.4 Revolving Credit Loan Requests................................. 34 2.4.1 Revolving Credit Loan Requests......................... 34 2.4.2 Swing Loan Requests.................................... 34 2.5 Making Revolving Credit Loans and Swing Loans; Revolving Credit Notes and Swing Notes.......................................... 35 2.5.1 Making Revolving Credit Loans.......................... 35 2.5.2 Making Swing Loans..................................... 35 2.6 Revolving Credit Notes......................................... 35 2.7 Swing Loan Note................................................ 35 2.8 Borrowings to Repay Swing Loans................................ 36 2.9 Utilization of Commitments in Optional Currencies.............. 36 2.9.1 Periodic Computations of Dollar Equivalent Amounts of Revolving Credit Loans and Letters of Credit Outstanding............................................ 36 2.9.2 Notices From Lenders That Optional Currencies Are Unavailable to Fund New Loans.......................... 36 2.9.3 Notices From Lenders That Optional Currencies Are Unavailable to Fund Renewals of the Euro-Rate Option... 37 2.9.4 European Monetary Union................................ 37| +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
We will use a few pretrained binary classifier models to try to identify clause types in each page. We will also replace all \n
(linebreak) from the pages' text to avoid extra tokenization (we keep the cleaned text in the column page_clean
).
pages = pages.withColumn("page_clean", F.regexp_replace("page", "\s+", " "))
pages.limit(1).select("page_clean").show(truncate=150)
+------------------------------------------------------------------------------------------------------------------------------------------------------+ | page_clean| +------------------------------------------------------------------------------------------------------------------------------------------------------+ |Exhibit 10.1 EXECUTION COPY $225,000,000.00 REVOLVING CREDIT FACILITY CREDIT AGREEMENT by and among P.H. GLATFELTER COMPANY and Certain of its Subs...| +------------------------------------------------------------------------------------------------------------------------------------------------------+
document_assembler = (
nlp.DocumentAssembler().setInputCol("page_clean").setOutputCol("document")
)
embeddings = (
nlp.BertSentenceEmbeddings.pretrained("sent_bert_base_cased", "en")
.setInputCols("document")
.setOutputCol("sentence_embeddings")
)
whereas_classifier = (
legal.ClassifierDLModel.pretrained(
"legclf_cuad_whereas_clause", "en", "legal/models"
)
.setInputCols(["sentence_embeddings"])
.setOutputCol("is_whereas")
)
warranty_classifier = (
legal.ClassifierDLModel.pretrained(
"legclf_cuad_warranty_clause", "en", "legal/models"
)
.setInputCols(["sentence_embeddings"])
.setOutputCol("is_warranty")
)
termination_classifier = (
legal.ClassifierDLModel.pretrained("legclf_termination_md", "en", "legal/models")
.setInputCols(["sentence_embeddings"])
.setOutputCol("is_termination")
)
pipeline = nlp.Pipeline(
stages=[
document_assembler,
embeddings,
whereas_classifier,
warranty_classifier,
termination_classifier,
]
)
model = pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
sent_bert_base_cased download started this may take some time. Approximate size to download 389.1 MB [OK!] legclf_cuad_whereas_clause download started this may take some time. [OK!] legclf_cuad_warranty_clause download started this may take some time. [OK!] legclf_termination_md download started this may take some time. [OK!]
result = model.transform(pages)
result.select(
F.expr("is_whereas.result as whereas"),
F.expr("is_warranty.result as survival"),
F.expr("is_termination.result as termination"),
).show()
+---------+----------+-----------+ | whereas| survival|termination| +---------+----------+-----------+ |[whereas]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]|[warranty]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| |[whereas]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| | [other]| [other]| [other]| +---------+----------+-----------+ only showing top 20 rows
How many survival clauses on pages❓
result.select(F.expr("is_whereas.result[0] as whereas")).filter(
"whereas != 'other'"
).count()
3
result.filter("is_whereas.result[0] == 'whereas'").select(
"page_clean", F.expr("is_whereas.result[0] as whereas")
).show(3, truncate=80)
+--------------------------------------------------------------------------------+-------+ | page_clean|whereas| +--------------------------------------------------------------------------------+-------+ |Exhibit 10.1 EXECUTION COPY $225,000,000.00 REVOLVING CREDIT FACILITY CREDIT ...|whereas| |CREDIT AGREEMENT THIS CREDIT AGREEMENT is dated as of April 29, 2010, and is ...|whereas| |ANY KIND ARISING OUT OF OR RELATED TO THIS AGREEMENT OR ANY OTHER LOAN DOCUME...|whereas| +--------------------------------------------------------------------------------+-------+
How many warranty clauses on pages❓
result.select(F.expr("is_warranty.result[0] as warranty")).filter(
"warranty != 'other'"
).count()
9
result.filter("is_warranty.result[0] == 'warranty'").select(
"page_clean", F.expr("is_warranty.result[0] as warranty")
).show(9, truncate=80)
+--------------------------------------------------------------------------------+--------+ | page_clean|warranty| +--------------------------------------------------------------------------------+--------+ |TABLE OF CONTENTS Section Page ------- ---- 5.1.14 Patents, Trademarks, Copyr...|warranty| |Document as a whole and not to any particular provision of this Agreement or ...|warranty| |(v) the lack of power or authority of any signer of (or any defect in or forg...|warranty| |such taxes, fees, assessments and other charges are being contested in good f...|warranty| |5.1.16 COMPLIANCE WITH LAWS. The Loan Parties and their Subsidiaries are in c...|warranty| |(ii) To the best of the Loan Parties' knowledge, each Multiemployer Plan and ...|warranty| |9.18 NO RELIANCE ON ADMINISTRATIVE AGENT'S CUSTOMER IDENTIFICATION PROGRAM. E...|warranty| |ANY KIND ARISING OUT OF OR RELATED TO THIS AGREEMENT OR ANY OTHER LOAN DOCUME...|warranty| |COBANK, ACB, as a Lender By: /s/ Xxxxxxx X. Norte ---------------------------...|warranty| +--------------------------------------------------------------------------------+--------+
How many termination clauses on pages❓
result.select(F.expr("is_termination.result[0] as termination")).filter(
"termination != 'other'"
).count()
7
result.filter("is_termination.result[0] == 'termination'").select(
"page_clean", F.expr("is_termination.result[0] as termination")
).show(7, truncate=80)
+--------------------------------------------------------------------------------+-----------+ | page_clean|termination| +--------------------------------------------------------------------------------+-----------+ |Fees shall be payable quarterly in arrears on the first day of each July, Oct...|termination| |the calculation of Equivalent Amounts which thereafter are actually in effect...|termination| |certificate to the other Lenders and the Borrowers. Upon such date as shall b...|termination| |normal banking procedures each Lender could purchase the Original Currency wi...|termination| |7.1.12 GERMAN AND ENGLISH BORROWERS. On or before the Closing Date, and such ...|termination| |(H) the Loan Parties shall deliver to the Administrative Agent at least five ...|termination| |8.1.4 BREACH OF OTHER COVENANTS. Any of the Loan Parties shall default in the...|termination| +--------------------------------------------------------------------------------+-----------+
Now that we found the clauses in each page, we select one NER model to identify the entities present in the whereas
clauses. Our pretrained model can identify the following entities:
WHEREAS_SUBJECT
WHEREAS_OBJECT
WHEREAS_ACTION
We will filter the whereas clauses and extract them as raw text, so we can build a new pipeline from scratch.
Note: The model was trained with
Roberta Embeddings
instead ofBert
.
In addition to these entities, we will also use other models to identify person, organization, location, and dates. We can use the ChunkMergeApproach
to merge two NER chunks in an unified field, containing all the entities.
document_assembler = (
nlp.DocumentAssembler().setInputCol("page_clean").setOutputCol("document")
)
text_splitter = (
legal.TextSplitter().setInputCols(["document"]).setOutputCol("page_sentence")
)
tokenizer = nlp.Tokenizer().setInputCols(["page_sentence"]).setOutputCol("token")
embeddings = (
nlp.WordEmbeddingsModel.pretrained("w2v_cc_300d", "en")
.setInputCols(["page_sentence", "token"])
.setOutputCol("embeddings")
)
roberta_embeddings = (
nlp.RoBertaEmbeddings.pretrained("roberta_embeddings_legal_roberta_base", "en")
.setInputCols(["page_sentence", "token"])
.setOutputCol("roberta")
.setMaxSentenceLength(512)
)
ner_whereas = (
legal.NerModel.pretrained("legner_whereas_md", "en", "legal/models")
.setInputCols(["page_sentence", "token", "roberta"])
.setOutputCol("ner_whereas")
)
ner_converter_whereas = (
legal.NerConverterInternal()
.setInputCols(["page_sentence", "token", "ner_whereas"])
.setOutputCol("ner_chunk_whereas")
)
ner_generic = (
legal.NerModel.pretrained("legner_cuad_silver", "en", "legal/models")
.setInputCols(["page_sentence", "token", "embeddings"])
.setOutputCol("ner_generic")
)
ner_converter_generic = (
legal.NerConverterInternal()
.setInputCols(["page_sentence", "token", "ner_generic"])
.setOutputCol("ner_chunk_generic")
.setGreedyMode(True)
)
chunk_merge = (
legal.ChunkMergeApproach()
.setInputCols("ner_chunk_whereas", "ner_chunk_generic")
.setOutputCol("merged_chunk")
)
ner_pipeline = nlp.Pipeline(
stages=[
document_assembler,
text_splitter,
tokenizer,
embeddings,
roberta_embeddings,
ner_whereas,
ner_converter_whereas,
ner_generic,
ner_converter_generic,
chunk_merge
]
)
ner_model = ner_pipeline.fit(spark.createDataFrame([[""]]).toDF("page_clean"))
w2v_cc_300d download started this may take some time. Approximate size to download 1.2 GB [OK!] roberta_embeddings_legal_roberta_base download started this may take some time. Approximate size to download 447.2 MB [OK!] legner_whereas_md download started this may take some time. [OK!] legner_cuad_silver download started this may take some time. [OK!]
example_clauses = result.filter("is_whereas.result[0] == 'whereas'").select(
"page_clean"
)
example_clauses.show(1)
+--------------------+ | page_clean| +--------------------+ |Exhibit 10.1 EXEC...| +--------------------+ only showing top 1 row
ner_results = ner_model.transform(example_clauses)
ner_results.select(
F.explode(F.arrays_zip(ner_results.merged_chunk.result, ner_results.merged_chunk.metadata)).alias(
"cols"
)
).select(
F.expr("cols['0']").alias("chunk"),
F.expr("cols['1']['entity']").alias("ner_label"),
F.expr("cols['1']['confidence']").alias("confidence"),
).show(300, truncate=False)
+-----------------------------------------------------------------------+---------------+----------+ |chunk |ner_label |confidence| +-----------------------------------------------------------------------+---------------+----------+ |EXECUTION COPY |DOC |0.99565 | |REVOLVING CREDIT FACILITY CREDIT AGREEMENT |DOC |0.8406199 | |GLATFELTER COMPANY |PARTY |0.52175 | |PNC BANK, NATIONAL ASSOCIATION, as Administrative Agent |ORG |0.87597775| |PNC CAPITAL MARKETS LLC |ORG |0.631425 | |CITIZENS BANK OF PENNSYLVANIA |ORG |0.6161 | |Joint Lead Arrangers and Joint Bookrunners |ORG |0.8393666 | |CITIZENS BANK OF PENNSYLVANIA |ORG |0.603675 | |April 29, 2010 |DATE |0.95145 | |CERTAIN DEFINITIONS |ORG |0.43734998| |Construction |ORG |0.9439 | |28 1.2.1 Number |DATE |0.6659667 | |Inclusion |ORG |0.9177 | |28 1.2.2 Determination |DATE |0.6773667 | |Whole |ORG |0.7297 | |28 1.2.5 Headings |DATE |0.67876667| |Agreement |DOC |0.9939 | |29 1.2.7 |DATE |0.7389 | |Persons |ORG |0.9189 | |Documents |ORG |0.5535 | |29 1.2.9 |DATE |0.90805 | |29 1.2.10 |DATE |0.7398 | |Shall |ORG |0.6872 | |Will |DOC |0.9934 | |Quebec |LOC |0.793 | |Accounting Principles |ORG |0.51035 | |30 |DATE |0.85 | |REVOLVING CREDIT |DOC |0.74405 | |Credit |DOC |0.9644 | |Credit |DOC |0.803 | |Swing Loan Commitment |ORG |0.72103333| |Credit |DOC |0.6604 | |Credit |DOC |0.8468 | |Credit |DOC |0.8577 | |Swing Loan Requests |ORG |0.6699 | |Credit |DOC |0.8247 | |Credit |DOC |0.9194 | |Credit |DOC |0.8865 | |Making Swing Loans |ORG |0.7957333 | |Credit |DOC |0.9688 | |Loan Note |DOC |0.66545 | |Repay Swing Loans |ORG |0.6506 | |Credit |DOC |0.9285 | |Letters of Credit |DOC |0.97900003| |Fund New Loans |ORG |0.5782 | |European Monetary Union |ORG |0.5827 | |CREDIT AGREEMENT |DOC |0.99310005| |CREDIT AGREEMENT |DOC |0.9901 | |April 29, 2010 |DATE |0.8848001 | |GLATFELTER COMPANY |ORG |0.6135 | |Pennsylvania |LOC |0.7951 | |the "COMPANY") |ORG |0.78772503| |BORROWER |ROLE |0.9892 | |PNC BANK, NATIONAL ASSOCIATION, |ORG |0.7839833 | |Agreement |DOC |0.9553 | |PNC CAPITAL MARKETS LLC |PARTY |0.8001 | |CITIZENS BANK OF PENNSYLVANIA |ORG |0.549325 | |CITIZENS BANK OF PENNSYLVANIA |ORG |0.631575 | |the Borrowers |WHEREAS_SUBJECT|0.9951 | |have requested |WHEREAS_ACTION |0.97775 | |the Lenders to provide a revolving credit facility |WHEREAS_OBJECT |0.80509996| |proceeds of the |WHEREAS_SUBJECT|0.84663326| |credit facility |DOC |0.82975 | |WHEREAS |ORG |0.9781 | |the Lenders |WHEREAS_SUBJECT|0.9569 | |are willing to provide |WHEREAS_ACTION |0.836 | |such credit |WHEREAS_OBJECT |0.85765004| |the parties |WHEREAS_SUBJECT|0.96730006| |DEFINITIONS 1.1 CERTAIN DEFINITIONS |LAW |0.70685 | |Agreement |DOC |0.9834 | |2006 |DATE |0.9707 | |2006 |DATE |0.9988 | |May 1, 2016 |DATE |0.97282505| |the Loan Parties |ORG |0.76336664| |2010 |DATE |0.9851 | |2010 |DATE |0.9993 | |May 1, 2016 |DATE |0.97245 | |the Loan Parties |ORG |0.77570003| |RECEIVABLE FACILITY |DOC |0.7375 | |Company |ORG |0.9981 | |the Receivables Entity |ORG |0.60770005| |a Permitted Accounts Receivable Program |ORG |0.54974 | |ADMINISTRATIVE AGENT |PARTY |0.67195 | |Section 9.15 |LAW |0.99465 | |AGENT'S LETTER |DOC |0.55605 | |Section 9.15 |LAW |0.9943 | |AFFILIATE |ROLE |0.4648 | |Person |ORG |0.8968 | |Person |ORG |0.9664 | |contract |DOC |0.9829 | |AGREEMENT |DOC |0.8701 | |Credit Agreement |DOC |0.99915 | |Executive Order No. 13224 |LAW |0.70444 | |the USA Patriot |LAW |0.4611 | |Act |DOC |0.9427 | |Act |DOC |0.9735 | |the United States Treasury Department's Office of Foreign Asset Control|ORG |0.52493 | |Laws |LAW |0.9754 | |SCHEDULE 1.1(A |LAW |0.80805004| |SCHEDULE 1.1(A |LAW |0.98765004| |Euro-Rate |ORG |0.9422 | |SCHEDULE 1.1(A |LAW |0.7896 | |SCHEDULE 1.1(A |LAW |0.9899 | |ASSIGNMENT AND ASSUMPTION AGREEMENT |DOC |0.74009997| |Assumption Agreement |DOC |0.8543 | |EXHIBIT 1.1(A |LAW |0.71099997| |AGREEMENT |DOC |0.9904 | |LOAN DOCUMENT |DOC |0.95755005| |10.17.1 TAX WITHHOLDING |ORG |0.7043667 | |the United States of America |LOC |0.79712 | |the Administrative Agent |ORG |0.81193334| |Company |ORG |0.9975 | |the Administrative Agent |ORG |0.77356666| |Section 1.1441-1 |LAW |0.9625 | |the Income Tax Regulations |LAW |0.64004993| |U.S. |LOC |0.87049997| |U.S |LOC |0.8581 | |treaty |DOC |0.8876 | |the Internal Revenue Code |LAW |0.58395 | |CERTIFICATE |DOC |0.9479 | |Form |DOC |0.999 | |Form |DOC |0.9962 | |Form |DOC |0.9922 | |W-8ECI |ORG |0.6855 | |Form |DOC |0.9803 | |Section 1.1441-1 |LAW |0.95815 | |Regulations |LAW |0.9789 | |statement |DOC |0.9981 | |Section 1.871-14 |LAW |0.90325 | |Regulations |LAW |0.9828 | |the Internal Revenue Code |LAW |0.63045 | |owner |ROLE |1.0 | |U.S. |LOC |0.78575 | |Company |ORG |0.9976 | |Certificate |DOC |0.9994 | |Certificate |DOC |0.9994 | |Certificate |DOC |0.9985 | |Business Days |DATE |0.87845004| |first |ORDINAL |0.5447 | |Certificate |DOC |0.998 | |Business Days |DATE |0.9174 | |the Administrative Agent |ORG |0.86280006| |permit |DOC |0.9784 | |Certificate |DOC |0.9983 | |Business Days |DATE |0.89515 | |the Administrative Agent |ORG |0.8864667 | |Certificate |DOC |0.9996 | |Company |ORG |0.9968 | |the Administrative Agent |ORG |0.7749667 | |Certificate |DOC |0.9994 | |Certificate |DOC |0.9994 | |Certificate |DOC |0.9995 | |Borrowers |ORG |0.9825 | |the Administrative Agent |ORG |0.8297 | |the United States of America |LOC |0.78362 | |the Administrative Agent |ORG |0.8483667 | |agreement |DOC |0.9514 | |U.S |LOC |0.8164 | |U.S |LOC |0.8927 | |agreement |DOC |0.9969 | |agreement |DOC |0.9893 | |the United States |LOC |0.8441 | +-----------------------------------------------------------------------+---------------+----------+
Using the visualization package:
ner_visualizer = viz.NerVisualizer()
results_collected = ner_results.collect()
ner_visualizer.display(
results_collected[1], label_col="merged_chunk", document_col="document"
)
LightPipelines are Spark NLP specific Pipelines, equivalent to Spark ML Pipeline, but meant to deal with smaller amounts of data. They’re useful working with small datasets, debugging results, or when running either training or prediction from an API that serves one-off requests.
Spark NLP LightPipelines are Spark ML pipelines converted into a single machine but the multi-threaded task, becoming more than 10x times faster for smaller amounts of data (small is relative, but 50k sentences are roughly a good maximum). To use them, we simply plug in a trained (fitted) pipeline and then annotate a plain text. We don't even need to convert the input text to DataFrame in order to feed it into a pipeline that's accepting DataFrame as an input in the first place. This feature would be quite useful when it comes to getting a prediction for a few lines of text from a trained ML model.
For more details: https://medium.com/spark-nlp/spark-nlp-101-lightpipeline-a544e93f20f1
light_model = nlp.LightPipeline(ner_model)
You can use strings or list of strings with the method .annotate() to get the results. To get more metadata in the result, use the method .fullAnnotate() instead. The result is a list
if a list
is given, or a dict
if a string was given.
To extract the results from the object, you just need to parse the dictionary.
text = results_collected[1].page_clean
lp_results = light_model.annotate(text)
lp_results.keys()
dict_keys(['ner_whereas', 'document', 'merged_chunk', 'ner_chunk_generic', 'token', 'page_sentence', 'embeddings', 'ner_chunk_whereas', 'roberta', 'ner_generic'])
# List with all the chunks
lp_results["merged_chunk"]
['CREDIT AGREEMENT', 'CREDIT AGREEMENT', 'April 29, 2010', 'GLATFELTER COMPANY', 'Pennsylvania', 'the "COMPANY")', 'BORROWER', 'PNC BANK, NATIONAL ASSOCIATION,', 'Agreement', 'PNC CAPITAL MARKETS LLC', 'CITIZENS BANK OF PENNSYLVANIA', 'CITIZENS BANK OF PENNSYLVANIA', 'the Borrowers', 'have requested', 'the Lenders to provide a revolving credit facility', 'proceeds of the', 'credit facility', 'WHEREAS', 'the Lenders', 'are willing to provide', 'such credit', 'the parties', 'DEFINITIONS 1.1 CERTAIN DEFINITIONS', 'Agreement', '2006', '2006', 'May 1, 2016', 'the Loan Parties', '2010', '2010', 'May 1, 2016', 'the Loan Parties', 'RECEIVABLE FACILITY', 'Company', 'the Receivables Entity', 'a Permitted Accounts Receivable Program', 'ADMINISTRATIVE AGENT', 'Section 9.15', "AGENT'S LETTER", 'Section 9.15', 'AFFILIATE', 'Person', 'Person', 'contract', 'AGREEMENT', 'Credit Agreement', 'Executive Order No. 13224', 'the USA Patriot', 'Act', 'Act', "the United States Treasury Department's Office of Foreign Asset Control", 'Laws', 'SCHEDULE 1.1(A', 'SCHEDULE 1.1(A', 'Euro-Rate', 'SCHEDULE 1.1(A', 'SCHEDULE 1.1(A', 'ASSIGNMENT AND ASSUMPTION AGREEMENT', 'Assumption Agreement', 'EXHIBIT 1.1(A']
We can see that the .annotate()
did't return the labels in the ner_chunk
item. How can we obtain them? Using the .fullAnnotate()
instead. This method always returns a list.
lp_results_full = light_model.fullAnnotate(text)
lp_results_full[0].keys()
dict_keys(['ner_whereas', 'document', 'merged_chunk', 'ner_chunk_generic', 'token', 'page_sentence', 'embeddings', 'ner_chunk_whereas', 'roberta', 'ner_generic'])
lp_results_full[0]["merged_chunk"]
[Annotation(chunk, 0, 15, CREDIT AGREEMENT, {'entity': 'DOC', 'confidence': '0.99310005', 'ner_source': 'ner_chunk_generic', 'chunk': '0', 'sentence': '0'}, []), Annotation(chunk, 22, 37, CREDIT AGREEMENT, {'entity': 'DOC', 'confidence': '0.9901', 'ner_source': 'ner_chunk_generic', 'chunk': '1', 'sentence': '0'}, []), Annotation(chunk, 54, 67, April 29, 2010, {'entity': 'DATE', 'confidence': '0.8848001', 'ner_source': 'ner_chunk_generic', 'chunk': '2', 'sentence': '0'}, []), Annotation(chunk, 100, 117, GLATFELTER COMPANY, {'entity': 'ORG', 'confidence': '0.6135', 'ner_source': 'ner_chunk_generic', 'chunk': '3', 'sentence': '1'}, []), Annotation(chunk, 122, 133, Pennsylvania, {'entity': 'LOC', 'confidence': '0.7951', 'ner_source': 'ner_chunk_generic', 'chunk': '4', 'sentence': '1'}, []), Annotation(chunk, 149, 162, the "COMPANY"), {'entity': 'ORG', 'confidence': '0.78772503', 'ner_source': 'ner_chunk_generic', 'chunk': '5', 'sentence': '1'}, []), Annotation(chunk, 246, 253, BORROWER, {'entity': 'ROLE', 'confidence': '0.9892', 'ner_source': 'ner_chunk_generic', 'chunk': '6', 'sentence': '1'}, []), Annotation(chunk, 379, 409, PNC BANK, NATIONAL ASSOCIATION,, {'entity': 'ORG', 'confidence': '0.7839833', 'ner_source': 'ner_chunk_generic', 'chunk': '7', 'sentence': '1'}, []), Annotation(chunk, 463, 471, Agreement, {'entity': 'DOC', 'confidence': '0.9553', 'ner_source': 'ner_chunk_generic', 'chunk': '8', 'sentence': '1'}, []), Annotation(chunk, 618, 640, PNC CAPITAL MARKETS LLC, {'entity': 'PARTY', 'confidence': '0.8001', 'ner_source': 'ner_chunk_generic', 'chunk': '9', 'sentence': '1'}, []), Annotation(chunk, 646, 674, CITIZENS BANK OF PENNSYLVANIA, {'entity': 'ORG', 'confidence': '0.549325', 'ner_source': 'ner_chunk_generic', 'chunk': '10', 'sentence': '1'}, []), Annotation(chunk, 723, 751, CITIZENS BANK OF PENNSYLVANIA, {'entity': 'ORG', 'confidence': '0.631575', 'ner_source': 'ner_chunk_generic', 'chunk': '11', 'sentence': '1'}, []), Annotation(chunk, 797, 809, the Borrowers, {'entity': 'WHEREAS_SUBJECT', 'confidence': '0.9951', 'ner_source': 'ner_chunk_whereas', 'chunk': '12', 'sentence': '2'}, []), Annotation(chunk, 811, 824, have requested, {'entity': 'WHEREAS_ACTION', 'confidence': '0.97775', 'ner_source': 'ner_chunk_whereas', 'chunk': '13', 'sentence': '2'}, []), Annotation(chunk, 826, 875, the Lenders to provide a revolving credit facility, {'entity': 'WHEREAS_OBJECT', 'confidence': '0.80509996', 'ner_source': 'ner_chunk_whereas', 'chunk': '14', 'sentence': '2'}, []), Annotation(chunk, 968, 982, proceeds of the, {'entity': 'WHEREAS_SUBJECT', 'confidence': '0.84663326', 'ner_source': 'ner_chunk_whereas', 'chunk': '15', 'sentence': '3'}, []), Annotation(chunk, 994, 1008, credit facility, {'entity': 'DOC', 'confidence': '0.82975', 'ner_source': 'ner_chunk_generic', 'chunk': '16', 'sentence': '3'}, []), Annotation(chunk, 1107, 1113, WHEREAS, {'entity': 'ORG', 'confidence': '0.9781', 'ner_source': 'ner_chunk_generic', 'chunk': '17', 'sentence': '4'}, []), Annotation(chunk, 1116, 1126, the Lenders, {'entity': 'WHEREAS_SUBJECT', 'confidence': '0.9569', 'ner_source': 'ner_chunk_whereas', 'chunk': '18', 'sentence': '4'}, []), Annotation(chunk, 1128, 1149, are willing to provide, {'entity': 'WHEREAS_ACTION', 'confidence': '0.836', 'ner_source': 'ner_chunk_whereas', 'chunk': '19', 'sentence': '4'}, []), Annotation(chunk, 1151, 1161, such credit, {'entity': 'WHEREAS_OBJECT', 'confidence': '0.85765004', 'ner_source': 'ner_chunk_whereas', 'chunk': '20', 'sentence': '4'}, []), Annotation(chunk, 1232, 1242, the parties, {'entity': 'WHEREAS_SUBJECT', 'confidence': '0.96730006', 'ner_source': 'ner_chunk_whereas', 'chunk': '21', 'sentence': '5'}, []), Annotation(chunk, 1416, 1450, DEFINITIONS 1.1 CERTAIN DEFINITIONS, {'entity': 'LAW', 'confidence': '0.70685', 'ner_source': 'ner_chunk_generic', 'chunk': '22', 'sentence': '6'}, []), Annotation(chunk, 1510, 1518, Agreement, {'entity': 'DOC', 'confidence': '0.9834', 'ner_source': 'ner_chunk_generic', 'chunk': '23', 'sentence': '7'}, []), Annotation(chunk, 1654, 1657, 2006, {'entity': 'DATE', 'confidence': '0.9707', 'ner_source': 'ner_chunk_generic', 'chunk': '24', 'sentence': '7'}, []), Annotation(chunk, 1726, 1729, 2006, {'entity': 'DATE', 'confidence': '0.9988', 'ner_source': 'ner_chunk_generic', 'chunk': '25', 'sentence': '7'}, []), Annotation(chunk, 1739, 1749, May 1, 2016, {'entity': 'DATE', 'confidence': '0.97282505', 'ner_source': 'ner_chunk_generic', 'chunk': '26', 'sentence': '7'}, []), Annotation(chunk, 1828, 1843, the Loan Parties, {'entity': 'ORG', 'confidence': '0.76336664', 'ner_source': 'ner_chunk_generic', 'chunk': '27', 'sentence': '7'}, []), Annotation(chunk, 1846, 1849, 2010, {'entity': 'DATE', 'confidence': '0.9851', 'ner_source': 'ner_chunk_generic', 'chunk': '28', 'sentence': '8'}, []), Annotation(chunk, 1918, 1921, 2010, {'entity': 'DATE', 'confidence': '0.9993', 'ner_source': 'ner_chunk_generic', 'chunk': '29', 'sentence': '8'}, []), Annotation(chunk, 1931, 1941, May 1, 2016, {'entity': 'DATE', 'confidence': '0.97245', 'ner_source': 'ner_chunk_generic', 'chunk': '30', 'sentence': '8'}, []), Annotation(chunk, 2020, 2035, the Loan Parties, {'entity': 'ORG', 'confidence': '0.77570003', 'ner_source': 'ner_chunk_generic', 'chunk': '31', 'sentence': '8'}, []), Annotation(chunk, 2047, 2065, RECEIVABLE FACILITY, {'entity': 'DOC', 'confidence': '0.7375', 'ner_source': 'ner_chunk_generic', 'chunk': '32', 'sentence': '9'}, []), Annotation(chunk, 2121, 2127, Company, {'entity': 'ORG', 'confidence': '0.9981', 'ner_source': 'ner_chunk_generic', 'chunk': '33', 'sentence': '9'}, []), Annotation(chunk, 2182, 2203, the Receivables Entity, {'entity': 'ORG', 'confidence': '0.60770005', 'ner_source': 'ner_chunk_generic', 'chunk': '34', 'sentence': '9'}, []), Annotation(chunk, 2312, 2350, a Permitted Accounts Receivable Program, {'entity': 'ORG', 'confidence': '0.54974', 'ner_source': 'ner_chunk_generic', 'chunk': '35', 'sentence': '9'}, []), Annotation(chunk, 2503, 2522, ADMINISTRATIVE AGENT, {'entity': 'PARTY', 'confidence': '0.67195', 'ner_source': 'ner_chunk_generic', 'chunk': '36', 'sentence': '10'}, []), Annotation(chunk, 2679, 2690, Section 9.15, {'entity': 'LAW', 'confidence': '0.99465', 'ner_source': 'ner_chunk_generic', 'chunk': '37', 'sentence': '11'}, []), Annotation(chunk, 2708, 2721, AGENT'S LETTER, {'entity': 'DOC', 'confidence': '0.55605', 'ner_source': 'ner_chunk_generic', 'chunk': '38', 'sentence': '12'}, []), Annotation(chunk, 2771, 2782, Section 9.15, {'entity': 'LAW', 'confidence': '0.9943', 'ner_source': 'ner_chunk_generic', 'chunk': '39', 'sentence': '12'}, []), Annotation(chunk, 2785, 2793, AFFILIATE, {'entity': 'ROLE', 'confidence': '0.4648', 'ner_source': 'ner_chunk_generic', 'chunk': '40', 'sentence': '13'}, []), Annotation(chunk, 2939, 2944, Person, {'entity': 'ORG', 'confidence': '0.8968', 'ner_source': 'ner_chunk_generic', 'chunk': '41', 'sentence': '14'}, []), Annotation(chunk, 3249, 3254, Person, {'entity': 'ORG', 'confidence': '0.9664', 'ner_source': 'ner_chunk_generic', 'chunk': '42', 'sentence': '16'}, []), Annotation(chunk, 3396, 3403, contract, {'entity': 'DOC', 'confidence': '0.9829', 'ner_source': 'ner_chunk_generic', 'chunk': '43', 'sentence': '17'}, []), Annotation(chunk, 3494, 3502, AGREEMENT, {'entity': 'DOC', 'confidence': '0.8701', 'ner_source': 'ner_chunk_generic', 'chunk': '44', 'sentence': '18'}, []), Annotation(chunk, 3520, 3535, Credit Agreement, {'entity': 'DOC', 'confidence': '0.99915', 'ner_source': 'ner_chunk_generic', 'chunk': '45', 'sentence': '18'}, []), Annotation(chunk, 3760, 3784, Executive Order No. 13224, {'entity': 'LAW', 'confidence': '0.70444', 'ner_source': 'ner_chunk_generic', 'chunk': '46', 'sentence': '19'}, []), Annotation(chunk, 3787, 3801, the USA Patriot, {'entity': 'LAW', 'confidence': '0.4611', 'ner_source': 'ner_chunk_generic', 'chunk': '47', 'sentence': '19'}, []), Annotation(chunk, 3803, 3805, Act, {'entity': 'DOC', 'confidence': '0.9427', 'ner_source': 'ner_chunk_generic', 'chunk': '48', 'sentence': '19'}, []), Annotation(chunk, 3861, 3863, Act, {'entity': 'DOC', 'confidence': '0.9735', 'ner_source': 'ner_chunk_generic', 'chunk': '49', 'sentence': '19'}, []), Annotation(chunk, 3895, 3965, the United States Treasury Department's Office of Foreign Asset Control, {'entity': 'ORG', 'confidence': '0.52493', 'ner_source': 'ner_chunk_generic', 'chunk': '50', 'sentence': '19'}, []), Annotation(chunk, 3992, 3995, Laws, {'entity': 'LAW', 'confidence': '0.9754', 'ner_source': 'ner_chunk_generic', 'chunk': '51', 'sentence': '19'}, []), Annotation(chunk, 4197, 4210, SCHEDULE 1.1(A, {'entity': 'LAW', 'confidence': '0.80805004', 'ner_source': 'ner_chunk_generic', 'chunk': '52', 'sentence': '20'}, []), Annotation(chunk, 4355, 4368, SCHEDULE 1.1(A, {'entity': 'LAW', 'confidence': '0.98765004', 'ner_source': 'ner_chunk_generic', 'chunk': '53', 'sentence': '20'}, []), Annotation(chunk, 4438, 4446, Euro-Rate, {'entity': 'ORG', 'confidence': '0.9422', 'ner_source': 'ner_chunk_generic', 'chunk': '54', 'sentence': '21'}, []), Annotation(chunk, 4583, 4596, SCHEDULE 1.1(A, {'entity': 'LAW', 'confidence': '0.7896', 'ner_source': 'ner_chunk_generic', 'chunk': '55', 'sentence': '21'}, []), Annotation(chunk, 4752, 4765, SCHEDULE 1.1(A, {'entity': 'LAW', 'confidence': '0.9899', 'ner_source': 'ner_chunk_generic', 'chunk': '56', 'sentence': '22'}, []), Annotation(chunk, 4769, 4803, ASSIGNMENT AND ASSUMPTION AGREEMENT, {'entity': 'DOC', 'confidence': '0.74009997', 'ner_source': 'ner_chunk_generic', 'chunk': '57', 'sentence': '23'}, []), Annotation(chunk, 4834, 4853, Assumption Agreement, {'entity': 'DOC', 'confidence': '0.8543', 'ner_source': 'ner_chunk_generic', 'chunk': '58', 'sentence': '23'}, []), Annotation(chunk, 5032, 5044, EXHIBIT 1.1(A, {'entity': 'LAW', 'confidence': '0.71099997', 'ner_source': 'ner_chunk_generic', 'chunk': '59', 'sentence': '23'}, [])]
Now we can see all the metadata in the annotation objects. Let's get the results in a tabular form.
results_tabular = []
for res in lp_results_full[0]["merged_chunk"]:
results_tabular.append(
(
res.begin,
res.end,
res.result,
res.metadata["entity"],
res.metadata["confidence"],
)
)
import pandas as pd
pd.DataFrame(results_tabular, columns=["begin", "end", "chunk", "entity", "confidence"])
begin | end | chunk | entity | confidence | |
---|---|---|---|---|---|
0 | 0 | 15 | CREDIT AGREEMENT | DOC | 0.99310005 |
1 | 22 | 37 | CREDIT AGREEMENT | DOC | 0.9901 |
2 | 54 | 67 | April 29, 2010 | DATE | 0.8848001 |
3 | 100 | 117 | GLATFELTER COMPANY | ORG | 0.6135 |
4 | 122 | 133 | Pennsylvania | LOC | 0.7951 |
5 | 149 | 162 | the "COMPANY") | ORG | 0.78772503 |
6 | 246 | 253 | BORROWER | ROLE | 0.9892 |
7 | 379 | 409 | PNC BANK, NATIONAL ASSOCIATION, | ORG | 0.7839833 |
8 | 463 | 471 | Agreement | DOC | 0.9553 |
9 | 618 | 640 | PNC CAPITAL MARKETS LLC | PARTY | 0.8001 |
10 | 646 | 674 | CITIZENS BANK OF PENNSYLVANIA | ORG | 0.549325 |
11 | 723 | 751 | CITIZENS BANK OF PENNSYLVANIA | ORG | 0.631575 |
12 | 797 | 809 | the Borrowers | WHEREAS_SUBJECT | 0.9951 |
13 | 811 | 824 | have requested | WHEREAS_ACTION | 0.97775 |
14 | 826 | 875 | the Lenders to provide a revolving credit faci... | WHEREAS_OBJECT | 0.80509996 |
15 | 968 | 982 | proceeds of the | WHEREAS_SUBJECT | 0.84663326 |
16 | 994 | 1008 | credit facility | DOC | 0.82975 |
17 | 1107 | 1113 | WHEREAS | ORG | 0.9781 |
18 | 1116 | 1126 | the Lenders | WHEREAS_SUBJECT | 0.9569 |
19 | 1128 | 1149 | are willing to provide | WHEREAS_ACTION | 0.836 |
20 | 1151 | 1161 | such credit | WHEREAS_OBJECT | 0.85765004 |
21 | 1232 | 1242 | the parties | WHEREAS_SUBJECT | 0.96730006 |
22 | 1416 | 1450 | DEFINITIONS 1.1 CERTAIN DEFINITIONS | LAW | 0.70685 |
23 | 1510 | 1518 | Agreement | DOC | 0.9834 |
24 | 1654 | 1657 | 2006 | DATE | 0.9707 |
25 | 1726 | 1729 | 2006 | DATE | 0.9988 |
26 | 1739 | 1749 | May 1, 2016 | DATE | 0.97282505 |
27 | 1828 | 1843 | the Loan Parties | ORG | 0.76336664 |
28 | 1846 | 1849 | 2010 | DATE | 0.9851 |
29 | 1918 | 1921 | 2010 | DATE | 0.9993 |
30 | 1931 | 1941 | May 1, 2016 | DATE | 0.97245 |
31 | 2020 | 2035 | the Loan Parties | ORG | 0.77570003 |
32 | 2047 | 2065 | RECEIVABLE FACILITY | DOC | 0.7375 |
33 | 2121 | 2127 | Company | ORG | 0.9981 |
34 | 2182 | 2203 | the Receivables Entity | ORG | 0.60770005 |
35 | 2312 | 2350 | a Permitted Accounts Receivable Program | ORG | 0.54974 |
36 | 2503 | 2522 | ADMINISTRATIVE AGENT | PARTY | 0.67195 |
37 | 2679 | 2690 | Section 9.15 | LAW | 0.99465 |
38 | 2708 | 2721 | AGENT'S LETTER | DOC | 0.55605 |
39 | 2771 | 2782 | Section 9.15 | LAW | 0.9943 |
40 | 2785 | 2793 | AFFILIATE | ROLE | 0.4648 |
41 | 2939 | 2944 | Person | ORG | 0.8968 |
42 | 3249 | 3254 | Person | ORG | 0.9664 |
43 | 3396 | 3403 | contract | DOC | 0.9829 |
44 | 3494 | 3502 | AGREEMENT | DOC | 0.8701 |
45 | 3520 | 3535 | Credit Agreement | DOC | 0.99915 |
46 | 3760 | 3784 | Executive Order No. 13224 | LAW | 0.70444 |
47 | 3787 | 3801 | the USA Patriot | LAW | 0.4611 |
48 | 3803 | 3805 | Act | DOC | 0.9427 |
49 | 3861 | 3863 | Act | DOC | 0.9735 |
50 | 3895 | 3965 | the United States Treasury Department's Office... | ORG | 0.52493 |
51 | 3992 | 3995 | Laws | LAW | 0.9754 |
52 | 4197 | 4210 | SCHEDULE 1.1(A | LAW | 0.80805004 |
53 | 4355 | 4368 | SCHEDULE 1.1(A | LAW | 0.98765004 |
54 | 4438 | 4446 | Euro-Rate | ORG | 0.9422 |
55 | 4583 | 4596 | SCHEDULE 1.1(A | LAW | 0.7896 |
56 | 4752 | 4765 | SCHEDULE 1.1(A | LAW | 0.9899 |
57 | 4769 | 4803 | ASSIGNMENT AND ASSUMPTION AGREEMENT | DOC | 0.74009997 |
58 | 4834 | 4853 | Assumption Agreement | DOC | 0.8543 |
59 | 5032 | 5044 | EXHIBIT 1.1(A | LAW | 0.71099997 |
In this section we will use the MultiClassifierDL
annotator to idenfity more than one classes in texts.
document_assembler = (
nlp.DocumentAssembler().setInputCol("text").setOutputCol("document")
)
embeddings = (
nlp.BertSentenceEmbeddings.pretrained("sent_bert_base_uncased_legal", "en")
.setInputCols("document")
.setOutputCol("sentence_embeddings")
)
multiClassifier = (
nlp.MultiClassifierDLModel.pretrained("legmulticlf_edgar", "en", "legal/models")
.setInputCols(["sentence_embeddings"])
.setOutputCol("class")
)
clf_pipeline = nlp.Pipeline(stages=[document_assembler, embeddings, multiClassifier])
light_pipeline = nlp.LightPipeline(
clf_pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
)
sent_bert_base_uncased_legal download started this may take some time. Approximate size to download 390.8 MB [OK!] legmulticlf_edgar download started this may take some time. Approximate size to download 13.3 MB [OK!]
We will experiment in simpler sentences to showcase it's capabilities.
result = light_pipeline.annotate(
"""No failure or delay by the Administrative Agent or any Lender in exercising any right or power hereunder shall operate as a waiver thereof, nor shall any single or partial exercise of any such right or power, or any abandonment or discontinuance of steps to enforce such a right or power, preclude any other or further exercise thereof or the exercise of any other right or power. The rights and remedies of the Administrative Agent and the Lenders hereunder are cumulative and are not exclusive of any rights or remedies that they would otherwise have. No waiver of any provision of this Agreement or consent to any departure by the Borrower therefrom shall in any event be effective unless the same shall be permitted by paragraph (b) of this Section, and then such waiver or consent shall be effective only in the specific instance and for the purpose for which given. Without limiting the generality of the foregoing, the making of a Loan shall not be construed as a waiver of any Default, regardless of whether the Administrative Agent or any Lender may have had notice or knowledge of such Default at the time.
""".lower()
)
result["class"]
['waivers']
result = light_pipeline.annotate(
"""The provisions of this Agreement shall be binding upon and inure to the benefit of the parties hereto and their respective successors and assigns permitted hereby (including any Affiliate of the Issuing Bank that issues any Letter of Credit), except that (i) the Borrower may not assign or otherwise transfer any of its rights or obligations hereunder without the prior written consent of each Lender (and any attempted assignment or transfer by the Borrower without such consent shall be null and void) and (ii) no Lender may assign or otherwise transfer its rights or obligations hereunder except in accordance with this Section. Nothing in this Agreement, expressed or implied, shall be construed to confer upon any Person (other than the parties hereto, their respective successors and assigns permitted hereby (including any Affiliate of the Issuing Bank that issues any Letter of Credit), Participants (to the extent provided in paragraph (c) of this Section) and, to the extent expressly contemplated hereby, the Related Parties of each of the Administrative Agent, the Issuing Bank and the Lenders) any legal or equitable right, remedy or claim under or by reason of this Agreement.
""".lower()
)
result["class"]
[]
result = light_pipeline.annotate(
"""After the effectiveness of this Amendment, the representations and warranties of the Borrower set forth in the Credit Agreement and in the other Loan Documents are true and correct in all material respects on and as of the date hereof, with the same force and effect as if made on and as of such date, except to the extent that such representations and warranties (i) specifically refer to an earlier date, in which case they shall be true and correct in all material respects as of such earlier date (except to the extent of changes in facts or circumstances that have been disclosed to the Lenders and do not constitute an Event of Default or a Potential Default under the Credit Agreement or any other Loan Document), and (ii) are already qualified by materiality, in which case they shall be true and correct in all respects, and except that for purposes of this Section 4.1 , the representations and warranties contained in Section 7.6 of the Credit Agreement shall be deemed to refer to the most recent financial statements furnished pursuant to Section 8.1(a) of the Credit Agreement.
""".lower()
)
result["class"]
['warranties', 'representations']
result = light_pipeline.annotate("""All notices and other communications provided for in this Agreement and the other Loan Documents shall be in writing and may (subject to paragraph (b) below) be telecopied (faxed), mailed by certified mail return receipt requested, or delivered by hand or overnight courier service to the intended recipient at the addresses specified below or at such other address as shall be designated by any party listed below in a notice to the other parties listed below given in accordance with this Section.""".lower())
result["class"]
['notices']
In this section, we will illustrate how to use a different NER annotator that is based on Transformer architecture.
BertForTokenClassification
annotator can load Bert Models with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.
Models from the HuggingFace 🤗 Transformers library are also compatible with Spark NLP 🚀. To see which models are compatible and how to import them see Import Transformers into Spark NLP 🚀.
Using these models is very similar to the NerModel
we used before. We adjust the pipeline by adding the BertFotTokenClassification
step instead of NerModel
, and don't need to add the Embeddings
step as it is already part of the new annotator.
Then, the pipeline is just:
def bert_pipeline(model_name="legner_obligations", language="en"):
documentAssembler = (
nlp.DocumentAssembler().setInputCol("text").setOutputCol("document")
)
tokenizer = nlp.Tokenizer().setInputCols("document").setOutputCol("token")
tokenClassifier = (
legal.BertForTokenClassification.pretrained(
model_name, language, "legal/models"
)
.setInputCols(["token", "document"])
.setOutputCol("label")
.setCaseSensitive(True)
)
ner_converter = (
nlp.NerConverter()
.setInputCols(["document", "token", "label"])
.setOutputCol("ner_chunk")
)
pipeline = nlp.Pipeline(
stages=[documentAssembler, tokenizer, tokenClassifier, ner_converter]
)
empty_data = spark.createDataFrame([[""]]).toDF("text")
model = pipeline.fit(empty_data)
return model
For Legal NLP we currently have Bert models for English, German, Arabic, and (Brazilian) Portuguese available, but we are constantly adding new models with every release. The model we will use here is the legner_obligations
, which identifies obligations (what the different parties commit to do) in agreement documents.
This model extracts the subject (who commits to doing what), the action (the verb - will provide, shall sign…) and the object (what subject will provide, what subject shall sign, etc). Also, if the recipient of the obligation is a third party (a subject will provide to the Company X …), then that third party (Company X) will be extracted as an indirect object.
The model was trained with in-house annotated documents on CUAD dataset.
bert_model = bert_pipeline("legner_obligations", "en")
bert_model.stages
legner_obligations download started this may take some time. [OK!]
[DocumentAssembler_95ad50798e30, REGEX_TOKENIZER_67928cc401e1, BERT_FOR_TOKEN_CLASSIFICATION_d6615a22d5c2, NerConverter_a3663db7abd4]
bert_model.stages[-2].getClasses()
['B-OBLIGATION_ACTION', 'I-OBLIGATION_INDIRECT_OBJECT', 'I-OBLIGATION', 'B-OBLIGATION_INDIRECT_OBJECT', 'PAD', 'I-OBLIGATION_SUBJECT', 'I-OBLIGATION_ACTION', 'O', 'B-OBLIGATION_SUBJECT', 'B-OBLIGATION']
import pyspark.sql.functions as F
text = """The Buyer shall use such materials and supplies only in accordance with the present agreement"""
res = bert_model.transform(spark.createDataFrame([[text]]).toDF("text"))
result_df = res.select(
F.explode(
F.arrays_zip(res.token.result, res.label.result, res.label.metadata)
).alias("cols")
).select(
F.expr("cols['0']").alias("token"),
F.expr("cols['1']").alias("label"),
F.expr("cols['2']['confidence']").alias("confidence"),
)
result_df.show(truncate=100)
+----------+--------------------+----------+ | token| label|confidence| +----------+--------------------+----------+ | The| O|0.71809256| | Buyer|B-OBLIGATION_SUBJECT|0.86514723| | shall| B-OBLIGATION_ACTION|0.99315745| | use| I-OBLIGATION_ACTION| 0.9729679| | such| B-OBLIGATION| 0.7499739| | materials| I-OBLIGATION| 0.9127689| | and| I-OBLIGATION|0.88955635| | supplies| I-OBLIGATION| 0.9182221| | only| I-OBLIGATION|0.82361615| | in| I-OBLIGATION| 0.8662357| |accordance| I-OBLIGATION| 0.9251934| | with| I-OBLIGATION| 0.8835488| | the| I-OBLIGATION|0.53246284| | present| I-OBLIGATION| 0.8670555| | agreement| I-OBLIGATION| 0.8018013| +----------+--------------------+----------+