PII analysis with Azure Open AI¶

In [1]:

import datetime
import openai
import os
import sys

from dotenv import load_dotenv

In [2]:

load_dotenv("azure.env")

# Azure Open AI
openai.api_type: str = "azure"
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_version = os.getenv("OPENAI_API_VERSION")

print("Open AI version:", openai.__version__)

Open AI version: 0.28.1

In [3]:

print("Today is:", datetime.datetime.today().strftime("%d-%b-%Y %H:%M:%S"))

Today is: 12-Oct-2023 14:43:37

In [4]:

sys.version

Out[4]:

'3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0]'

Function¶

In [5]:

model = "text-davinci-003"

In [6]:

def azure_openai(prompt, temperature=0.8):
    """
    Get Azure Open AI results
    """
    prompt = prompt + "\n" + text

    results = openai.Completion.create(
        engine=model,
        prompt=prompt,
        temperature=temperature,
        max_tokens=800,
    )

    answer = results["choices"][0]["text"].strip("\n")

    return answer

PII analysis¶

In [7]:

text = """
Parker Doe has repaid all of their loans as of 2020-04-25.
Their SSN is 859-98-0987. To contact them, use their phone number
555-555-5555. They are originally from Brazil and have Brazilian CPF number 998.214.865-68
"""

print(text)

Parker Doe has repaid all of their loans as of 2020-04-25.
Their SSN is 859-98-0987. To contact them, use their phone number
555-555-5555. They are originally from Brazil and have Brazilian CPF number 998.214.865-68

In [8]:

answer = azure_openai("What are the Personally identifiable information in this text?")
print(answer)

Personally identifiable information in this text:
1. Parker Doe 
2. Social Security Number (SSN): 859-98-0987
3. Phone Number: 555-555-5555
4. Country of Origin: Brazil
5. CPF Number: 998.214.865-68

In [9]:

answer = azure_openai(
    "What are the Personally identifiable information in this text? Save in a json file"
)
json = answer
print(json)

{
  "PII": {
    "Name": "Parker Doe",
    "SSN": "859-98-0987",
    "Phone Number": "555-555-5555",
    "Country of Origin": "Brazil",
    "CPF Number": "998.214.865-68"
  }
}

In [ ]:

In [10]:

text = """
RECORD #333582770390100 | MH | 85986313 | | 054351 | 2/14/2001 12:00:00 AM | \
CORONARY ARTERY DISEASE | Signed | DIS | Admission Date: 5/22/2001 \
eport Status: Signed Discharge Date: 4/24/2001 ADMISSION DIAGNOSIS: \
CORONARY ARTERY DISEASE. HISTORY OF PRESENT ILLNESS: \
The patient is a 54-year-old gentleman with a history of progressive angina over the past several months. \
The patient had a cardiac catheterization in July of this year revealing total occlusion of the RCA and \
50% left main disease , with a strong family history of coronary artery disease with a brother dying at \
the age of 52 from a myocardial infarction and another brother who is status post coronary artery bypass grafting. \
The patient had a stress echocardiogram done on July , 2001 , which showed no wall motion abnormalities ,\
but this was a difficult study due to body habitus. The patient went for six minutes with minimal ST depressions \
in the anterior lateral leads , thought due to fatigue and wrist pain , his anginal equivalent. Due to the patient's \
increased symptoms and family history and history left main disease with total occasional of his RCA was referred \
for revascularization with open heart surgery."
"""
print(text)

RECORD #333582770390100 | MH | 85986313 | | 054351 | 2/14/2001 12:00:00 AM | CORONARY ARTERY DISEASE | Signed | DIS | Admission Date: 5/22/2001 eport Status: Signed Discharge Date: 4/24/2001 ADMISSION DIAGNOSIS: CORONARY ARTERY DISEASE. HISTORY OF PRESENT ILLNESS: The patient is a 54-year-old gentleman with a history of progressive angina over the past several months. The patient had a cardiac catheterization in July of this year revealing total occlusion of the RCA and 50% left main disease , with a strong family history of coronary artery disease with a brother dying at the age of 52 from a myocardial infarction and another brother who is status post coronary artery bypass grafting. The patient had a stress echocardiogram done on July , 2001 , which showed no wall motion abnormalities ,but this was a difficult study due to body habitus. The patient went for six minutes with minimal ST depressions in the anterior lateral leads , thought due to fatigue and wrist pain , his anginal equivalent. Due to the patient's increased symptoms and family history and history left main disease with total occasional of his RCA was referred for revascularization with open heart surgery."

In [11]:

answer = azure_openai(
    "What are the Personally identifiable information in this text? Extract this into a json file"
)
print(answer)

{
"Personally Identifiable Information": [
"Name: MH",
"Age: 54",
"Date of birth: 2/14/2001",
"Diagnosis: Coronary Artery Disease",
"Admission Date: 5/22/2001",
"Discharge Date: 4/24/2001",
"Family History: Brother died at age 52 from a myocardial infarction and another brother who is status post coronary artery bypass grafting"
]
}

In [12]:

answer = azure_openai(
    "What are the Personally identifiable information in this text? Extract only the main events with their date"
)
print(answer)

Personally Identifiable Information: Patient’s name, age (54), gender (male), family history (brothers died at age 52 from myocardial infarction; brother who is status post coronary artery bypass grafting).

Main Events: 
July 2001: Cardiac catheterization reveals total occlusion of RCA and 50% left main disease; Stress echocardiogram done 
May 22, 2001: Admission date 
April 24, 2001: Discharge date

In [ ]: