Since data linking tasks can take a long time to execute, it is often useful to be able to save the results. For example, this allows model parameters to be applied to new data, or iterations to be re-started from where they left off.
In this demo, we see how we can save a model to a json file and reload it.
It assumes you have already completed the data deduplication quick start.
The following is just boilerplate code that sets up the Spark session and sets some other non-essential configuration options
import logging
from utility_functions.demo_utils import get_spark
logging.basicConfig() # Means logs will print in Jupyter Lab
# Set to DEBUG if you want splink to log the SQL statements it's executing under the hood
logging.getLogger("splink").setLevel(logging.INFO)
spark = get_spark()
df = spark.read.parquet("data/fake_1000.parquet")
settings = {
"link_type": "dedupe_only",
"max_iterations": 5,
"blocking_rules": [
"l.first_name = r.first_name",
"l.surname = r.surname",
"l.dob = r.dob"
],
"comparison_columns": [
{
"col_name": "first_name",
"num_levels": 3,
"term_frequency_adjustments": True
},
{
"col_name": "surname",
"num_levels": 3,
"term_frequency_adjustments": True
},
{
"col_name": "dob"
},
{
"col_name": "city"
},
{
"col_name": "email"
}
]
}
from splink import Splink
linker = Splink(settings, spark, df=df)
df_e = linker.get_scored_comparisons()
df_e.limit(5).toPandas()
INFO:splink.iterate:Iteration 0 complete INFO:splink.params:The maximum change in parameters was 0.5087412834167481 for key π_gamma_surname_prob_dist_non_match_level_2_probability INFO:splink.iterate:Iteration 1 complete INFO:splink.params:The maximum change in parameters was 0.0954439640045166 for key π_gamma_surname_prob_dist_match_level_2_probability INFO:splink.iterate:Iteration 2 complete INFO:splink.params:The maximum change in parameters was 0.021286725997924805 for key π_gamma_dob_prob_dist_non_match_level_0_probability INFO:splink.iterate:Iteration 3 complete INFO:splink.params:The maximum change in parameters was 0.010865330696105957 for key π_gamma_dob_prob_dist_non_match_level_0_probability INFO:splink.iterate:Iteration 4 complete INFO:splink.params:The maximum change in parameters was 0.008596867322921753 for key π_gamma_email_prob_dist_match_level_0_probability
tf_adjusted_match_prob | match_probability | unique_id_l | unique_id_r | first_name_l | first_name_r | gamma_first_name | prob_gamma_first_name_non_match | prob_gamma_first_name_match | first_name_adj | ... | city_l | city_r | gamma_city | prob_gamma_city_non_match | prob_gamma_city_match | email_l | email_r | gamma_email | prob_gamma_email_non_match | prob_gamma_email_match | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.999991 | 0.999646 | 0 | 3 | Julia | Julia | 2 | 0.47229 | 0.567037 | 0.975943 | ... | London | None | -1 | 1.00000 | 1.000000 | hannah88@powers.com | hannah88opowersc@m | 1 | 0.007089 | 0.894782 |
1 | 0.999645 | 0.985811 | 0 | 2 | Julia | Julia | 2 | 0.47229 | 0.567037 | 0.975943 | ... | London | London | 1 | 0.14658 | 0.780896 | hannah88@powers.com | hannah88@powers.com | 1 | 0.007089 | 0.894782 |
2 | 0.999645 | 0.985811 | 0 | 1 | Julia | Julia | 2 | 0.47229 | 0.567037 | 0.975943 | ... | London | London | 1 | 0.14658 | 0.780896 | hannah88@powers.com | hannah88@powers.com | 1 | 0.007089 | 0.894782 |
3 | 0.988914 | 0.916171 | 1 | 3 | Julia | Julia | 2 | 0.47229 | 0.567037 | 0.975943 | ... | London | None | -1 | 1.00000 | 1.000000 | hannah88@powers.com | hannah88opowersc@m | 1 | 0.007089 | 0.894782 |
4 | 0.997900 | 0.983115 | 1 | 2 | Julia | Julia | 2 | 0.47229 | 0.567037 | 0.975943 | ... | London | London | 1 | 0.14658 | 0.780896 | hannah88@powers.com | hannah88@powers.com | 1 | 0.007089 | 0.894782 |
5 rows × 31 columns
We are going to save the model settings, current parameters, and iteration history to a file called saved_model.json
.
linker.save_model_as_json("saved_model.json", overwrite=True)
Reloading the model creates a new Splink object. It populates the settings with the settings saved in the json files, and restores the parameters (the m_probabilities
and u_probabilities
) from the file
from splink import load_from_json
linker_2 = load_from_json("saved_model.json", spark=spark, df=df)
# Perform another set of iterations
df_e_2 = linker_2.get_scored_comparisons()
df_e_2.limit(5).toPandas()
INFO:splink.iterate:Iteration 0 complete INFO:splink.params:The maximum change in parameters was 0.006465733051300049 for key π_gamma_email_prob_dist_match_level_1_probability INFO:splink.iterate:Iteration 1 complete INFO:splink.params:The maximum change in parameters was 0.0047650933265686035 for key π_gamma_email_prob_dist_match_level_0_probability INFO:splink.iterate:Iteration 2 complete INFO:splink.params:The maximum change in parameters was 0.0035470128059387207 for key π_gamma_email_prob_dist_match_level_0_probability INFO:splink.iterate:Iteration 3 complete INFO:splink.params:The maximum change in parameters was 0.0026850104331970215 for key π_gamma_email_prob_dist_match_level_1_probability INFO:splink.iterate:Iteration 4 complete INFO:splink.params:The maximum change in parameters was 0.0020679831504821777 for key π_gamma_email_prob_dist_match_level_1_probability
tf_adjusted_match_prob | match_probability | unique_id_l | unique_id_r | first_name_l | first_name_r | gamma_first_name | prob_gamma_first_name_non_match | prob_gamma_first_name_match | first_name_adj | ... | city_l | city_r | gamma_city | prob_gamma_city_non_match | prob_gamma_city_match | email_l | email_r | gamma_email | prob_gamma_email_non_match | prob_gamma_email_match | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1.000000 | 0.999962 | 0 | 3 | Julia | Julia | 2 | 0.469065 | 0.568189 | 0.995446 | ... | London | None | -1 | 1.000000 | 1.000000 | hannah88@powers.com | hannah88opowersc@m | 1 | 0.001349 | 0.875251 |
1 | 0.999989 | 0.997613 | 0 | 2 | Julia | Julia | 2 | 0.469065 | 0.568189 | 0.995446 | ... | London | London | 1 | 0.140833 | 0.769179 | hannah88@powers.com | hannah88@powers.com | 1 | 0.001349 | 0.875251 |
2 | 0.999989 | 0.997613 | 0 | 1 | Julia | Julia | 2 | 0.469065 | 0.568189 | 0.995446 | ... | London | London | 1 | 0.140833 | 0.769179 | hannah88@powers.com | hannah88@powers.com | 1 | 0.001349 | 0.875251 |
3 | 0.999653 | 0.984606 | 1 | 3 | Julia | Julia | 2 | 0.469065 | 0.568189 | 0.995446 | ... | London | None | -1 | 1.000000 | 1.000000 | hannah88@powers.com | hannah88opowersc@m | 1 | 0.001349 | 0.875251 |
4 | 0.999936 | 0.997146 | 1 | 2 | Julia | Julia | 2 | 0.469065 | 0.568189 | 0.995446 | ... | London | London | 1 | 0.140833 | 0.769179 | hannah88@powers.com | hannah88@powers.com | 1 | 0.001349 | 0.875251 |
5 rows × 31 columns
# We can now see 10 iterations
linker_2.params.all_charts_write_html_file("more_charts.html", overwrite=True)
e.g. to copy and paste into a settings object as starting values
linker_2.params._print_m_u_probs()
gamma_first_name "m_probabilities": [0.36426427960395813, 0.06754633039236069, 0.5681893825531006], "u_probabilities": [0.5297731161117554, 0.0011621455196291208, 0.46906471252441406] gamma_surname "m_probabilities": [0.3780061900615692, 0.05709553509950638, 0.5648982524871826], "u_probabilities": [0.3240525722503662, 1.755945777404122e-05, 0.6759299039840698] gamma_dob "m_probabilities": [0.1348014920949936, 0.8651984930038452], "u_probabilities": [0.9818383455276489, 0.01816166192293167] gamma_city "m_probabilities": [0.23082058131694794, 0.7691794037818909], "u_probabilities": [0.8591668009757996, 0.14083316922187805] gamma_email "m_probabilities": [0.12474856525659561, 0.8752514123916626], "u_probabilities": [0.9986510276794434, 0.0013489817501977086]