Descriptive Visualizations¶

The purpose of this notebook is to visually examine the nominal features, discard the useless ones among them, and create new factor variables.

The "main" plot used in this notebook is Gr Liv Area vs. SalePrice as the overall living area is the most correlated predictor (which is also very intuitive). Many of the nominal variables change the slopes of the regression lines for sub-groups of data points significantly.

"Housekeeping"¶

In [1]:

import json

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn.ensemble import IsolationForest

from utils import (
    ALL_COLUMNS,
    NOMINAL_VARIABLES,
    TARGET_VARIABLES,
    load_clean_data,
    encode_ordinals,
    print_column_list,
)

In [2]:

random_state = np.random.RandomState(42)

In [3]:

pd.set_option("display.max_columns", 120)

In [4]:

sns.set_style("white")

Load the Data¶

In [5]:

df = load_clean_data("data/data_clean_with_transformations.csv")

In [6]:

df.shape

Out[6]:

(2898, 86)

In [7]:

df.head()

Out[7]:

		1st Flr SF	1st Flr SF (box-cox-0)	2nd Flr SF	3Ssn Porch	Alley	Bedroom AbvGr	Bldg Type	Bsmt Cond	Bsmt Exposure	Bsmt Full Bath	Bsmt Half Bath	Bsmt Qual	Bsmt Unf SF	BsmtFin SF 1	BsmtFin SF 2	BsmtFin Type 1	BsmtFin Type 2	Central Air	Condition 1	Condition 2	Electrical	Enclosed Porch	Exter Cond	Exter Qual	Exterior 1st	Exterior 2nd	Fence	Fireplace Qu	Fireplaces	Foundation	Full Bath	Functional	Garage Area	Garage Cars	Garage Cond	Garage Finish	Garage Qual	Garage Type	Gr Liv Area	Gr Liv Area (box-cox-0)	Half Bath	Heating	Heating QC	House Style	Kitchen AbvGr	Kitchen Qual	Land Contour	Land Slope	Lot Area	Lot Area (box-cox-0.1)	Lot Config	Lot Shape	Low Qual Fin SF	MS SubClass	MS Zoning	Mas Vnr Area	Mas Vnr Type	Misc Feature	Misc Val	Mo Sold	Neighborhood	Open Porch SF	Overall Cond	Overall Qual	Paved Drive	Pool Area	Pool QC	Roof Matl	Roof Style	Sale Condition	Sale Type	Screen Porch	Street	TotRms AbvGrd	Total Bath	Total Bsmt SF	Total Porch SF	Total SF	Total SF (box-cox-0.2)	Utilities	Wood Deck SF	Year Built	Year Remod/Add	Yr Sold	SalePrice	SalePrice (box-cox-0)
Order	PID
1	526301100	1656.0	7.412160	0.0	0.0	NA	3	1Fam	Gd	Gd	1	0	TA	441.0	639.0	0.0	BLQ	Unf	Y	Norm	Norm	SBrkr	0.0	TA	TA	BrkFace	Plywood	NA	Gd	2	CBlock	1	Typ	528.0	2	TA	Fin	TA	Attchd	1656.0	7.412160	0	GasA	Fa	1Story	1	TA	Lvl	Gtl	31770.0	18.196923	Corner	IR1	0.0	020	RL	112.0	Stone	NA	0.0	5	Names	62.0	5	6	P	0.0	NA	CompShg	Hip	Normal	WD	0.0	Pave	7	2.0	1080.0	272.0	2736.0	19.344072	AllPub	210.0	1960	1960	2010	215000.0	12.278393
2	526350040	896.0	6.797940	0.0	0.0	NA	2	1Fam	TA	No	0	0	TA	270.0	468.0	144.0	Rec	LwQ	Y	Feedr	Norm	SBrkr	0.0	TA	TA	VinylSd	VinylSd	MnPrv	NA	0	CBlock	1	Typ	730.0	1	TA	Unf	TA	Attchd	896.0	6.797940	0	GasA	TA	1Story	1	TA	Lvl	Gtl	11622.0	15.499290	Inside	Reg	0.0	020	RH	0.0	None	NA	0.0	6	Names	0.0	6	5	Y	0.0	NA	CompShg	Gable	Normal	WD	120.0	Pave	5	1.0	882.0	260.0	1778.0	17.333478	AllPub	140.0	1961	1961	2010	105000.0	11.561716
3	526351010	1329.0	7.192182	0.0	0.0	NA	3	1Fam	TA	No	0	0	TA	406.0	923.0	0.0	ALQ	Unf	Y	Norm	Norm	SBrkr	0.0	TA	TA	Wd Sdng	Wd Sdng	NA	NA	0	CBlock	1	Typ	312.0	1	TA	Unf	TA	Attchd	1329.0	7.192182	1	GasA	TA	1Story	1	Gd	Lvl	Gtl	14267.0	16.027549	Corner	IR1	0.0	020	RL	108.0	BrkFace	Gar2	12500.0	6	Names	36.0	6	6	Y	0.0	NA	CompShg	Hip	Normal	WD	0.0	Pave	6	1.5	1329.0	429.0	2658.0	19.203658	AllPub	393.0	1958	1958	2010	172000.0	12.055250
4	526353030	2110.0	7.654443	0.0	0.0	NA	3	1Fam	TA	No	1	0	TA	1045.0	1065.0	0.0	ALQ	Unf	Y	Norm	Norm	SBrkr	0.0	TA	Gd	BrkFace	BrkFace	NA	TA	2	CBlock	2	Typ	522.0	2	TA	Fin	TA	Attchd	2110.0	7.654443	1	GasA	Ex	1Story	1	Ex	Lvl	Gtl	11160.0	15.396064	Corner	Reg	0.0	020	RL	0.0	None	NA	0.0	4	Names	0.0	5	7	Y	0.0	NA	CompShg	Hip	Normal	WD	0.0	Pave	8	3.5	2110.0	0.0	4220.0	21.548042	AllPub	0.0	1968	1968	2010	244000.0	12.404924
5	527105010	928.0	6.833032	701.0	0.0	NA	3	1Fam	TA	No	0	0	Gd	137.0	791.0	0.0	GLQ	Unf	Y	Norm	Norm	SBrkr	0.0	TA	TA	VinylSd	VinylSd	MnPrv	TA	1	PConc	2	Typ	482.0	2	TA	Fin	TA	Attchd	1629.0	7.395722	1	GasA	Gd	2Story	1	TA	Lvl	Gtl	13830.0	15.946705	Inside	IR1	0.0	060	RL	0.0	None	NA	0.0	3	Gilbert	34.0	5	5	Y	0.0	NA	CompShg	Gable	Normal	WD	0.0	Pave	6	2.5	928.0	246.0	2557.0	19.016856	AllPub	212.0	1997	1998	2010	189900.0	12.154253

Newly created variables are collected in the new_variables list.

In [8]:

new_variables = []

Derived Characteristics¶

Certain characteristics of a house are assumed to have a "binary" influence on the sales price. For example, the existence of a pool could be an important predictor while the exact size of the pool can be deemed not so important.

The below cell creates boolean factor variables out of a set of numeric variables.

In [9]:

derived_variables = {
    "has 2nd Flr": "2nd Flr SF",
    "has Bsmt": "Total Bsmt SF",
    "has Fireplace": "Fireplaces",
    "has Garage": "Garage Area",
    "has Pool": "Pool Area",
    "has Porch": "Total Porch SF",
}
# Factorize numeric columns.
for factor_column, column in derived_variables.items():
    df[factor_column] = df[column].apply(lambda x: 1 if x > 0 else 0)
derived_variables = list(derived_variables.keys())

In [10]:

new_variables.extend(derived_variables)

In [11]:

df[derived_variables].head()

Out[11]:

		has 2nd Flr	has Bsmt	has Fireplace	has Garage	has Pool	has Porch
Order	PID
1	526301100	0	1	1	1	0	1
2	526350040	0	1	0	1	0	1
3	526351010	0	1	0	1	0	1
4	526353030	0	1	1	1	0	0
5	527105010	1	1	1	1	0	1

2nd Floors¶

A second floor may have a positive effect on the sales price. However, having a second floor correlates with overall living space. The individual effect is therefore not as clear as it seems in the plot below.

In [12]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has 2nd Flr", s=15, data=df);

Basements¶

Nearly all houses in Ames, IA, have a basement. Therefore, has Bsmt is most likely not an important predictor.

In [13]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has Bsmt", s=15, data=df);

Fireplaces¶

Bigger houses are more likely to have a fireplace. Thus, the variable has Fireplace might be an interesting predictor.

In [14]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has Fireplace", s=15, data=df);

Garages¶

Holding the overall living area fixed adding a garage seems to affect the price positively. Thus, has Garage seems like an interesting predictor as well.

In [15]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has Garage", s=15, data=df);

Pools¶

Unfortunately, almost no one in Ames, IA, has a pool. The predictor has Pool seems quite uninteresting.

In [16]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has Pool", s=15, data=df);

Porch¶

Most houses have a porch.

In [17]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="has Porch", s=15, data=df);

Neighborhoods¶

The instructors' notes say:

For instructors who cover nominal variables in their class, I would suggest incorporating the neighborhood variable into their models by converting it to a set of dummy (indicator) variables. I have found that the coefficients for the continuous variables tend to have values with more realistic interpretations when used in conjunction with the neighborhood variable.

Indeed, plotting the price distributions by neighborhood reveals significant differences in the price level.

In [18]:

_, ax = plt.subplots(figsize=(10, 8))
sns.boxplot(x="Neighborhood", y="SalePrice", data=df, ax=ax)
ax.set_title("Prices by Neighborhood", fontsize=24)
ax.set_xlabel("Neighborhood", fontsize=18)
ax.tick_params(axis='x', labelrotation = 45)
ax.set_ylabel("House Price", fontsize=18);

The 28 neighborhoods are encoded as factor variables.

In [19]:

neighborhood = pd.get_dummies(df["Neighborhood"], prefix="nhood", dtype=int)
df = pd.concat([df, neighborhood], axis=1)
del df["Neighborhood"]

In [20]:

new_variables.extend(neighborhood.columns)

In [21]:

df[neighborhood.columns].shape

Out[21]:

(2898, 28)

In [22]:

df[neighborhood.columns].head()

Out[22]:

		nhood_Blmngtn	nhood_Blueste	nhood_BrDale	nhood_BrkSide	nhood_ClearCr	nhood_CollgCr	nhood_Crawfor	nhood_Edwards	nhood_Gilbert	nhood_Greens	nhood_GrnHill	nhood_IDOTRR	nhood_Landmrk	nhood_MeadowV	nhood_Mitchel	nhood_Names	nhood_NoRidge	nhood_NPkVill	nhood_NridgHt	nhood_NWAmes	nhood_OldTown	nhood_SWISU	nhood_Sawyer	nhood_SawyerW	nhood_Somerst	nhood_StoneBr	nhood_Timber	nhood_Veenker
Order	PID
1	526301100	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0
2	526350040	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0
3	526351010	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0
4	526353030	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0
5	527105010	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

Nominal Features¶

This section investigates the rest of the nominal variables with regard to which realizations / encoding might be a useful predictor.

In [23]:

print_column_list(set(NOMINAL_VARIABLES) - set(["Neighborhood"]))

Alley           Type of alley access to property
Bldg Type       Type of dwelling
Central Air     Central air conditioning
Condition 1     Proximity to various conditions
Condition 2     Proximity to various conditions (if more than one is present)
Exterior 1st    Exterior covering on house
Exterior 2nd    Exterior covering on house (if more than one material)
Foundation      Type of foundation
Garage Type     Garage location
Heating         Type of heating
House Style     Style of dwelling
Land Contour    Flatness of the property
Lot Config      Lot configuration
MS SubClass     Identifies the type of dwelling involved in the sale.
MS Zoning       Identifies the general zoning classification of the sale.
Mas Vnr Type    Masonry veneer type
Misc Feature    Miscellaneous feature not covered in other categories
Roof Matl       Roof material
Roof Style      Type of roof
Sale Condition  Condition of sale
Sale Type       Type of sale
Street          Type of road access to property

Alleys¶

Almost no house has access to an alley.

In [24]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Alley", s=15, data=df);

In [25]:

del df["Alley"]

Building Type¶

The type of a building clearly affects the valuation. The two types of townhouses as well as the 2-family condo and duplex type are summarized into a single category. This makes sense a) semantically, and b) by looking at the two sub-clusters in the scatter plot.

In [26]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Bldg Type", s=15, data=df);

In [27]:

# Unify the two townhouse types into one.
df["Bldg Type"] = df["Bldg Type"].apply(
    lambda x: "Twnhs" if x in ("TwnhsE", "TwnhsI") else x
)
# Unify the two kinds of 2-family homes.
df["Bldg Type"] = df["Bldg Type"].apply(
    lambda x: "2Fam" if x in ("2FmCon", "Duplx") else x
)

In [28]:

build_type = pd.get_dummies(df["Bldg Type"], prefix="build_type", dtype=int)
df = pd.concat([df, build_type], axis=1)
del df["Bldg Type"]

In [29]:

new_variables.extend(build_type.columns)

In [30]:

df[build_type.columns].head()

Out[30]:

		build_type_1Fam	build_type_2Fam	build_type_Twnhs
Order	PID
1	526301100	1	0	0
2	526350040	1	0	0
3	526351010	1	0	0
4	526353030	1	0	0
5	527105010	1	0	0

Air Conditioning¶

Air conditioning clearly increases the valuation ("steeper" regression line with respect to the overall living area).

In [31]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Central Air", s=15, data=df);

Use a new variable name to cleary show that the variable's dtype is changed from str to int.

In [32]:

df["air_cond"] = df["Central Air"].apply(lambda x: 1 if x == "Y" else 0).astype(int)
del df["Central Air"]

In [33]:

new_variables.append("air_cond")

In [34]:

df[["air_cond"]].head()

Out[34]:

		air_cond
Order	PID
1	526301100	1
2	526350040	1
3	526351010	1
4	526353030	1
5	527105010	1

"Proximity to various Conditions"¶

The columns Condition 1 and Condition 2 have the same realizations and can be regarded as "tags" given to a house indicating the nearby presence of a) a major street, b) a railroad, or c) a park.

The default tag "Norm" (implying no "condition") is given to 86% of the houses (this realization should therefore not be regarded as a tag!).

From the comparison of the grouped scatter plots below, it can be assumed that the proximity of a major street decreases the valuation (lower regression slope through the cloud of blue and orange dots). Therefore, a factor variable major_street is extracted indicating the proximity of an "artery" or "feeder" street.

Further, a factor variable railway is extracted as a relatively high proportion of the houses has such a tag. From the plots, a railway seems to not affect the valuations strongly.

Lastly, a factor variable park is extracted. From the plots, this does not seem to affect the valuation much.

List the "raw" realizations:

In [35]:

(
    (
        df["Condition 1"].value_counts() + df["Condition 2"].value_counts()
    )
    .sort_values(ascending=False)[1:]
)

Out[35]:

Feedr     174
Artery     97
RRAn       48
PosN       43
RRAe       29
PosA       24
RRNn       11
RRNe        6
Name: count, dtype: int64

In [36]:

# Condition 2 is only filled with anything other than "Norm"
# if Condition 1 already has such a tag.
assert not ((df["Condition 1"] == "Norm") & (df["Condition 2"] != "Norm")).any()

In [37]:

# 86% of the houses actually have no tag.
round(100* (df["Condition 1"] == "Norm").sum() / df.shape[0])

Out[37]:

From a simple scatter plot it is hard to see any significant impact by a predictor.

In [38]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Condition 1", s=15, data=df);

However, plotting the groups seperately reveals different slopes.

In [39]:

street = ["Artery", "Feedr"]
railway = ["RRNn", "RRAn", "RRNe", "RRAe"]
park = ["PosA", "PosN"]
plot = sns.lmplot(
    x="Gr Liv Area", y="SalePrice", col="Condition 1", hue="Condition 1",
    col_order=["Norm"] + street + railway + park,
    data=df, robust=True, col_wrap=4, ci=None, truncate=True, scatter_kws={"s": 15},
)
# Adjust font sizes.
for ax in plot.axes:
    ax.set_title(ax.get_title(), fontsize=20)
    ax.set_xlabel(ax.get_xlabel(), fontsize=16)
    ax.set_ylabel(ax.get_ylabel(), fontsize=16)

Extract factor variables major_street, railway, and park.

In [40]:

df["major_street"] = 0
df.loc[
    df["Condition 1"].isin(street) | df["Condition 2"].isin(street),
    "major_street",
] = 1

In [41]:

df["railway"] = 0
df.loc[
    df["Condition 1"].isin(railway) | df["Condition 2"].isin(railway),
    "railway",
] = 1

In [42]:

df["park"] = 0
df.loc[
    df["Condition 1"].isin(park) | df["Condition 2"].isin(park),
    "park",
] = 1

In [43]:

del df["Condition 1"]
del df["Condition 2"]

In [44]:

new_variables.extend(["major_street", "railway", "park"])

Show summary of counts:

In [45]:

df[["major_street", "railway", "park"]].sum()

Out[45]:

major_street    264
railway          94
park             60
dtype: int64

In [46]:

df[["major_street", "railway", "park"]].head()

Out[46]:

		major_street	railway	park
Order	PID
1	526301100	0	0	0
2	526350040	1	0	0
3	526351010	0	0	0
4	526353030	0	0	0
5	527105010	0	0	0

Exterior¶

This dimensions tells the main material with which the houses are made of. The category is too diverse and the various grouped scatter plots did not reveal differing slopes. For simplicity, this variable is dropped.

This variable actually also represents tags that could be associated with a house (possibly up to two different tags).

In [47]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Exterior 1st", s=15, data=df);

In [48]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Exterior 2nd", s=15, data=df);

In [49]:

del df["Exterior 1st"]
del df["Exterior 2nd"]
# Also discard the associated ordinal variables.
del df["Exter Cond"]
del df["Exter Qual"]

Foundation¶

The type of foundation appears to have an effect. However, only three of the six realizations occur in a large number. Factor variables found_BrkTil, found_CBlock, and found_PConc are extracted but not regarded as "interesting".

In [50]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Foundation", s=15, data=df);

In [51]:

df["Foundation"].value_counts()

Out[51]:

Foundation
PConc     1282
CBlock    1242
BrkTil     310
Slab        48
Stone       11
Wood         5
Name: count, dtype: int64

In [52]:

plot = sns.lmplot(
    x="Gr Liv Area", y="SalePrice", col="Foundation", hue="Foundation",
    col_order=["PConc", "CBlock", "BrkTil"],
    data=df, robust=True, col_wrap=4, ci=None, truncate=True, scatter_kws={"s": 15},
)
# Adjust font sizes.
for ax in plot.axes:
    ax.set_title(ax.get_title(), fontsize=20)
    ax.set_xlabel(ax.get_xlabel(), fontsize=16)
    ax.set_ylabel(ax.get_ylabel(), fontsize=16)

In [53]:

foundation = pd.get_dummies(df["Foundation"], prefix="found", dtype=int)
# Only keep the top 3 realizations.
del foundation["found_Slab"]
del foundation["found_Stone"]
del foundation["found_Wood"]
df = pd.concat([df, foundation], axis=1)
del df["Foundation"]

In [54]:

new_variables.extend(["found_BrkTil", "found_CBlock", "found_PConc"])

In [55]:

df[foundation.columns].head()

Out[55]:

		found_BrkTil	found_CBlock	found_PConc
Order	PID
1	526301100	0	1	0
2	526350040	0	1	0
3	526351010	0	1	0
4	526353030	0	1	0
5	527105010	0	0	1

Garage Type¶

As can be expected, the Garage Type looks very similar to the above has Garage variable. Therefore, it is dropped.

In [56]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Garage Type", s=15, data=df);

In [57]:

del df["Garage Type"]

Heating¶

Most of the houses have gas. The variable is not helpful.

In [58]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Heating", s=15, data=df);

In [59]:

del df["Heating"]
# Also discard the associated ordinal variable.
del df["Heating QC"]

House Style¶

In summary, this variable is very similar to the above derived variable has 2nd Flr. Therefore, it is dropped.

In [60]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="House Style", s=15, data=df);

In [61]:

del df["House Style"]

Land Contour¶

This variable is assumed to contain the same information as the ordinal variable Land Slope and is dropped.

In [62]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Land Contour", s=15, data=df);

In [63]:

del df["Land Contour"]

Lot Configuration¶

This variable shows no good pattern and is dropped.

In [64]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Lot Config", s=15, data=df);

In [65]:

del df["Lot Config"]

MS SubClass¶

By looking at this variable's realizations, one can see that several distinct features are lumped together in one. In particular, the above variables has 2nd Flr and *build_type_** and the future age related features at the bottom of this notebook together should comprise the same patterns in a more advantagous way. Thus, the column is dropped.

In [66]:

list(ALL_COLUMNS["MS SubClass"]["lookups"].values())

Out[66]:

['1-STORY 1946 & NEWER ALL STYLES',
 '1-STORY 1945 & OLDER',
 '1-STORY W/FINISHED ATTIC ALL AGES',
 '1-1/2 STORY - UNFINISHED ALL AGES',
 '1-1/2 STORY FINISHED ALL AGES',
 '2-STORY 1946 & NEWER',
 '2-STORY 1945 & OLDER',
 '2-1/2 STORY ALL AGES',
 'SPLIT OR MULTI-LEVEL',
 'SPLIT FOYER',
 'DUPLEX - ALL STYLES AND AGES',
 '1-STORY PUD (Planned Unit Development) - 1946 & NEWER',
 '1-1/2 STORY PUD - ALL AGES',
 '2-STORY PUD - 1946 & NEWER',
 'PUD - MULTILEVEL - INCL SPLIT LEV/FOYER',
 '2 FAMILY CONVERSION - ALL STYLES AND AGES']

In [67]:

del df["MS SubClass"]

MS Zoning¶

This variable is dropped as most houses are located in a "residential" zone.

In [68]:

df["MS Zoning"].value_counts()

Out[68]:

MS Zoning
RL    2252
RM     459
FV     131
RH      27
C       25
A        2
I        2
RP       0
Name: count, dtype: int64

In [69]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="MS Zoning", s=15, data=df);

In [70]:

del df["MS Zoning"]

Masonry Veneer Type¶

None of the groups have a slope differing from the overall one.

In [71]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Mas Vnr Type", s=15, data=df);

In [72]:

del df["Mas Vnr Type"]

Miscellaneous Features¶

This variable is basically a "other" field with no pattern.

In [73]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Misc Feature", s=15, data=df);

In [74]:

del df["Misc Feature"]

Roof¶

Roofs in Ames, IA, are not special enough to make a difference in the price. Even "hip" roofs seem already priced in bigger houses.

In [75]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Roof Matl", s=15, data=df);

In [76]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Roof Style", s=15, data=df);

In [77]:

del df["Roof Matl"]
del df["Roof Style"]

Sale Info¶

Partial and abnormal (= foreclosure) sales seem to make a change with higher and lower prices respectively. These two types will be encoded in factor variables partial_sale and abnormal_sale. The impact seems to be not big though.

In [78]:

df["Sale Condition"].value_counts()

Out[78]:

Sale Condition
Normal     2396
Partial     233
Abnorml     189
Family       46
Alloca       22
AdjLand      12
Name: count, dtype: int64

In [79]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Sale Condition", s=15, data=df);

In [80]:

plot = sns.lmplot(
    x="Gr Liv Area", y="SalePrice", col="Sale Condition", hue="Sale Condition",
    data=df, robust=True, col_wrap=4, ci=None, truncate=True, scatter_kws={"s": 15},
)
# Adjust font sizes.
for ax in plot.axes:
    ax.set_title(ax.get_title(), fontsize=20)
    ax.set_xlabel(ax.get_xlabel(), fontsize=16)
    ax.set_ylabel(ax.get_ylabel(), fontsize=16)

In [81]:

df["partial_sale"] = df["Sale Condition"].apply(lambda x: 1 if x == "Partial" else 0)
df["abnormal_sale"] = df["Sale Condition"].apply(lambda x: 1 if x == "Abnorml" else 0)

Homes that are sold for the first time cleare are priced higher. A factor variable new_home is introduced.

In [82]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="Sale Type", s=15, data=df);

In [83]:

df["new_home"] = df["Sale Type"].apply(lambda x: 1 if x == "New" else 0)

In [84]:

new_variables.extend(["partial_sale", "abnormal_sale", "new_home"])

In [85]:

del df["Sale Condition"]
del df["Sale Type"]

Show summary of counts:

In [86]:

df[["partial_sale", "abnormal_sale", "new_home"]].sum()

Out[86]:

partial_sale     233
abnormal_sale    189
new_home         227
dtype: int64

In [87]:

df[["partial_sale", "abnormal_sale", "new_home"]].head()

Out[87]:

		partial_sale	abnormal_sale	new_home
Order	PID
1	526301100	0	0	0
2	526350040	0	0	0
3	526351010	0	0	0
4	526353030	0	0	0
5	527105010	0	0	0

Street Name¶

Looking at the value counts this variable is pretty useless.

In [88]:

df["Street"].value_counts()

Out[88]:

Street
Pave    2886
Grvl      12
Name: count, dtype: int64

In [89]:

del df["Street"]

Age & Remodeling¶

The dataset was put together over several years. Therefore, the variables with year numbers need to be aligned to indicate the right ages.

In [90]:

# For one house the year of being remodeled is one year
# before it was built. That input error is corrected.
input_error = (df["Year Remod/Add"] < df["Year Built"])
assert input_error.sum() == 1
df.loc[input_error, "Year Remod/Add"] = df.loc[input_error, "Year Built"]

Introduce a factor variable remodeled. Almost half the houses were remodeled at some point in time.

In [91]:

remodeled = (df["Year Remod/Add"] > df["Year Built"])
df["remodeled"] = 0
df.loc[remodeled, "remodeled"] = 1
round(100 * remodeled.sum() / df.shape[0])

Out[91]:

Create discrete variables years_since_built and years_since_remodeled.

In [92]:

df["years_since_built"] = df["Yr Sold"] - df["Year Built"]
df["years_since_remodeled"] = df["Yr Sold"] - df["Year Remod/Add"]

In [93]:

df[["years_since_built", "years_since_remodeled"]].hist(bins=20);

Two factor variables recently_built and recently_remodeled are created indicating that the corresponding action took place in the last 10 years. The two scatter plots below suggest that these groups of "recent vs. old" affect the price.

In [94]:

df["recently_built"] = df["years_since_built"].apply(lambda x: 1 if x <= 10 else 0)
df["recently_remodeled"] = df["years_since_remodeled"].apply(lambda x: 1 if x <= 10 else 0)

In [95]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="recently_built", s=15, data=df);

In [96]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="recently_remodeled", s=15, data=df);

In [97]:

del df["Yr Sold"]
del df["Year Built"]
del df["Year Remod/Add"]

In [98]:

age_columns = [
    "remodeled", "years_since_built", "years_since_remodeled",
    "recently_built", "recently_remodeled",
]
new_variables.extend(age_columns)

In [99]:

df[age_columns].head()

Out[99]:

		remodeled	years_since_built	years_since_remodeled	recently_built	recently_remodeled
Order	PID
1	526301100	0	50	50	0	0
2	526350040	0	49	49	0	0
3	526351010	0	52	52	0	0
4	526353030	0	42	42	0	0
5	527105010	1	13	12	0	0

Outliers¶

The instructors' notes state:

Five observations that an instructor may wish to remove from the data set before giving it to students (a plot of SALE PRICE versus GR LIV AREA will quickly indicate these

points). Three of them are true outliers (Partial Sales that likely don’t represent actual market values) and two of them are simply unusual sales (very large houses priced relatively appropriately). I would recommend removing any houses with more than 4000 square feet from the data set (which eliminates these five unusual observations) before assigning it to students.

To apply a more "rigorous" approach, outlier detection is conducted with a so-called Isolation Forest.

In [100]:

# Use only numeric columns that are strongly correlated with the target.
# This mitigates the risk that a "not so good" chosen factor variable introduced
# in this notebook causes an observation to be removed as an outlier.
with open("data/correlated_variables.json", "r") as file:
    content = json.loads(file.read())
strongly_correlated = content["strongly_correlated"]
df_encoded = encode_ordinals(df[list(set(strongly_correlated) & set(df.columns))])
iso = IsolationForest(n_estimators=100, bootstrap=True, contamination=0.005, random_state=random_state)
outliers = pd.DataFrame(
    iso.fit_predict(df_encoded), columns=["outlier"], index=df.index
)
outliers["outlier"] = outliers["outlier"].apply(lambda x: 1 if x < 0 else 0)
df = pd.concat([df, outliers], axis=1)

The five aforementioned outliers are among the ones detected.

In [101]:

sns.scatterplot(x="Gr Liv Area", y="SalePrice", hue="outlier", s=15, data=df);

We concur with the notes from the paper and remove only the sales of houses with more than 4000 square feet.

In [102]:

# Remove the outliers.
df = df[df["Gr Liv Area"] <= 4000]

Save the Results¶

Save the Data¶

In [103]:

# Re-order the columns for convenience.
final_columns = (
    sorted(set(list(ALL_COLUMNS.keys()) + new_variables) & set(df.columns))
    + TARGET_VARIABLES
)
df = df[final_columns]

Discarding useless and adding new predictors changed the final dataset significantly.

In [104]:

df.shape

Out[104]:

(2893, 109)

In [105]:

df.head()

Out[105]:

		1st Flr SF	1st Flr SF (box-cox-0)	2nd Flr SF	3Ssn Porch	Bedroom AbvGr	Bsmt Cond	Bsmt Exposure	Bsmt Full Bath	Bsmt Half Bath	Bsmt Qual	Bsmt Unf SF	BsmtFin SF 1	BsmtFin SF 2	BsmtFin Type 1	BsmtFin Type 2	Electrical	Enclosed Porch	Fence	Fireplace Qu	Fireplaces	Full Bath	Functional	Garage Area	Garage Cars	Garage Cond	Garage Finish	Garage Qual	Gr Liv Area	Gr Liv Area (box-cox-0)	Half Bath	Kitchen AbvGr	Kitchen Qual	Land Slope	Lot Area	Lot Area (box-cox-0.1)	Lot Shape	Low Qual Fin SF	Mas Vnr Area	Misc Val	Mo Sold	Open Porch SF	Overall Cond	Overall Qual	Paved Drive	Pool Area	Pool QC	Screen Porch	TotRms AbvGrd	Total Bath	Total Bsmt SF	Total Porch SF	Total SF	Total SF (box-cox-0.2)	Utilities	Wood Deck SF	abnormal_sale	air_cond	build_type_1Fam	build_type_2Fam	build_type_Twnhs	found_BrkTil	found_CBlock	found_PConc	has 2nd Flr	has Bsmt	has Fireplace	has Garage	has Pool	has Porch	major_street	new_home	nhood_Blmngtn	nhood_Blueste	nhood_BrDale	nhood_BrkSide	nhood_ClearCr	nhood_CollgCr	nhood_Crawfor	nhood_Edwards	nhood_Gilbert	nhood_Greens	nhood_GrnHill	nhood_IDOTRR	nhood_Landmrk	nhood_MeadowV	nhood_Mitchel	nhood_NPkVill	nhood_NWAmes	nhood_Names	nhood_NoRidge	nhood_NridgHt	nhood_OldTown	nhood_SWISU	nhood_Sawyer	nhood_SawyerW	nhood_Somerst	nhood_StoneBr	nhood_Timber	nhood_Veenker	park	partial_sale	railway	recently_built	recently_remodeled	remodeled	years_since_built	years_since_remodeled	SalePrice	SalePrice (box-cox-0)
Order	PID
1	526301100	1656.0	7.412160	0.0	0.0	3	Gd	Gd	1	0	TA	441.0	639.0	0.0	BLQ	Unf	SBrkr	0.0	NA	Gd	2	1	Typ	528.0	2	TA	Fin	TA	1656.0	7.412160	0	1	TA	Gtl	31770.0	18.196923	IR1	0.0	112.0	0.0	5	62.0	5	6	P	0.0	NA	0.0	7	2.0	1080.0	272.0	2736.0	19.344072	AllPub	210.0	0	1	1	0	0	0	1	0	0	1	1	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	50	50	215000.0	12.278393
2	526350040	896.0	6.797940	0.0	0.0	2	TA	No	0	0	TA	270.0	468.0	144.0	Rec	LwQ	SBrkr	0.0	MnPrv	NA	0	1	Typ	730.0	1	TA	Unf	TA	896.0	6.797940	0	1	TA	Gtl	11622.0	15.499290	Reg	0.0	0.0	0.0	6	0.0	6	5	Y	0.0	NA	120.0	5	1.0	882.0	260.0	1778.0	17.333478	AllPub	140.0	0	1	1	0	0	0	1	0	0	1	0	1	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	49	49	105000.0	11.561716
3	526351010	1329.0	7.192182	0.0	0.0	3	TA	No	0	0	TA	406.0	923.0	0.0	ALQ	Unf	SBrkr	0.0	NA	NA	0	1	Typ	312.0	1	TA	Unf	TA	1329.0	7.192182	1	1	Gd	Gtl	14267.0	16.027549	IR1	0.0	108.0	12500.0	6	36.0	6	6	Y	0.0	NA	0.0	6	1.5	1329.0	429.0	2658.0	19.203658	AllPub	393.0	0	1	1	0	0	0	1	0	0	1	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	52	52	172000.0	12.055250
4	526353030	2110.0	7.654443	0.0	0.0	3	TA	No	1	0	TA	1045.0	1065.0	0.0	ALQ	Unf	SBrkr	0.0	NA	TA	2	2	Typ	522.0	2	TA	Fin	TA	2110.0	7.654443	1	1	Ex	Gtl	11160.0	15.396064	Reg	0.0	0.0	0.0	4	0.0	5	7	Y	0.0	NA	0.0	8	3.5	2110.0	0.0	4220.0	21.548042	AllPub	0.0	0	1	1	0	0	0	1	0	0	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	42	42	244000.0	12.404924
5	527105010	928.0	6.833032	701.0	0.0	3	TA	No	0	0	Gd	137.0	791.0	0.0	GLQ	Unf	SBrkr	0.0	MnPrv	TA	1	2	Typ	482.0	2	TA	Fin	TA	1629.0	7.395722	1	1	TA	Gtl	13830.0	15.946705	IR1	0.0	0.0	0.0	3	34.0	5	5	Y	0.0	NA	0.0	6	2.5	928.0	246.0	2557.0	19.016856	AllPub	212.0	0	1	1	0	0	0	0	1	1	1	1	1	0	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	13	12	189900.0	12.154253

In [106]:

df.to_csv("data/data_clean_with_transformations_and_factors.csv")