School Budgeting with Machine Learning in Python¶

A Summary of lecture "Case Study- School Budgeting with Machine Learning in Python", via datacamp

• toc: true
• badges: true
• comments: true
• author: Chanseok Kang
• categories: [Python, Datacamp, Machine_Learning]
• image:

Introducing the challenge¶

• Budgets for schools are huge, complex, and not standardize.
  • Hundreds of hours each year are spent manually labelling
• Goal: Build a machine learning algorithm that can automate the process
• Supervised Learning problem

Note: Due to the size of dataset, it is not included in this repository, however, you can download it through kaggle repo

Loading the data¶

Now it's time to check out the dataset! You'll use pandas (which has been pre-imported as pd) to load your data into a DataFrame and then do some Exploratory Data Analysis (EDA) of it.

Some of the column names correspond to features - descriptions of the budget items - such as the Job_Title_Description column. The values in this column tell us if a budget item is for a teacher, custodian, or other employee.

Some columns correspond to the budget item labels you will be trying to predict with your model. For example, the Object_Type column describes whether the budget item is related classroom supplies, salary, travel expenses, etc.

Summarizing the data¶

You'll continue your EDA in this exercise by computing summary statistics for the numeric data in the dataset.

You can use df.info() in the IPython Shell to determine which columns of the data are numeric, specifically type float64. You'll notice that there are two numeric columns, called FTE and Total.

• FTE: Stands for "full-time equivalent". If the budget item is associated to an employee, this number tells us the percentage of full-time that the employee works. A value of 1 means the associated employee works for the school full-time. A value close to 0 means the item is associated to a part-time or contracted employee.
• Total: Stands for the total cost of the expenditure. This number tells us how much the budget item cost.

Looking at the datatypes¶

• ML algorithms work on numbers, not strings
  • Need a numeric representation of these strings
• Strings can be slow compared to numbers
• In pandas, category dtype encodes categorical data numerically,
  • Can speed up code

Exploring datatypes in pandas¶

It's always good to know what datatypes you're working with, especially when the inefficient pandas type object may be involved. Towards that end, let's explore what we have.

Encode the labels as categorical variables¶

Remember, your ultimate goal is to predict the probability that a certain label is attached to a budget line item. You just saw that many columns in your data are the inefficient object type. Does this include the labels you're trying to predict? Let's find out!

There are 9 columns of labels in the dataset. Each of these columns is a category that has many possible values it can take.

You will notice that every label is encoded as an object datatype. Because category datatypes are much more efficient your task is to convert the labels to category types using the .astype() method.

Counting unique labels¶

As Peter mentioned in the video, there are over 100 unique labels. In this exercise, you will explore this fact by counting and plotting the number of unique values for each category of label.

How do we measure success?¶

• Accuracy can be misleading when classes are imbalanced
  • Legitmate email: 99%, Spam: 1%
  • Model that never predicts spam will be 99% accurate!
• Metric used in this problem: log loss
  • Loss function
  • Measure of error
  • Want to minimize the error (unlike accuracy)
• Log loss binary classification

- Actual value: $y: {1=\text{yes}, 0=\text{no}}$
- Prediction (probability that the value is 1): $p$

Computing log loss with NumPy¶

To see how the log loss metric handles the trade-off between accuracy and confidence, we will use some sample data generated with NumPy and compute the log loss using the provided function compute_log_loss(), which Peter showed you in the video.

def compute_log_loss(predicted, actual, eps=1e-14):
    """Compute the logarithmic loss between predicted and
       actual when these are 1D arrays

       :param predicted: The predicted probabilties as floats between 0-1
       :param actual: The actual binary labels. Either 0 or 1
       :param eps (optional): log(0) is inf, so we need to offset our
                               predicted values slightly by eps from 0 or 1.
    """
    predicted = np.clip(predicted, eps, 1-eps)
    loss = -1 * np.mean(actual * np.log(predicted) + (1 - actual) * np.log(1 - predicted))
    return loss

Time to build model¶

• Always a good approach to start with a very simple model
• Gives a sense of how challengeing the problem is
• Many more things can go wrong in complex models
• How much signal can we pull out using basic methods?
• Train basic model on numeric data only
  • Want to go from raw data to predictions quickly
• Multiclass logistic regression
  • Train classifier on each label separately and use those to predict
• Format predictions and save to csv
• Compute log loss score
• Splitting the multi-class dataset
  • Recall: Train-test split
    • Will not work here
    • May end up with labels in test set that never appear in training set
  • Solution: StratifiedShyffleSplit
    • Only works with a single target variable

Setting up a train-test split in scikit-learn¶

Alright, you've been patient and awesome. It's finally time to start training models!

The first step is to split the data into a training set and a test set. Some labels don't occur very often, but we want to make sure that they appear in both the training and the test sets. We provide a function that will make sure at least min_count examples of each label appear in each split: multilabel_train_test_split.

You'll start with a simple model that uses just the numeric columns of your DataFrame when calling multilabel_train_test_split.

Training a model¶

With split data in hand, you're only a few lines away from training a model.

In this exercise, you will import the logistic regression and one versus rest classifiers in order to fit a multi-class logistic regression model to the NUMERIC_COLUMNS of your feature data.

Then you'll test and print the accuracy with the .score() method to see the results of training.

Before you train! Remember, we're ultimately going to be using logloss to score our model, so don't worry too much about the accuracy here. Keep in mind that you're throwing away all of the text data in the dataset - that's by far most of the data! So don't get your hopes up for a killer performance just yet. We're just interested in getting things up and running at the moment.

Ok! The good news is that your workflow didn't cause any errors. The bad news is that your model scored the lowest possible accuracy: 0.0! But hey, you just threw away ALL of the text data in the budget. Later, you won't. Before you add the text data, let's see how the model does when scored by log loss.

Making predictions¶

Use your model to predict values on holdout data¶

You're ready to make some predictions! Remember, the train-test-split you've carried out so far is for model development. The original competition provides an additional test set, for which you'll never actually see the correct labels. This is called the "holdout data."

The point of the holdout data is to provide a fair test for machine learning competitions. If the labels aren't known by anyone but DataCamp, DrivenData, or whoever is hosting the competition, you can be sure that no one submits a mere copy of labels to artificially pump up the performance on their model.

Remember that the original goal is to predict the probability of each label. In this exercise you'll do just that by using the .predict_proba() method on your trained model.

Writing out your results to a csv for submission¶

At last, you're ready to submit some predictions for scoring. In this exercise, you'll write your predictions to a .csv using the .to_csv() method on a pandas DataFrame. Then you'll evaluate your performance according to the LogLoss metric discussed earlier!

You'll need to make sure your submission obeys the correct format.

To do this, you'll use your predictions values to create a new DataFrame, prediction_df.

Interpreting LogLoss & Beating the Benchmark:

When interpreting your log loss score, keep in mind that the score will change based on the number of samples tested. To get a sense of how this very basic model performs, compare your score to the DrivenData benchmark model performance: 2.0455, which merely submitted uniform probabilities for each class.

Remember, the lower the log loss the better. Is your model's log loss lower than 2.0455?

A very brief introduction to NLP¶

• A very brief introduction to NLP
  • Data fpr NLP:
    • Text, documents, speech,...
  • Tokenization
    • Spliting a string into segments
    • Store segments as list
  • Example: "Natural Langauge Processing" -> ["Natural", "Language", "Processing"]
• Bag of words representation
  • Count the number of times a particular token appears
  • "Bag of words"
    • Count the number of times a word was pulled out of the bag
  • This approach discards information about word order
    • "Red, not blue" is the same as "blue, not red"

Representing text numerically¶

• Representing text numerically
  • Bag-of-words
    • Simple way to represent text in machine learning
    • Discards information about grammar and word order
    • Computes frequency of occurance

Creating a bag-of-words in scikit-learn¶

In this exercise, you'll study the effects of tokenizing in different ways by comparing the bag-of-words representations resulting from different token patterns.

You will focus on one feature only, the Position_Extra column, which describes any additional information not captured by the Position_Type label.

For example, in the Shell you can check out the budget item in row 8960 of the data using df.loc[8960]. Looking at the output reveals that this Object_Description is overtime pay. For who? The Position Type is merely "other", but the Position Extra elaborates: "BUS DRIVER". Explore the column further to see more instances. It has a lot of NaN values.

Your task is to turn the raw text in this column into a bag-of-words representation by creating tokens that contain only alphanumeric characters.

For comparison purposes, the first 15 tokens of vec_basic, which splits df.Position_Extra into tokens when it encounters only whitespace characters, have been printed along with the length of the representation.

Combining text columns for tokenization¶

In order to get a bag-of-words representation for all of the text data in our DataFrame, you must first convert the text data in each row of the DataFrame into a single string.

In the previous exercise, this wasn't necessary because you only looked at one column of data, so each row was already just a single string. CountVectorizer expects each row to just be a single string, so in order to use all of the text columns, you'll need a method to turn a list of strings into a single string.

In this exercise, you'll complete the function definition combine_text_columns(). When completed, this function will convert all training text data in your DataFrame to a single string per row that can be passed to the vectorizer object and made into a bag-of-words using the .fit_transform() method.

What's in a token?¶

Now you will use combine_text_columns to convert all training text data in your DataFrame to a single vector that can be passed to the vectorizer object and made into a bag-of-words using the .fit_transform() method.

You'll compare the effect of tokenizing using any non-whitespace characters as a token and using only alphanumeric characters as a token.

Pipelines, feature & text preprocessing¶

• The pipeline workflow
  • Repeatable way to go from raw data to trained model
  • Pipeline object takes sequential list of steps
    • Output of one step is input to next step
  • Each step is a tuple with two elements
    • Name: String
    • Transform: obj implementing .fit() and .transform()
  • Flexible: a step can itself be another pipeline!

Instantiate pipeline¶

In order to make your life easier as you start to work with all of the data in your original DataFrame, df, it's time to turn to one of scikit-learn's most useful objects: the Pipeline.

For the next few exercises, you'll reacquaint yourself with pipelines and train a classifier on some synthetic (sample) data of multiple datatypes before using the same techniques on the main dataset.

• Preprocess

Preprocessing numeric features¶

What would have happened if you had included the with 'with_missing' column in the last exercise? Without imputing missing values, the pipeline would not be happy (try it and see). So, in this exercise you'll improve your pipeline a bit by using the Imputer() imputation transformer from scikit-learn to fill in missing values in your sample data.

By default, the imputer transformer replaces NaNs with the mean value of the column. That's a good enough imputation strategy for the sample data, so you won't need to pass anything extra to the imputer.

After importing the transformer, you will edit the steps list used in the previous exercise by inserting a (name, transform) tuple. Recall that steps are processed sequentially, so make sure the new tuple encoding your preprocessing step is put in the right place.

Text features and feature unions¶

• Preprocessing multiple dtypes
  • Want to use all available features in one pipeline
  • Problem
    • Pipeline steps for numeric and text processing can't follow each other
    • E.g., output of CountVectorizer can`t be input to Imputer
  • Solution
    • FunctionTransformer() & FeatureUnion()
• FunctionTransformer
  • Turns a Python function into an object that a scikit-learn pipeline can understand
  • Need to write two functions for pipeline preprocessing
    • Take entire DataFrame, return numeric columns
    • Take entire DataFrame, return text columns
  • Can then preprocess numeric and text data in separate pipelines

Preprocessing text features¶

Here, you'll perform a similar preprocessing pipeline step, only this time you'll use the text column from the sample data.

To preprocess the text, you'll turn to CountVectorizer() to generate a bag-of-words representation of the data. Using the default arguments, add a (step, transform) tuple to the steps list in your pipeline.

Make sure you select only the text column for splitting your training and test sets.

Multiple types of processing: FunctionTransformer¶

The next two exercises will introduce new topics you'll need to make your pipeline truly excel.

Any step in the pipeline must be an object that implements the fit and transform methods. The FunctionTransformer creates an object with these methods out of any Python function that you pass to it. We'll use it to help select subsets of data in a way that plays nicely with pipelines.

You are working with numeric data that needs imputation, and text data that needs to be converted into a bag-of-words. You'll create functions that separate the text from the numeric variables and see how the .fit() and .transform() methods work.

Multiple types of processing: FeatureUnion¶

Now that you can separate text and numeric data in your pipeline, you're ready to perform separate steps on each by nesting pipelines and using FeatureUnion().

These tools will allow you to streamline all preprocessing steps for your model, even when multiple datatypes are involved. Here, for example, you don't want to impute our text data, and you don't want to create a bag-of-words with our numeric data. Instead, you want to deal with these separately and then join the results together using FeatureUnion().

In the end, you'll still have only two high-level steps in your pipeline: preprocessing and model instantiation. The difference is that the first preprocessing step actually consists of a pipeline for numeric data and a pipeline for text data. The results of those pipelines are joined using FeatureUnion().

Choosing a classification model¶

Using FunctionTransformer on the main dataset¶

In this exercise you're going to use FunctionTransformer on the primary budget data, before instantiating a multiple-datatype pipeline in the next exercise.

Recall from Chapter 2 that you used a custom function combine_text_columns to select and properly format text data for tokenization; it is loaded into the workspace and ready to be put to work in a function transformer!

Add a model to the pipeline¶

You're about to take everything you've learned so far and implement it in a Pipeline that works with the real, DrivenData budget line item data you've been exploring.

Surprise! The structure of the pipeline is exactly the same as earlier in this chapter:

• the preprocessing step uses FeatureUnion to join the results of nested pipelines that each rely on FunctionTransformer to select multiple datatypes
• the model step stores the model object

Try a different class of model¶

Now you're cruising. One of the great strengths of pipelines is how easy they make the process of testing different models.

Until now, you've been using the model step ('clf', OneVsRestClassifier(LogisticRegression())) in your pipeline.

But what if you want to try a different model? Do you need to build an entirely new pipeline? New nests? New FeatureUnions? Nope! You just have a simple one-line change, as you'll see in this exercise.

In particular, you'll swap out the logistic-regression model and replace it with a random forest classifier, which uses the statistics of an ensemble of decision trees to generate predictions.

Can you adjust the model or parameters to improve accuracy?¶

You just saw a substantial improvement in accuracy by swapping out the model. Pipelines are amazing!

Learning from the expert: processing¶

• Text preprocessing
  • NLP tricks for text data
    • Tokenize on punctuation to avoid hyphens, underscores, etc.
    • Includes unigrams and bi-grams in the model to capture important information involving multiple tokens - e.g. "middle school"

Deciding what's a word¶

Before you build up to the winning pipeline, it will be useful to look a little deeper into how the text features will be processed.

In this exercise, you will use CountVectorizer on the training data X_train to see the effect of tokenization on punctuation.

Remember, since CountVectorizer expects a vector, you'll need to use the preloaded function, combine_text_columns before fitting to the training data.

N-gram range in scikit-learn¶

In this exercise you'll insert a CountVectorizer instance into your pipeline for the main dataset, and compute multiple n-gram features to be used in the model.

In order to look for ngram relationships at multiple scales, you will use the ngram_range parameter as Peter discussed in the video.

Special functions: You'll notice a couple of new steps provided in the pipeline in this and many of the remaining exercises. Specifically, the dim_red step following the vectorizer step , and the scale step preceeding the clf (classification) step.

These have been added in order to account for the fact that you're using a reduced-size sample of the full dataset in this course. To make sure the models perform as the expert competition winner intended, we have to apply a dimensionality reduction technique, which is what the dim_red step does, and we have to scale the features to lie between -1 and 1, which is what the scale step does.

The dim_red step uses a scikit-learn function called SelectKBest(), applying something called the chi-squared test to select the K "best" features. The scale step uses a scikit-learn function called MaxAbsScaler() in order to squash the relevant features into the interval -1 to 1.

You won't need to do anything extra with these functions here, just complete the vectorizing pipeline steps below. However, notice how easy it was to add more processing steps to our pipeline!

Learning from the expert: a stats trick¶

• Interaction terms
  • Example
    • English teacher for 2nd grade
    • 2nd grade - budget for English teacher
  • Interaction terms mathematically describe when tokens appear together
  • the math:
  $$ \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \times x_2) $$

Implement interaction modeling in scikit-learn¶

It's time to add interaction features to your model. The PolynomialFeatures object in scikit-learn does just that, but here you're going to use a custom interaction object, SparseInteractions. Interaction terms are a statistical tool that lets your model express what happens if two features appear together in the same row.

SparseInteractions does the same thing as PolynomialFeatures, but it uses sparse matrices to do so. You can get the code for SparseInteractions at this GitHub Gist.

PolynomialFeatures and SparseInteractions both take the argument degree, which tells them what polynomial degree of interactions to compute.

You're going to consider interaction terms of degree=2 in your pipeline. You will insert these steps after the preprocessing steps you've built out so far, but before the classifier steps.

Pipelines with interaction terms take a while to train (since you're making n features into n-squared features!), so as long as you set it up right, we'll do the heavy lifting and tell you what your score is!

Learning from the expert the winning model¶

• The hashing trick
  • Adding new features may cause enormous increase in array size
  • Hashing is a way of increasing memory efficiency
    • Hash function limits possible outputs, fixing array size
• When to use the hashing trick
  • Want to make array of features as small as possible
    • Dimensionality reduction
    • Particularly useful on large datasets
    • E.g., lots of text data!

Why is hashing a useful trick?¶

In the video, Peter explained that a hash function takes an input, in your case a token, and outputs a hash value. For example, the input may be a string and the hash value may be an integer.

By explicitly stating how many possible outputs the hashing function may have, we limit the size of the objects that need to be processed. With these limits known, computation can be made more efficient and we can get results faster, even on large datasets.

Implementing the hashing trick in scikit-learn¶

In this exercise you will check out the scikit-learn implementation of HashingVectorizer before adding it to your pipeline later.

As you saw in the video, HashingVectorizer acts just like CountVectorizer in that it can accept token_pattern and ngram_range parameters. The important difference is that it creates hash values from the text, so that we get all the computational advantages of hashing!

Build the winning model¶

You have arrived! This is where all of your hard work pays off. It's time to build the model that won DrivenData's competition.

You've constructed a robust, powerful pipeline capable of processing training and testing data. Now that you understand the data and know all of the tools you need, you can essentially solve the whole problem in a relatively small number of lines of code. Wow!