Vacation Lodge Project¶

Analyze the supplied data to determine what factors lead to cancelled hotel bookings. The data includes the following features:

• Booking_ID: unique identifier of each booking
• no_of_adults: Number of adults
• no_of_children: Number of Children
• no_of_weekend_nights: Number of weekend nights (Saturday or Sunday) the guest stayed or booked to stay at the hotel
• no_of_week_nights: Number of week nights (Monday to Friday) the guest stayed or booked to stay at the hotel
• type_of_meal_plan: Type of meal plan booked by the customer:
  • Not Selected – No meal plan selected
  • Meal Plan 1 – Breakfast
  • Meal Plan 2 – Half board (breakfast and one other meal)
  • Meal Plan 3 – Full board (breakfast, lunch, and dinner)
• required_car_parking_space: Does the customer require a car parking space? (0 - No, 1- Yes)
• room_type_reserved: Type of room reserved by the customer. The values are ciphered (encoded) by Vacation Lodge.
• lead_time: Number of days between the date of booking and the arrival date
• arrival_year: Year of arrival date
• arrival_month: Month of arrival date
• arrival_date: Date of the month
• market_segment_type: Market segment designation.
• repeated_guest: Is the customer a repeated guest? (0 - No, 1- Yes)
• no_of_previous_cancellations: Number of previous bookings that were canceled by the customer prior to the current booking
• no_of_previous_bookings_not_canceled: Number of previous bookings not canceled by the customer prior to the current booking
• avg_price_per_room: Average price per day of the reservation; prices of the rooms are dynamic. (in euros)
• no_of_special_requests: Total number of special requests made by the customer (e.g. high floor, view from the room, etc)
• booking_status: Flag indicating if the booking was canceled or not.

Importing necessary libraries and data¶

Data Overview¶

Many of the features, despite being numerical-type columns, appear to be categorical. For example, required car parking space is simply a yes or no encoded with 1 or 0. We will have to convert these columns to categorical data type later.

There does appear to be some missing data in the meal plan column: 'Not Selected' is recorded for some bookings.

All the bookings in our sample were made online. While there are likely other values for this feature, it is interesting that 'Online' is so prevalent as to dominate our sample.

Looking at the columns of our data that contain strings, we find first that Booking_ID has no repeats, as the mode only has frequency of 1. This indicates an integrity of the data: There are no duplicate records. Meal plan 1 is the most common, as is room type 1. The majority of bookings are initiated online, and (thankfully) the majority of bookings are not cancelled.

• The number of adults on a booking ranges from 0 to 4, although it looks like the most common number is 2.
• Between the minimum value and the 75th quartile, the number of children is 0, but the maximum is 10, indicating there may be outliers in the data.
• Number of weekend nights and weekday nights have maxima greater than 2 and 5 respectively, indicating the presence of multi-week bookings in the data set.
• Arrival year seems to be either 2017 or 2018.
• Repeated guest looks like a binary value: yes or no.
• The average price per room ranges from 0€ to 540€, with a median of 99.45€ and a mean of 103.42€. I wonder how there could be a room offered for 0€? We should investigate later.

There are no null values in any of the columns, though there may still be missing data, as was described above.

There are no duplicate rows.

Exploratory Data Analysis (EDA)¶

Let's look at the values in each object column.

These columns all contain categorical data. There are three options for meal plan, with plan 1 (just breakfast) being the most common. There are seven room types, with type 1 being most common and type 3 the least. About twice as many reservations are made online as offline. Similarly, the ratio of reservations that were not cancelled to those that were is roughly 2:1.

Note: Though only the above columns are labelled as objects, we saw in the sample and the data dictionary that some of the numerical columns are categorical as well.

The vast majority of bookings include 2 adults. There are a few that have either 0 or 4 adults, but these are rare (0.4% and 0.04% respectively).

Almost all bookings include 0 children. There are also bookings that include 3, 9, or 10 children, but these are so few that we could consider them outliers.

While many bookings include 1 or 2 weekend nights, most do not! This means Vacation Lodge sees more business on weekdays. There also appears to be some outliers present, namely 7 nights.

Most bookings do not span more than one week. The most common booking is 2 week nights, but there are many with 0 week nights (presumably all weekend bookings). There are scarely few records with more than 10 week nights (two weeks) booked. We will treat these as outliers and remove them from the data.

Meal plan 1 (just breakfast) is the most common type. Very few guests choose full board (plan 3), and many select no meal plan. We will not treat the latter as missing data, however, as many may bring their own food or eat out, thus not requiring any dining accomodations.

Few guests require parking accomodations.

Room type 1 is most common. It is odd that type 3 has almost no records, but we will not remove these records: This would remove an entire class from this categorical variable.

Lead time is the number of days in advance the room is reserved. Many rooms are booking same-day, and the general trend shows a steady decline in frequency as the lead time incleases. Put another way, fewer people reserve further in advance.

There are 102 reservations with a lead time greater than 400 days. With such a wide spread in this data, there are many apparent outliers on the high end. We will keep these extreme values, as they are approximately continuously distributed and do not necessarily represent anomolous behavior.

Around 82% of reservations were for dates in 2018. We only have records for 2017 and 2018.

Fall reservations (August, September, October) are most common, while winter (November, December, January, February) sees a steep decline.

Arrival date fluctuates throughout the month, with reservations in week three (15th-21st) being apparently most common. Note that the 31st has far fewer than the rest: This is simply because not all months have 31 days.

About twice as many reservations are made online as offline. Corporate and complementary reservations are less common, and aviation bookings are rare.

Nearly all of the guests are new. Returning guests make up only a small percentage of bookings. This is an area of growth for Vacation Lodge, as returning customers are generally satisfied with the accomodations and can be counted on for future business.

Thankfully, most guests have never previously cancelled.

However: Note that most guests are not repeats. It would be more insightful to look at the number of bookings cancelled by returning guests.

About 36% of repeated guests have at least one previous cancellation, which means nearly two-thirds don't cancel. Still, there is room for improvement here.

While some returning guests had previously cancelled, only 1.7% of returning guests cancelled their current booking. That's fantastic customer loyalty!

As most records are for first-time guests, there is an over-abundance of 0 in the no_of_previous_bookings_not_canceled column. Also common is 1 and 2 bookings, but the data trails off all the way to 59 kept reservations.

Most of the rooms are priced around 100€, somewhere between about 50€ and 200€. There are many apparent outliers on the high end of the prices, but most look suitable for the feature. All but one of the prices are less than 400€; the remaining record is a room priced at 540€. While the rest of this record does not look erroneous, we will nonetheless excise it, as the price is so far outside the scope of our data.

Rooms priced at 0€ indicate missing data. We will address this in our data processing.

Thankfully, most guests do not have any special requests. There were still over 16,000 special requests made in Vacation Lodge bookings, though thankfully there were scarcely few bookings with more than 3 special requests.

About a third of bookings at Vacation Lodge are cancelled.

We can see again that room type 1 is by far the most common type in our records. Looking at the KDE, type 2 appears to have a slightly lower mean price, while types 4 and 6 skew higher.

Prices vary significantly by market segment. Complementary bookings are understandably the cheapest. Aviation skews higher than Corporate, with the middle 50% of aviation data landing fully above the middle 50% of corporate data. Online prices appear higher, in fact, than offline, the opposite of what I would have expected. The mean prices by market segment are shown above. Complementary rooms may have come with additional charges, like room service, that brough up the mean price. Online bookings have the highest mean price, followed by aviation, though the former sees great influence from extreme values.

With some reservations, booking earlier affords the guest a better deal. Here the data is mixed: Sometimes booking the same day can be affordable.

Note that, while the data is noisy, it forms a triangle in the lower left corner. This aligns with prices being lower with longer lead times. That is to say, while the pattern is not so readily apparent, there is a trend to this data.

Let's start looking at cancellations. Above we see the ratio of cancellations to kept bookings, broken down by the number of special requests. It seems that bookings with 0 special requests are more likely to be cancelled than those with at least one special request.

The plot on the left shows cancellations by month. I would argue there's less to be gleaned here than one may initially think: The number of cancellations seem to be largely a function of the number of reservations made for each month. Recall the graph on the right.

The relationship between arrival month frequency and number of cancellations can be observed in the linear regression below.

The trend is quite plain, in fact: The more reservations there are in a month, the greater number of cancellations observed for that month.

Cancellation history is also an important factor. Let's see which market segments cancel most often.

We see that 0 previous cancellations dominates the data set. As discussed before, this is largely due to most guest being new, i.e., not returning. We can remove the 0s to get a clearer picture of the breakdown for the nonzero records.

Without the 0s, the scale now allows for a clear view of the rest of the data. Corporate bookings appear most prone to cancellations, followed by either complementary or offline, depending on the number of previous cancellations. All told, online bookings are more reliable than I would have expected.

Now let's try to use past trends to inform cancellations. Here we plot number of previous bookings not cancelled versus the booking status for the current reservation. We see that the upper line (Not_Canceled) demonstrates that reliable past guests tend to keep their future reservations. We can remove the 0s again to get a better picture.

It is now more clear that cancelled bookings generally have fewer kept past reservations. Put another way: Past reliability appears to be a good indicator of future booking status.

Data Preprocessing¶

missing value treatment¶

We start by imputing the average price per room to replace the 0€ values. We replaced 0 with the mean price by room type.

We found 78 records that list 0 for both week nights booked and weekend nights book. This is clearly an error! We must drop these records.

outlier treatment¶

See EDA above for drop justifications.

preparing data¶

EDA¶

• It is a good idea to explore the data once again after manipulating it.

The vast majority of bookings include 2 adults. The percentages of each class have changed only slighly.

With outliers removed, we still see that the data is dominated by 0.

The 6 and 7 night outliers have been removed.

Here again, our data is more concentrated now that the outliers (more than 10 nights) have been removed.

Meal plan 1 (just breakfast) is the most common type. Very few guests choose full board (plan 3), and many select no meal plan. There is no perceptable difference after data processing.

Few guests require parking accomodations. There is no real change to the ratio after data processing.

The value counts change only slightly, not enough to affect the ratio of types.

Lead time looks unchanged.

There is no change in the ratio of arrival years after data processing.

Fall reservations (August, September, October) are most common, while winter (November, December, January, February) sees a steep decline. This mirrors the trend observed before data processing.

The trend in arrival date has not changed with data processing.

Market segment trends were not affected by data processing.

Our records continue to reflect that most guests are staying with Vacation Lodge for the first time.

Price data represents the biggest change after processing. The outlier on the high end was removed and all the 0€ records were imputed. With this, the average price looks more like a right-skewed smooth distribution.

There was little change to the ratio of special requests.

The ratio of cancelled to not cancelled is essentially unchanged when compared with the data before processing.

Checking Multicollinearity¶

• In order to make statistical inferences from a logistic regression model, it is important to ensure that there is no multicollinearity present in the data.

The VIF is low (<5) for all numerical variables. The market segment types are the only columns with high multicollinearity. We can try removing the top one.

As the rest of the high VIF scores disappeared, we can conclude that market_segment_type_Online has high multicollinearity with the others. We will actually remove the entire market_segment_type feature. This is for two reasons:

1. Removing only part of a categorical variable disrupts interpretability of the model.
2. After trying to build the model with just market_segment_type_Online removed, the optimizer was still running up against convergence issues. Removing all market_segment_type columns remedied the convergence issue.

Building a Logistic Regression model¶

Even though there are higher p-values, we will drop no_of_previous_bookings_not_canceled. This is because the other columns are parts of categorical variables, and I do not want to remove only part of a categorical feature.

There are no other columns that I am comfortable dropping, as the remaining columns with high p-values are all part of categorical features.

Model performance evaluation¶

Initial Logistic Regression¶

A false negative is more costly for Vacation Lodge than a false positive, since the former anticipates the guest keeping their booking when in fact they cancel. This wastes resources and takes up a room that could have been re-listed. A false positive, on the other hand, is a booking predicted to be cancelled that is not; in this case we still profit from their stay.

Measuring model performance with recall will cut down on false negatives, but we will monitor F1 as well so our model does not become overly imbalanced.

Tuning Logistic Regression¶

We will compare recall scores and F1 scores for multiple thresholds. First we collect the scores in lists.

Then we plot the scores.

The maximum F1 score is identified by the horizontal dashed (green) line. A vertical dashed (purple) line is plotted at the intersection of this tangent and the F1 curve. The associated recall score is marked by a (red) point at the intersection of the vertical dashed line and the recall curve.

Since recall decreases essentially continuously, we can get a robust model by looking at the threshold associated to the max F1 score. This leaves us with high recall and optimal F1, ensuring we are also cutting down on false positives.

Test Performance¶

Accuracy on training data and testing data is nearly the same: about 76%. Precision is similarly close, at around 61%. Recall is around the same on the testing data and the training, with 77% on testing and 76% on training. Lastly F1 scores are comparable: roughly 68% on both data sets.

Note that recall and F1 on the initial model were 57% and 64% respectively on training data, indicating that the alternate threshold choice improved performance, especially on recall.

Final Model Summary¶

The final model performs best with a threshold of 0.32. See below for a table including recall and F1 scores.

The table illustrates that the model performance was improved by adjusting the threshold value. The model is best at reducing false negatives, which are costly to Vacation Lodge.

Building a Decision Tree model¶

We start by redefining our training and testing data. For our logistic regression, we removed several features to improve model performance. Since those same restrictions are not present in our decision tree model, we add those features (like market segment) back in.

From the scores, we can be almost certain that this model suffers from overfitting on the training data. We can check with a simple function.

Indeed, the recall on testing data is nearly 20 points lower than on training data. While 80% recall is still good performance, a model with such a gap between training and testing performance is not sufficiently generalized.

Remember that we will measure model performance based on recall, as false negatives are most costly for Vacation Lodge.

F1 score is similarly lower on the testing data when compared to the training set. We will add this tree's performance to our scores table for reference.

Do we need to prune the tree?¶

We have already detected some overfitting on our tree. Let's visualize it now to determine its complexity.

It is now abundantly clear that this tree is overgrown. We must simplify!

We find that lead_time is the most important feature in this first tree, followed by room price. These two features will largely determine the booking status.

Pre-pruning: Limited depth¶

We grow a new tree with maximum depth of 4, with the goal of getting decent performance out of a much simpler model.

While recall has certainly dropped from the original tree's scores, the new model promises to be more generalized: There is almost no gap between training and testing recall.

Accuracy on the new model is still 83%, not too bad considering how much simpler it is.

This model is easily comprehensible and still produces fairly good results.

While the original tree suffered from overfitting, our new model does not, and it already performs better than the best logistical regression model on F1 score. Let's see if we can push that performance further with a more targeted parameter search. Hopefully we can get a recall score that surpasses the logistical regression too.

Pre-pruning: GridSearchCV¶

After establishing parameters to check in our grid search, we perform the search with a cross validation size of 5.

The above parameters give the best result from the grid search. The best estimator was scored on recall.

To illustrate the process of grid search, we can plot the mean scores of the model over the values of max_depth. Recall had a consistently higher score than F1 across both training and testing data. The best estimator was scored on recall, and it found the optimal depth to be 8. Note that the shaded regions are confidence intervals.

Looking at the confusion matrix for the result of our grid search, we find our model is balanced. Accuracy is 84%, despite scoring the grid search for recall. Precision is about 76%, recall is around 78%, and F1 is roughly 77%.

It is promising that the scores for training data and testing data are so close. This indicates that our model is more generalized.

We record recall and F1 scores in our table for comparison later.

Visualizing our tree now proves more comprehensible. This tree is actually somewhat practical and interpretable. We could adjust the figure dimensions for better readability, but this is just to get an idea of the structure.

As with the logistic regression, lead time is the most important feature. However unlike logistic regression, online bookings and special requirements contribute more than average price per room.

Post-pruning: Cost Complexity¶

We start by defining the cost complexity pruning path for a decision tree with the training data.

Now we grow trees along the pruning path. We collect these trees in a list.

The final entry in this list is the trivial tree, i.e., a tree consisting of just the root node.

We can cut out the last tree, since it's trivial.

Maximum Recall¶

We start by collecting the recall scores for trees along the path.

Plotting ccp_alpha versus recall score, we get a good idea of how performance changes as complexity decreases.

The model with the best recall along the path can be studied above. Both false negative and false positives are low. Accuracy on testing data is nearly 87%, and both recall and F1 are over 80%. While at first glance this looks fantastic, there is a caveat.