3 Exploratory Data Analysis¶

3.1 Contents¶

• 3 Exploratory Data Analysis
  • 3.1 Contents
  • 3.2 Introduction
  • 3.3 Imports
  • 3.4 Load The Data
    • 3.4.1 Ski data
    • 3.4.2 State-wide summary data
  • 3.5 Explore The Data
    • 3.5.1 Top States By Order Of Each Of The Summary Statistics
      • 3.5.1.1 Total state area
      • 3.5.1.2 Total state population
      • 3.5.1.3 Resorts per state
      • 3.5.1.4 Total skiable area
      • 3.5.1.5 Total night skiing area
      • 3.5.1.6 Total days open
    • 3.5.2 Resort density
      • 3.5.2.1 Top states by resort density
    • 3.5.3 Visualizing High Dimensional Data
      • 3.5.3.1 Scale the data
        • 3.5.3.1.1 Verifying the scaling
      • 3.5.3.2 Calculate the PCA transformation
      • 3.5.3.3 Average ticket price by state
      • 3.5.3.4 Adding average ticket price to scatter plot
    • 3.5.4 Conclusion On How To Handle State Label
    • 3.5.5 Ski Resort Numeric Data
      • 3.5.5.1 Feature engineering
      • 3.5.5.2 Feature correlation heatmap
      • 3.5.5.3 Scatterplots of numeric features against ticket price
  • 3.6 Summary

3.2 Introduction¶

At this point, we have a firm idea of what our data science problem is and have the data to solve it. The business problem was a general one of modeling resort ticket pricing and, by implication, revenue. We decided, based on data availability, that we would model AdultWeekend prices.

The data we started with contained some ticket price values, but with a number of missing values. This led to several rows being dropped completely. We also had two kinds of ticket prices. There were some obvious issues with some of the other features in the data. For example, one column was missing so many values it was completely dropped. Human errors in our data were corrected. We also obtained and cleaned some additional US state population and size data with which to augment the data set.

3.3 Imports¶

3.4 Load The Data¶

3.4.1 Ski Data¶

Let's review our newly cleaned data set.

We have 278 rows and 25 columns. All typecasting is just as one would expect. But what does the data frame actually look like?

The data appear to be in "tidy" format and ready to use.

3.4.2 State-Wide Summary Data¶

We'll load our state-level summary data as well, as our exploration will rely heavily on state-by-state comparisons.

Aside from the state and resorts_per_state features, our data speak to resort skiing capacity, the amount of time available at each resort for skiers to ski, as well as the resort's context (i.e., the state's population and area in square miles). Below, we take a quick look at our state-level data.

All visible values appear reasonable and the data are in "tidy" format.

3.5 Explore The Data¶

3.5.1 Top States By Order Of Each Summary Statistic¶

What does the state-wide picture look like for our market?

Below, we present data for ski resorts for our state of interest: Montana.

In isolation, it's impossible to draw any definitive, or interesting, conclusions from this data. Much like asset or company valuation, product pricing (our objective) is often done on a relative basis, though alternative approaches (e.g., "cost plus") do exist. The following cells compare Montana ski resorts to those both across the United States and regionally.

3.5.1.1 Total state skiable area¶

The plot on the left tells us Montana ranks third among the top 20 states in our data set by acre. This is hardly surprising to anyone who has seen a map of the U.S.

The data raise a few interesting questions. First, how much of the total skiable area is developed and, by extension, how many skiable acres remain undeveloped? A large quantity of undeveloped skiable area in any state or region would invite competition. This would place downward pressure on prices, as resorts compete for above-market economic profits. Second, per Montana's local market, it is the largest state in its regional market. Should we anticipate a boom in resort construction in the Montana market and, by implication, a growing skiable area -- along with fiercer competition -- for Big Mountain specifically?

Given that our youngest resort is 6 years old, we doubt a sudden boom in resort construction is likely to hit Montana or any other state. Thus, we believe downward pressure on ticket prices from an influx of new competitors is unlikely.

3.5.1.2 Total state population¶

While the United States is far from full, the population is not evenly distributed. California dominates the state population figures despite coming in second behind Alaska in size (by a considerable margin). Montana is the third largest state in our data set by area but is the 30th least populated. Thus, we conjecture Montana's low population density may be indicative of a low indigenous population of skiers. This implies there are fewer skiers in the immediate area, which means local demand may not be enough to support a high ticket price.

3.5.1.3 Resorts per state¶

Here, we examine the ski resort count per state. Surprisingly, overall state size is not dispositive when guessing the distribution of resorts across states. Alaska is larger than California, yet it contains the 10th smallest number of resorts. Montana has the 12th largest resort count.

New York comes out on top in our state-by-state resort count. Recall we saw shades of this during our data wrangling phase. This could be a function of its proximity to wealthy New Yorkers interested in skiing close to home. Then again, it could simply mean that Northern states are better locations for ski resorts. When is the last time you heard anyone mention going on their dream snow-skiing vacation in Alabama?

3.5.1.4 Total skiable area¶

In light of the data considered above, it's impossible to infer how much skiable area is contained in each state. New York must contain a large amount of skiable terrain, as it is home to more than 30 resorts. Still, the only conclusion we can draw from this limited amount of information is that competition in the state of New York for skiers is robust.Below, we look at skiable acreage by state across the entire data set unlike the chart above, which focused on the top 20 states alone.

New York may have the most resorts, but it does not maintain the most skiable acreage. In fact, New York isn't even one of our top five states. As expected, Colorado is a skier’s paradise and comes in first, while Montana is ranked fourth. This is far from a bad showing.

3.5.1.5 Total night skiing area¶

Honestly, we did not know night skiing existed. It sounds about as safe as night surfing in the Pacific, but we digress. The plot below shows us the amount of acreage available for the intrepid night skier.

New York dominates the area of skiing available at night. With one mid-Atlantic exception, the top 5 are Northern states. Is night skiing in and of itself an appeal to customers, or is it a consequence of simply trying to extend the skiing day where days are shorter? Is New York's domination here because it's trying to maximize its appeal to local visitors?

Clearly, our data generates more interesting questions than answers. This is a positive sign! Executive sponsors and data providers may be able to provide additional information. For example, the "number of night tickets," "number of day tickets," and "number of weekly passes" sold.

3.5.1.6 Total days open¶

The cumulative total days open for all resorts in each state yielded an interesting result.

The total days open seems to bear some resemblance to the number of resorts. This is reasonable. The more resorts in a state the greater the chances a resort will be open at any point in the season, thus, the state will have more ski days.

New Hampshire is an interesting case. It made it into the top five, despite being a small state that didn't make it into the top five for resorts per state. This is likely due both to its average annual snowfall of 60" to 150" and its proximity to New York and, oft overlooked, Connecticut. The Granite State is not without competition. Vermont and Maine have comparable snowfall and are two of New Hampshire's neighbors, yet they are not ranked next to one another. In fact, Maine is ranked 16th, just after Montana in our 15th spot.

3.5.2 Resort Density¶

There is a fair amount of heterogeneity in our data set with respect to ski destinations. The U.S. has large states that are not necessarily populous, as well as small states that are densely populated. Some states host far more resorts than others, while other states maintain a larger total-skiing area. States with the most total days skiing per season are not necessarily those with the most resorts. New York is well equipped for night-skiing enthusiasts and has the most resorts.

State size and/or population doesn't seem to tell us much. Relative comparisons like the ratio of resorts serving a given population or area may be fruitful to explore. Such metrics could provide insight into resort density.

Calculating resorts per 100,000 people and 100,000 square miles provides us with a window into resort density relative to each state's population and size.

Our population level data has a pronounced right skew; it cannot go below zero. This lower bound is responsible for the data's abnormal shape. The rightmost bars consist largely of states where few resorts reside and states whose populations dwarf the number available resorts.

Our data concerning resorts per 100k square miles has a similarly pronounced right skew for the same reasons listed above. The rightmost bars consist largely of states where few resorts reside and states that are quite large.

Despite the pronounced skew of both distributions, there is a disernable area of clustering.

3.5.2.1 Top states by resort density¶

Looking at resorts per capita, we see not only which states are top destinations, but also where Montana fits among them.

It's no surprise why New Hampshire, Vermont, Massachusetts, Connecticut, and Rhode Island lead the pack in resorts per 100,000 square miles: All are small states with extremely heavy snowfall.

3.5.3 Visualizing High Dimensional Data¶

We have an overview of the U.S. ski market and state-level statistics that could assist us in our primary task: pricing weekend tickets to ski in Montana. It's highly unlikely that all statistics are of equal value. One way to disentangle this interconnected web of relationships is via principle components analysis (PCA). Applying PCA to our design matrix will help us locate linear combinations of features that are uncorrelated with one another and order these derivative features by the amount of variance they explain. We can then use this information to visualize the data in a lower dimension (e.g., 2 down from 7) and quantify how much variance each component explains. We also have the option of exploring how our original features contribute to PCA-derived features.

The basic steps in this process are:

1. Scale the data (important here because our features are heterogenous).
2. Fit the PCA transformation to our design matrix by "learning" it from the data.
3. Apply the transformation to the data, creating derived features.
4. Optionally, look for patterns in the data using the aforementioned derived features and explore related coefficients.

3.5.3.1 Scale the data¶

We want numeric data here, yet need the state labels. Hence, it's wise to hang onto these and set the state as the index. We also want to capture the column names. This is inevitably a step in almost every data science project.

A cursory glance at the features above reveals our data are on vastly different scales. Standardization will remedy the situation, preventing values large in magnitude from dominating our analysis going forward.

Note, scale() returns an ndarray, so we lose all column names. Because we want to visualise scaled data, we had the foresight to copy feature names to a separate variable. Now we can construct a data frame from the ndarray and reintroduce the column names.

3.5.3.1.1 Verifying the scaling¶

Below, we validate our steps. When there's money on the line for the client, we would be remiss if we skipped this step.

First, we check the mean of the scaled features using Panda's mean() data frame method.

All means are vanishingly small and effectively zero.

Similarly, we check feature standard deviation using Pandas std() data frame method.

One would think the numbers would be closer to 1 than this. The scale() method, however, uses the biased estimator for standard deviation (dof=0). This doesn't mean our approach is deficient. Rather, it indicates scale() calculates sample standard deviation using the number of all observations provided. The std() method, on the other hand, defaults to using dof=1, that is it's normalized by N-1. Restated, the std() function assumes we want the best estimate of the population parameter based on the given sample and uses Bessel's correction to achieve this result. We can tell it to return the biased estimate instead, so as to match the results of scale().

If the reader is concerned about our indifference to Bessel's correction, don't be. We have the entire population we need and, thus, can side-step the Bessel.

Now our standard deviation calculations are consistent with both our scaled values and expectations. This exercise illustrates how different routines that do ostensibly the same thing can exhibit unanticipated behavior. Hence, we validate our work.

3.5.3.2 Calculate the PCA transformation¶

Fit the PCA transformation using the scaled data.¶

Plot the cumulative variance ratio with number of components.

The first two components seem to account for over 75% of the variance, and the first four for over 95%.

Below, we plot the first two derived features (the first two principle components) and label each point with the name of the state. The code extracts and plots the first and second columns from the transformed data (state_pca_x) as x and y coordinates. Recall we retained our state labels to enable the annotation of the plot using the aptly named plt.annotate function. We use the second (i.e., index 1) value of the cumulative variance ratio and include it in our plot's title. This boldly highlights the percentage variance explained by the two PCA components we've plotted. We finish by creating an appropriately sized and well-labelled scatterplot to convey all of this information.

3.5.3.3 Average ticket price by state¶

Here, all point markers for the states are the same size and color. We've visualized relationships between the states based on features such as the total skiable terrain area, but our ultimate interest lies in adult weekend ticket prices. We know ticket prices for resorts in each state, so it might be interesting to see if there's any pattern there.

Western states, followed by a mix of mid-Atlantic and Northeastern and Northwestern states have the highest adult weekend prices. This doesn't bode well for Montana, which is fourth from the bottom of our top 30 states for skiing.

The admittedly less aesthetically pleasing plot below confirms what we see above: We see clusters near 80 (Utah, Colorado, Vermont), as well as a substantial block of resorts that cluster between 40 and 60, which includes Montana.

3.5.3.4 Adding average ticket price to scatter plot¶

There's a lot to unpack here. We extracted indexes and column names from a data frame and the first two principle components from an array. It's becoming a bit hard to keep track of them all. Below we create a new data frame to do this.

That worked, and we have the states to use as an index! Now, let us reformat our average price data.

Here, we concatenate both of the above parts on axis 1 and using the indexes.

There seems to be a wide range of average ticket price histogram above, but it may be hard to pick out differences when using the value for point size. Consequently, we add another column where we seperate these prices into quartiles; that might show something.

We have no missing values and can clearly see that our design matrix makes sense!

We now have enough information to recreate the scatterplot, now adding marker size for ticket price and colour for the discrete quartile. Notice in the code below how we iterate over each quartile and plot the points in the same quartile group as one. This creates an informative legend containing quartiles with points coloured by quartile and sized by ticket price (higher prices are represented by larger point markers).

Below, we can see how many properties fell into each quartile. The lowest and second highest quartiles had 8 and 7 properties, respectively. The two remaining quartiles also had 8 and 7 properties with 8 resorts falling into the highest quartile and 7 resorts falling into the second lowest.

We see the same distribution of states in the dot plot above before, but with additional information about average price. There isn't an obvious pattern. The red points representing the upper quartile of price can be seen to the left, the right, and top. There's also significant spread among other quartiles. In this representation of the ski summaries for each state, which accounts for some 77% of the variance, there is simply no price pattern. The only noteworthy feature is that, with respect to Montana, principal component 1 accounts for slightly more of the variance than principal component 2.

Let's adjust our plot so that it contemplates ticket price, as doing so may help us better understand the spread of the data before us.

Once again, little in the way of a generalized pattern is exhibited by the data. Lower ticket price resorts seem to cluster in the middle of the plot. Variance with respect to California, Michigan, Colorado, and New York seem to dominated by PC1, while PC2 is vital to understanding the variance of New Hampshire and Vermont. A distinctive cluster on the far left includes resorts at different price points and in different geographical regions.

We can examine the contents of the principal components to identify the feature(s) that best explain the observed variance.

It looks like resorts_per_100kcapita and resorts_per_100ksq_mile might count for quite a lot, in a positive sense. Sign matters in this context. A large negative coefficient multiplied by a large negative feature produces a large positive PCA score.

Let's examine our two outliers to see if they are consistent with our above assertion. Both New Hampshire and Vermont seem to have relative high numbers of resorts per capita and resorts per square mile in levels.

Indeed, both states' large value for resorts_per_100ksq_mile in absolute terms puts them more than 3 standard deviations from the mean. But what do the data tell us after they have been scaled?

Vermont stands out on both resorts per 100k square miles and resorts per 100k people, however, New Hampshire has slightly more resorts per capita.

3.5.4 Conclusion On How To Handle State Label¶

Having examined our pricing data, we are now at a major decision point. Do we model at the national or local levels?

3.5.5 Ski Resort Numeric Data¶

The previous EDA gave us some potentially useful features, as well as validating an approach for how to subsequently handle the state labels in our modeling. Let's return to examining data.

3.5.5.1 Feature engineering¶

Having previously explored state-level data, we turn our attention to resort-level data. This will help us to determine, for example, whether or not to use state as a variable. It's now time to merge the two datasets and engineer some features.

We begin by merging the data we have

Having merged state-summary features into the ski resort data, we'll add "state resort competition" features:

• ratio of resort skiable area to total state skiable area
• ratio of resort days open to total state days open
• ratio of resort terrain park count to total state terrain park count
• ratio of resort night skiing area to total state night skiing area

Once we've derived these features, we situate each resort within the context of its local market and drop the raw, state-specific columns used in our feature engineering. Their main purpose was to understand what share of states' skiing "assets" is accounted for by each resort.

3.5.5.2 Feature correlation heatmap¶

We used the heat map below to reveal correlations among variables. It will later help us circumvent multicolinearity when modeling.

There is a lot to unpack here. First, summit and base elevation are highly correlated (light colors equate to high levels of correlation). This isn't a surprise. We can see that we've introduced a lot of multicollinearity with our new ratio features; they are negatively correlated with the number of resorts in each state. The data make sense: If we increase the number of resorts in a state, the share of all other state features will drop for each individual resort. An interesting observation in this region of the heatmap is that there is some positive correlation between the ratio of night skiing area with the number of resorts per capita. Restated, greater resort density with respect to population yields more night skiing is provided.

Turning our attention to our target feature, AdultWeekend ticket price, we see quite a few reasonable correlations. fastQuads stands out, along with Runs and Snow Making_ac. The last one is particularly interesting. Visitors seem to value more "guaranteed" snow, which would drive up resort costs and ticket prices. Of the new features, resort_night_skiing_state_ratio seems the most correlated with ticket price. If this is true, then perhaps seizing a greater share of night skiing capacity means the resort can charge more, as it should.

Runs and total_chairs are also well correlated with ticket price. This is plausible; the more runs you have, the more chairs you'd need to ferry people to them! In fact, they may count for more than SkiableTerrain_ac. While we are on the subject, SkiableTerrain_ac appears less useful than snow making. People seem to put more value in guaranteed snow cover, rather than more terrain.

Our vertical_drop seems to be a selling point that raises ticket prices as well.

3.5.5.3 Scatterplots of numeric features against ticket price¶

Correlation can mask relationships between two variables. Below, we present a series of scatterplots to divine how ticket price varies with other numeric features.

In the scatterplots above, we see multiple high correlations. There's a strong positive correlation with vertical_drop. fastQuads seems very useful. Runs and total_chairs appear quite similar and also useful. resorts_per_100kcapita shows something interesting that we don't see from just a headline correlation figure.

When the value is low, there is quite a variability in ticket price, although it's capable of going quite high. Ticket price may drop a little before then climbing upwards as the number of resorts per capita increases. Ticket price could climb with the number of resorts serving a population because it indicates a popular area for skiing with plenty of demand. The lower ticket price when fewer resorts serve a population may similarly be because it's a less popular state for skiing. The high price for some resorts when resorts are rare (relative to population size), which may indicate areas where a small number of resorts can benefit from a pseudo-monopoly.

Finally, we take a closer look at features that may be useful insofar as they relate to how easily a resort can transport people. We have the number of various chairs and runs, but lack the ratio of chairs to runs. It seems reasonable that this ratio would illuminate how easily, and quickly, people could get to their next ski slope!

At first, these relationships seem counterintuitive: The more chairs a resort has to move people, relative to the number of runs, ticket price rapidly plummets and stays low. What we may be witnessing is an exclusive vs. mass market resort effect. Stated plainly, if a resort doesn't have a surfeit of chairs, it can charge more for its tickets, although with fewer chairs you're inevitably going to be able to serve fewer visitors. The price per visitor is high, but the number of visitors may be low. Something very useful that's missing from the data is the number of visitors per year for all resorts. Big Mountain sees approximately 350k visitors annually.

Lastly, having no fast quads may limit the ticket price, but if a resort covers a wide area, then maintaining a small number of fast quads may increase ticket price.