Data Science: Analysis of Movies released in the cinema between 2000 and 2017¶

Author/Data Scientist: Nicolas CHEN¶

Introduction¶

The Internet Movie Database (IMDb) is a website that serves as an online database of world cinema. This website contains a large number of public data on films such as the title of the film, the year of release of the film, the genre of the film, the audience, the rating of critics, the duration of the film, the summary of the film, actors, directors and much more. Faced with the large amount of data available on this site, I thought that it would be interesting to analyze the movies data on the IMDb website between the year 2000 and the year 2017.

Objective¶

This notebook aims to analyse the movies data with many various graphics and gives an interpretation of these data.

Python Stack¶

Python stack: Numpy, Matplotlib, Pandas and Seaborn

Preparing the dataset¶

On the IMDb website, it is possible to filter the searches, and thus to display all the movies for one year, such as the year 2017. For example, the first page of all 2017 IMDb movies is available under the following URL:

http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1

So I started to list all the data available on this page, understand their meaning, and especially think of a way that can recover the data on IMDb. After having inventoried the data available on this page and understanding the meaning of each data item, I started the data selection phase, that is, the data I want to keep for my Data Science study.

Here are the data I want to keep:

• Movie title
• Genre of the film
• Duration of the film (in minutes)
• Release year of the film
• Number of public votes
• Public rating (score out of 10)
• Critics rating (score out of 100)
• Movie Gross (millions of dollars)

It remains now to recover these data on all the films between 2000 and 2017. My knowledge of HTML, CSS and Javascript helped me a lot to find a way to recover this data automatically. Like any website, the IMDb site code is HTML, CSS and Javascript. It was therefore necessary to parse this HTML code, and to recover only the concerned data between certain HTML tags and to apply this on several pages and on all the years of the year 2000 to the year 2017.

So I developed a Python script using the BeautifulSoup library, which allows to parse HTML code, I limited the parsing to 8 pages for each year, so starting with the year 2000, my Python script retrieves the data on 8 pages, then redo the same step on the following year until the year 2017. It is a webscraping technique.

In my Python script, I send a GET HTML request to the IMDb site to retrieve the concerned page at regular times. Before launching the Python script, I still looked at the IMDb website with the movie list, and I realized that some data is missing on this IMDb site. For some movies, there is for example, no gross, no votes or no duration of the film. Since there are a lot of movies, it is likely that there are other missing data, so if I had started my Python script, I would have got a dataset with missing values.

I have been thinking of several solutions to fix this dataset problem with missing values as follows:

1. Delete the line with the missing values
2. Fill empty fields with specific values
3. Fill empty fields with calculations

I opted for the first solution, so I updated my Python script, so that it does not take into account the movies whose data is missing during the parsing. Once done, I run my script then I got the dataset between 2000 and 2017. From this dataset, I built a dataframe then converted this dataframe to CSV file named 'dataframeMovies.csv'.

Data analysis¶

I loaded the CSV file 'dataframeMovies.csv' and with the Pandas library, it is possible to have an overview of this dataset and by applying functions like info(), describe() and head(), I checked the contents of my dataset.

Loading the dataset¶

My dataset contains 4583 lines.

I display the first 10 lines.

I delete the unnamed column.

Maximum of some columns¶

Overview of the dataset¶

Divide the dataframe¶

2000-2005¶

2006-2011¶

2012-2017¶

Overview of a sub dataframe for 2012-2017¶

To sum up, here are the data that I use:

• Movie title -> Movie
• Genre of the movie -> Genre
• Movie duration (in minutes) -> timeMin
• Release year of the film -> Year
• Number of public votes -> Vote
• Public rating (score out of 10) -> audienceRating
• Critics rating (score out of 100) -> criticRating
• Movie Gross (in millions of dollars) -> grossMillions

Visualization of data¶

Joinplots¶

Graphical representation of audience ratings based on critics ratings between 2000 and 2017¶

We see that there is a high concentration of points, following a straight line, which means that in most cases, the audience ratings of the movies are in agreement with those of the critics ratings. We also see that for the public, the distribution is stronger between 5/10 and 8/10 and those of the critics between 30/100 and 80/100, which confirms that in most cases, the coherence between the audience ratings and critics ratings.

However, we can see that for some movies, the public is not in agreement with the critics, for example, for some movies, the audience ratings are between 1/10 and 3/10 while the ratings of the critics are between 40/100 and 60/100. We can also see that for other films, the audience ratings (ratings of the public) are between 4/10 and 7/10 while those of the critics are between 20/100 and 50/100.

In this graph, we can conclude that the public often appreciates the movies and generally gives a score above 5/10 while the critics are more severe because the ratings of the critics are often lower than those of the public for any movie.

Joinplot with hexagonal¶

Hexagon representation of audience ratings based on critics ratings between 2000 and 2017¶

On this graph, we can see the linearity of the notes between the audience and the critics.

The most popular movies by the public and critics between 2000 and 2017¶

After searching the dataset, we can determine the most popular movies by the public and the critics. The best movies appreciated by the public between 2000 and 2017 are:

• "The Century of the Self” released in 2002 with a score of 9/10.
• "The Dark Knight: The Black Knight” released in 2008 with a score of 9/10.

The movie most appreciated by the critics is:

• “Boyhood” released in 2014 with a score of 100/100.

Graphical representation of audience ratings by length of film between 2000 and 2017¶

On this graph, we see that most of the movies last between 60 minutes and 120 minutes and collect the most scores and these scores are between 4/10 and 8/10 with a majority of scores above 6/10.

For some films that last more than 3 hours (180 minutes), we notice that the public appreciates them because it generally gives a score above 7/10. In this graph, we see that the longest film lasts 366 minutes, ie 6 hours and 10 minutes and has a score of 8.5/10, and after a search in the dataset, it is about the film “Our best years” released in 2003 which is a drama film.

Graphical representation of the ratings of the critics according to the duration of the film between 2000 and 2017¶

On this graph, we note that for films between 60 minutes and 120 minutes, the ratings of the critics are more concentrated and vary between 10/100 and 98/100.

Graphical representation of the gross of the films according to the notes of the public between 2000 and 2017¶

On this chart, it is clear that the movies that have been well rated by the public are movies that have generated the most millions of dollars, which is logical because if people have enjoyed a movie, they will talk about them, which will encourage other people to go to the cinema to see it, and thus increase the gross of the movie. Audience (public) ratings are more concentrated between 5/10 and 8/10.

Movies that last a long time usually have a high score given by the audience

In the dataset, the movie that brought in the most millions of dollars is the movie “Star Wars: Episode VII – The Force Awakens” with 936.66 million dollars released in 2015.

Graphic representation of the gross of the films according to the scores of the critics between 2000 and 2017¶

In this graph, we note that the ratings of the critics are more concentrated between 30/100 and 80/100, which means that the critics are more demanding towards the films than the public. We also note that the films that have high ratings from critics are those who have brought back a lot of money.

Graphical representation of the number of votes according to the scores of the public between 2000 and 2017¶

Graphic representation of the gross of the films according to the duration of the film between 2000 and 2017¶

On this graph, we notice that the movies between 60 minutes and 150 minutes (2h30) are the ones that bring the most. On the other hand, movies with a very long duration, exceeding 3 hours, yield much less, that is to say, under one million dollars.

We deduce that a director should avoid making a film with a duration at least 3 hours, and that he should limit his movie to duration between 1 and 2:30 so that his audience does not get tired during the projection of the film.

Distribution by audience, critics, duration, gross, votes and year¶

Distplot¶

• Audience Ratings: Most of the audience ratings are between 6/10 and 7/10.
• Critics Ratings: Most critics ratings are between 40/100 and 70/100.
• Duration of the movie: a large number of films have a duration of 100 minutes (1h40).
• Movie Gross: Most movies are worth between 0 and 100 million dollars.
• Number of votes: Most votes are between 0 and 250000 votes.
• Year: Many movies were released in 2000, 2009 and 2017.

Histograms of movies by genre between 2000 and 2017¶

Preparing data for histograms¶

Genre1 2000-2017¶

Genre2 2000-2017¶

Genre3 2000-2017¶

Visualization of histograms of movies by genre between 2000 and 2017¶

Faced with the large amount of data, I divided my dataset into 3 sub dataset by grouping by 6 genres for each dataset because I had 18 genres of films on my whole dataset.

The genres of movies are:

• Action
• Adventure
• Animation
• Biography
• Comedy
• Crime
• Documentary
• Drama
• Family
• Fancy
• Horror
• Music
• Mystery
• Romance
• Sci-Fi
• Thriller
• War
• Western

I thus obtain three graphs of histograms by group of 6 genres.

Histogram of the gross by genre of movie between 2000 and 2017¶