Notebook

Titanic¶

학습목표 : Titanic의 탑승자 정보를 통해 생존자를 예측하는 모델 만들기

In [1]:

import findspark
findspark.init()
import pyspark
sc = pyspark.SparkContext()

In [2]:

from pyspark.sql.session import SparkSession
spark = SparkSession(sc)
titanic = spark.read.option("header","true").csv("/Users/ryanshin/Downloads/train.csv")

In [3]:

titanic.show()

+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+
|PassengerId|Survived|Pclass|                Name|   Sex| Age|SibSp|Parch|          Ticket|   Fare|Cabin|Embarked|
+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+
|          1|       0|     3|Braund, Mr. Owen ...|  male|  22|    1|    0|       A/5 21171|   7.25| null|       S|
|          2|       1|     1|Cumings, Mrs. Joh...|female|  38|    1|    0|        PC 17599|71.2833|  C85|       C|
|          3|       1|     3|Heikkinen, Miss. ...|female|  26|    0|    0|STON/O2. 3101282|  7.925| null|       S|
|          4|       1|     1|Futrelle, Mrs. Ja...|female|  35|    1|    0|          113803|   53.1| C123|       S|
|          5|       0|     3|Allen, Mr. Willia...|  male|  35|    0|    0|          373450|   8.05| null|       S|
|          6|       0|     3|    Moran, Mr. James|  male|null|    0|    0|          330877| 8.4583| null|       Q|
|          7|       0|     1|McCarthy, Mr. Tim...|  male|  54|    0|    0|           17463|51.8625|  E46|       S|
|          8|       0|     3|Palsson, Master. ...|  male|   2|    3|    1|          349909| 21.075| null|       S|
|          9|       1|     3|Johnson, Mrs. Osc...|female|  27|    0|    2|          347742|11.1333| null|       S|
|         10|       1|     2|Nasser, Mrs. Nich...|female|  14|    1|    0|          237736|30.0708| null|       C|
|         11|       1|     3|Sandstrom, Miss. ...|female|   4|    1|    1|         PP 9549|   16.7|   G6|       S|
|         12|       1|     1|Bonnell, Miss. El...|female|  58|    0|    0|          113783|  26.55| C103|       S|
|         13|       0|     3|Saundercock, Mr. ...|  male|  20|    0|    0|       A/5. 2151|   8.05| null|       S|
|         14|       0|     3|Andersson, Mr. An...|  male|  39|    1|    5|          347082| 31.275| null|       S|
|         15|       0|     3|Vestrom, Miss. Hu...|female|  14|    0|    0|          350406| 7.8542| null|       S|
|         16|       1|     2|Hewlett, Mrs. (Ma...|female|  55|    0|    0|          248706|     16| null|       S|
|         17|       0|     3|Rice, Master. Eugene|  male|   2|    4|    1|          382652| 29.125| null|       Q|
|         18|       1|     2|Williams, Mr. Cha...|  male|null|    0|    0|          244373|     13| null|       S|
|         19|       0|     3|Vander Planke, Mr...|female|  31|    1|    0|          345763|     18| null|       S|
|         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|
+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+
only showing top 20 rows

데이터 타입¶

출처 : https://www.kaggle.com/c/titanic/data
Survived : 살았으면 1, 죽었으면 0
SibSp : 형제자매나 배우자가 몇명 있는지?
Parch : 자식이 몇명인지?

In [4]:

from pyspark.sql.functions import *
titanic.select(count("PassengerId"), sum("Survived"), sum("Survived")/count("PassengerId")).show()

+------------------+-------------+------------------------------------+
|count(PassengerId)|sum(Survived)|(sum(Survived) / count(PassengerId))|
+------------------+-------------+------------------------------------+
|               891|        342.0|                  0.3838383838383838|
+------------------+-------------+------------------------------------+

생존확율¶

생존자수( sum("Survived") ) /전체승객수( count("PassengerId") )

In [5]:

from IPython.display import IFrame
IFrame('https://www.zepl.com/viewer/notebooks/bm90ZTovL1NEUkx1cmtlci9jZTY4ZWU0ZDg2ODM0ZDNmOWExMTA0ZTdiMGJkYzI0ZS9ub3RlLmpzb24', width='100%', height=600)

Out[5]: