%%help
Magic | Example | Explanation |
---|---|---|
info | %%info | Outputs session information for the current Livy endpoint. |
cleanup | %%cleanup -f | Deletes all sessions for the current Livy endpoint, including this notebook's session. The force flag is mandatory. |
delete | %%delete -f -s 0 | Deletes a session by number for the current Livy endpoint. Cannot delete this kernel's session. |
logs | %%logs | Outputs the current session's Livy logs. |
configure | %%configure -f {"executorMemory": "1000M", "executorCores": 4} |
Configure the session creation parameters. The force flag is mandatory if a session has already been
created and the session will be dropped and recreated. Look at Livy's POST /sessions Request Body for a list of valid parameters. Parameters must be passed in as a JSON string. |
sql | %%sql -o tables -q SHOW TABLES |
Executes a SQL query against the sqlContext.
Parameters:
|
local | %%local a = 1 |
All the code in subsequent lines will be executed locally. Code must be valid Python code. |
%%info
%%logs
No logs yet.
sc.parallelize(range(1000)).count()
Creating SparkContext as 'sc'
ID | YARN Application ID | Kind | State | Spark UI | Driver log | Current session? |
---|---|---|---|---|---|---|
4 | None | pyspark | idle | ✔ |
Creating HiveContext as 'sqlContext' SparkContext and HiveContext created. Executing user code ... 1000
import os
print(os.environ.get('SPARK_HOME', None))
print(os.environ.get('HADOOP_CONF_DIR', None))
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/spark /etc/hadoop/conf:/etc/hive/conf:/etc/hive/conf
%%info
ID | YARN Application ID | Kind | State | Spark UI | Driver log | Current session? |
---|---|---|---|---|---|---|
4 | None | pyspark | idle | ✔ |
sc.parallelize(range(1000)).count()
1000
sc.parallelize(range(2000)).count()
2000
%%sql
show tables
%%sql
select * from movies_pq_s3 limit 100
%%sql -o ratings
select movieid, rating from ratings_pq_s3
%%local
%matplotlib inline
import matplotlib
import seaborn as sns
import matplotlib.pyplot as plt
sns.distplot(ratings.rating, kde=False, rug=True)
<matplotlib.axes._subplots.AxesSubplot at 0x115f7d0b8>