GCP에서 Apache Spark Cluster 설치¶
Reference¶
https://zzsza.github.io/data/2018/06/09/apache-spark-cluster/
GCP 가입¶
VM 인스턴스 생성¶
- Compute Engine 클릭
- 만들기 버튼 클릭
인스턴스 만들기¶
- CPU 1개, 메모리 3.75G, Ubuntu 18.04 LTS 선택
스파크 환경 설치¶
SSH 클릭
keygen 등록
~$ ssh-keygen
~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
~$ chmod og-wx ~/.ssh/authorized_keys
- Spark 다운로드
~$ wget http://mirror.navercorp.com/apache/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
- 자바설치
참고주소 : https://tecadmin.net/install-oracle-java-8-ubuntu-via-ppa/
~$ sudo add-apt-repository ppa:webupd8team/java
~$ sudo apt-get update
~$ sudo apt-get install oracle-java8-installer
- Spark 설치 및 실행
- Local : 클러스터 구성없이 Spark 머신 하나로 에뮬레이션 되도록 실행.
~$ tar xvfz spark-2.3.0-bin-hadoop2.7.tgz
~$ cd spark-2.3.0-bin-hadoop2.7/
~/spark-2.3.0-bin-hadoop2.7$ ./bin/spark-shell
- 마스터 주소 확인
~$ cd spark-2.3.0-bin-hadoop2.7/sbin
~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out
~/spark-2.3.0-bin-hadoop2.7/sbin$ grep "spark://" /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out
2018-06-18 14:33:38 INFO Master:54 - Starting Spark master at spark://instance-1.c.arctic-compass-206702.internal:7077
- 예제의 마스터주소 : spark://instance-1.c.arctic-compass-206702.internal:7077
- 스파크 실행
~/spark-2.3.0-bin-hadoop2.7/sbin$ cd ../bin
~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master spark://instance-1.c.arctic-compass-206702.internal:7077
- RDD 생성 및 count
scala> sc.makeRDD(List(1,2,3)).count
- 워커가 없어서 실행 안됨.
방화벽 규칙 만들기¶
방화벽 규칙 클릭
방화벽 규칙 만들기
- 이름: spark-1
- 대상 : 네트워크의 모든 인스턴스
- 소스 IP 범위 : 0.0.0.0/0
- 프로토콜 및 포트 : 모두 허용
만들기 클릭
스냅샷 만들기¶
기존 instance-1을 종료하고 다음 작업을 진행합니다.
Compute Engine -> 스냅샷
스냅샷 만들기 클릭
- 소스 디스크 : instance-1 선택
클러스터에서 Worker 띄우기¶
Slave 컴퓨터에 접속해서 각각 띄우기¶
~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-slave.sh
Usage: ./sbin/start-slave.sh [options] <master>
Master 컴퓨터에서 Worker들을 띄우기¶
기존 instance-1을 시작 버튼을 눌러 켭니다.
마스터 띄우기
~$ cd spark-2.3.0-bin-hadoop2.7/
~/spark-2.3.0-bin-hadoop2.7$ cd sbin
~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out
- 마스터 주소 다시 확인
~/spark-2.3.0-bin-hadoop2.7/bin$ grep "spark://" /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out
2018-06-19 05:15:04 INFO Master:54 - Starting Spark master at spark://instance-1.c.arctic-compass-206702.internal:7077
~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master=spark://instance-1.c.arctic-compass-206702.internal:7077
slaves 등록¶
~$ cd spark-2.3.0-bin-hadoop2.7/conf
~/spark-2.3.0-bin-hadoop2.7/conf$ cp -p slaves.template slaves
~/spark-2.3.0-bin-hadoop2.7/conf$ vi slaves
# slaves 내부 IP 또는 외부 IP 추가.
slaves 실행¶
~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-slaves.sh
Spark Web UI 확인.¶
In [1]:
import requests
r = requests.get("http://localhost:8080")
from IPython.display import HTML
HTML(r.text)
Out[1]:
2.3.0
Spark Master at spark://instance-1.c.arctic-compass-206702.internal:7077
- URL: spark://instance-1.c.arctic-compass-206702.internal:7077
- REST URL: spark://instance-1.c.arctic-compass-206702.internal:6066 (cluster mode)
- Alive Workers: 5
- Cores in use: 5 Total, 5 Used
- Memory in use: 13.0 GB Total, 5.0 GB Used
- Applications: 1 Running, 0 Completed
- Drivers: 0 Running, 0 Completed
- Status: ALIVE
Workers (5)
| Worker Id | Address | State | Cores | Memory |
|---|---|---|---|---|
| worker-20180619051632-10.146.0.2-37773 | 10.146.0.2:37773 | ALIVE | 1 (1 Used) | 2.6 GB (1024.0 MB Used) |
| worker-20180619051632-10.146.0.2-42601 | 10.146.0.2:42601 | ALIVE | 1 (1 Used) | 2.6 GB (1024.0 MB Used) |
| worker-20180619051637-10.146.0.3-44837 | 10.146.0.3:44837 | ALIVE | 1 (1 Used) | 2.6 GB (1024.0 MB Used) |
| worker-20180619051637-10.146.0.5-38961 | 10.146.0.5:38961 | ALIVE | 1 (1 Used) | 2.6 GB (1024.0 MB Used) |
| worker-20180619051639-10.146.0.4-45191 | 10.146.0.4:45191 | ALIVE | 1 (1 Used) | 2.6 GB (1024.0 MB Used) |
Running Applications (1)
| Application ID | Name | Cores | Memory per Executor | Submitted Time | User | State | Duration |
|---|---|---|---|---|---|---|---|
| app-20180619052812-0000 | Spark shell | 5 | 1024.0 MB | 2018/06/19 05:28:12 | sdrlurker2 | RUNNING | 4.4 min |
Completed Applications (0)
| Application ID | Name | Cores | Memory per Executor | Submitted Time | User | State | Duration |
|---|
Spark Web Worker UI 확인.¶
- 8081, 8082, ... 식으로 증가
In [2]:
r = requests.get("http://localhost:8081")
HTML(r.text)
Out[2]:
2.3.0
Spark Worker at 10.146.0.2:42601
- ID: worker-20180619051632-10.146.0.2-42601
- Master URL: spark://instance-1.c.arctic-compass-206702.internal:7077
- Cores: 1 (1 Used)
- Memory: 2.6 GB (1024.0 MB Used)
Running Executors (1)
| ExecutorID | Cores | State | Memory | Job Details | Logs |
|---|---|---|---|---|---|
| 1 | 1 | RUNNING | 1024.0 MB |
|
stdout stderr |
Spark Web Console¶
- 클러스터 모드로 Spark shell 실행
~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master=spark://instance-1.c.arctic-compass-206702.internal:7077
- 다음처럼 Spark Web Console 4040 포트로 확인 가능.
In [3]:
r = requests.get("http://localhost:4040/Jobs")
HTML(r.text)
In [4]:
from IPython.display import IFrame
IFrame('https://www.zepl.com/viewer/notebooks/bm90ZTovL1NEUkx1cmtlci84MjlmMTM4ZDEzZmY0Yjk0YTQ2MDQyNGFjMmZjMTcwYy9ub3RlLmpzb24', width='100%', height=600)
Out[4]: