Day01.Spark搭建和高可用

文章目录

1.Spark搭建2.高可用配置3.第一个应用的运行

1.Spark搭建

1.下载 Spark 安装包, 下载时候选择对应的 Hadoop 版本 https://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz

# 下载 Spark cd /export/softwares wget https://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz

2.解压

# 解压 Spark 安装包 tar xzvf spark-2.2.0-bin-hadoop2.7.tgz # 移动 Spark 安装包 mv spark-2.2.0-bin-hadoop2.7.tgz /export/servers/spark

3.修改配置文件spark/conf目录下spark-env.sh

#!/usr/bin/env bash # 指定 Java Home export JAVA_HOME=/usr/local/jdk1.8.0_231/ export SCALA_HOME=/usr/local/scala/scala-2.12.11 export HADOOP_HOME=/usr/local/hadoop-2.9.2 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop SPARK_LOCAL_DIRS=/usr/local/spark SPARK_DRIVER_MEMORY=512M # 指定 Spark Master 地址 export SPARK_MASTER_HOST=master export SPARK_MASTER_PORT=7077 export LD_LIBRARY_PATH=$JAVA_LIBRARY_PATH

顺便看下/etc/profile

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native/

4.conf下slaves配置从节点

node-1 node-2

5.配置HistoryServer–spark-defaults.conf #首先修改spark-defaults.conf.template为spark-defaults.conf

cp spark-defaults.conf.template spark-defaults.conf spark.eventLog.enabled true spark.eventLog.dir hdfs://master:8020/spark_log spark.eventLog.compress true

此处端口号和hadoop的core-site.xml文件中配置的保持一致 配置spark-env.sh

# 指定 Spark History 运行参数 export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://master:8020/spark_log"

最后在hdfs上创建目录

hdfs dfs -mkdir -p /spark_log

6.分发到各从节点

cd 到存放spark的目录 scp -r spark root@node-1:$PWD scp -r spark root@node-2:$PWD

7.启动

sbin/start-all.sh sbin/start-history-server.sh

2.高可用配置

1.首先配置Zookeeper zookeeper的安装和配置 Spark+zookeeper搭建高可用集群学习笔记

停止spark集群

sbin/stop-all.sh

spark-env.sh配置

# 指定 Spark Master 地址 #export SPARK_MASTER_HOST=master # 指定 Spark 运行时参数 export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node-1:2181,node-2:2181 -Dspark.deploy.zookeeper.dir=/spark"

再次启动后效果：

3.第一个应用的运行

spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 --executor-memory 1G --total-executor-cores 2 /usr/local/spark/examples/jars/spark-examples_2.11-2.4.5.jar 100 serviceportMaster WebUImaster:8080Worker WebUImaster:8081HIstory Servermaster:4000