linux 下

下载解压

https://archive.apache.org/dist/spark/

配置maven

http://spark.apache.org/docs/2.4.6/building-spark.html

spark pom.xml

<repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository>

编译

make-distribution.sh VERSION=2.4.6 SCALA_VERSION=2.12 SPARK_HADOOP_VERSION=2.6.0-cdh.5.16.2 SPARK_HIVE=1 dev/change-scala-version.sh 2.12 ./dev/make-distribution.sh --name 2.6.0-cdh5.16.2 --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.16.2 -Phive -Phive-thriftserver -Pyarn -Dscala.version=2.12.10

window 下

./build/mvn -Pyarn -Phive -Phive-thriftserver -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.16.2 -DskipTests clean package