spark使用文档.docxVIP

1.spark的安装和部署运行模式本地模式Standalone模式Mesoes模式yarn模式1.1. 本地模式安装jdk 安装scalatar -zxf scala-2.11.6.tgzspark版本只到 1.6.1，对应scala版本为 2.10.x配置文件?/etc/profileexport?JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-5.x86_64/jreexport?SCALA_HOME=/home/yy/scalaexport?SPARK_HOME=/home/yy/spark-1.2.0-bin-hadoop2.4export?PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbinconf/spark-env.sh配置-- 调试启动方式sh -x ./sbin/start-master.shSPARK_MASTER_IP=12SPARK_MASTER_PORT=7077SPARK_WORKER_CORES=3export MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}export SPARK_MASTER_IP=12export SPARK_MASTER_PORT=7077export SPARK_WORKER_CORES=1export SPARK_WORKER_INSTANCES=1export SPARK_WORKER_MEMORY=1Gexport SPARK_WORKER_PORT=8888export MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}修改 conf/slaves.template 为conf/slaves启动master/opt/soft/spark/sbin/start-master.sh本地模式运行例子/opt/soft/spark/bin/spark-submit --master local[*] --class org.apache.spark.examples.SparkPi /opt/soft/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hadoop 查看spark上传的文件hdfs dfs -ls /user/appuser/.sparkStagingspark shell任务提交spark-submit --class com.zxq.iov.cloud.spark.job.TBoxStreamJob --master yarn-cluster --num-executors 5 --conf spark.jars.excludes=org.apache.kafka:kafka_2.10 --jars hdfs://appcluster/appjars/spark-hbase-1.0.0-SNAPSHOT.jar,hdfs://appcluster/appjars/spark-kafka-1.0.0-SNAPSHOT.jar hdfs://appcluster/appjars/spark-tbox-1.0.0-SNAPSHOT.jar?--num-executorsspark-submit --class com.zxq.iov.cloud.spark.job.TBoxStreamJob --master yarn-cluster --num-executors 9 --executor-memory 2G --conf spark.streaming.kafka.maxRatePerPartition=100 --files hdfs://appcluster/appconf/perties hdfs://appcluster/appjars/spark-tbox-1.0.0-SNAPSHOT.jar部署中遇到的bugbug1启动master时报错failed to launch org.apache.spark.deploy.master.Master: at .InetAddress.getLocalHost(InetAddress.java:1469)windows下直接执行spark-shell报错Could not locate executable null\bin\winutils.exe初步怀疑是hadoop没有安装导致部署：Git库：Cloude/spark?打的jar包就在target根目录下?目前需要打jar包的模块有sp-