课件sparktraining2第一章部署概述独立式纱线上火花.pdfVIP

Question: How to scale up or scale down your spark cluster? What is yarn? A ver2.0 resource allocation and task management framework of MR, luding ResourceManager and Nodemanager. How to build up a Spark on Yarn cluster? 1.Set HADOOP_CONF_DIR or YARN_CONF_DIR point to the path which contains the configuration files for your Hadoop cluster. 2.Use yarn-cluster parameter to submit your spark APP ./bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] app jar [app options] What is mesos? Apache Mesos s CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. Spark can run on hardware clusters managed by Apache Mesos. The advantages of deploying Spark with Mesos lude: 1.dynamic partitioning between Spark and other frameworks 2.scalable partitioning between multiple instances of Spark How does Spark work on Mesos? 1. Mesos master re ces the Spark mas s the cluster manager 2. Mesos determines what machines handle what tasks. How to deploy a Spark cluster on Mesos? 1.Install Mesos. 2.Deploy Spark on Mesos Install Spark to same path on all Mesos nodes, and configure spark.meso ecutor.home to your Spark_Home. What is Amazon EC2? An Amazon web service providing computing instances.Refer to Why Amazon EC2? EC2 supports to setup Spark and HDFS clus utomatically via scripts in ./ec2 directory. It also provides security with a key pair which you may need to obtain from YourAccount Security Credentials Access Credentials. How to deploy Spark on Amazon EC2? ./spark-ec2 -k keypair -i key-file -s num-slaves launch cluster-name What is Docker? Docker is an open tform for developers and sysadmins to build, ship, and run distributed applications. How to deploy a