Apache Spark Installation Steps

Spark Installation

Step 1 : http://spark.incubator.apache.org/downloads.html

Step 2 : Download binaries for Hadoop 2 (HDP2, CDH5)

Step 3 : tar -xvf spark-0.9.0-incubating-bin-hadoop2.tgz

Step 4 : Download SCALA http://www.scala-lang.org/files/archive

Step 5 : configure the environment variable

       export SCALA_HOME=/home/notroot/lab/software/scala-2.10.4-RC3
    export PATH=$PATH:$SCALA_HOME/sbin

Step 6 : Building the SPARK
        Install github:

       sudo apt-get install git-core
     cd /home/notroot/lab/software/spark-0.9.0-incubating-bin-hadoop2
     ./sbt/sbt assembly

Step 7 : Open SPARK Shell
        ./bin/spark-shell
Step 8 : val text="krishna"
      println(text)

Step 9 : If you want for Hadoop YARN Architecture
             Set up environment variable for Hadoop 2.0.5-alpha, change the hadoop version if its for different version
          export SPARK_HADOOP_VERSION=2.0.5-alpha
       export SPARK_YARN=true

Create Assembly
    sbt/sbt clean
  sbt/sbt assembly

Step 10 : Monitoring jobs :    http://localhost:4040/executors/

Starting spark

./bin/spark-shell

One thought on “Apache Spark Installation Steps

Leave a Reply