Step by Step way to install spark on python in linux via virtual machine :

Hi friends below given steps can be used to install spark on pyhton in linux.

1) Download oracle virtual box from https://www.virtualbox.org/

2) Double click on the virtual box icon. In the new window click on new

3) Give a convenient name of the operating system and select the type as linux

4) Download the .iso file of the ubuntu desktop from https://www.ubuntu.com/download/desktop

5) Go to the setting of your Virtual machine and under storage tab add the .iso file

6) Install ubuntu

Downloading Anaconda 4.2.0 inside Ubuntu:

Inside Ubuntu open firefox > paste the following link >https://www.continuum.io/downloads > download Anaconda 4.0.2 for windows > once download is finished > open the terminal > type following command after $ symbol > sudo bash  “home/joy/download/Anaconda 3-4.2.0-Windows-x86_64.exe >

Now the installation takes place > once installation is done you get a message Anaconda 4.0.2 is successfully installed in command line >  close the terminal and again open it > now to check whether Anaconda 4.0.2 got installed properly > open terminal > type python in command line prompt and press enter

Step 4: Installing Java inside Ubuntu:

Connect to internet > open terminal inside Ubuntu > type $ sudo apt-get update > press enter > this command will update the package index > once it is done successfully > next type $ sudo apt-get install default-jre > Press enter > this command install java in your Ubuntu machine > it will take some time to install > them you will get message that java is successfully installed.

Downloading and installing Spark 2.0.1 inside Ubuntu:

 Connect to internet > Inside Ubuntu open internet browser >  paste the following link >  http://spark.apache.org/downloads.html >  Download spark-2.0.2-bin-hadoop2.7.tgz  > once it is downloaded > unzip the .tgz file  > copy the output folder in your home directory and rename it as spark201 > now open the terminal > type $ gedit .profile > a editor window will open > at end of the editor, after fi  paste below 5 lines of code >

export SPARK_HOME=/home/joy/spark16

export PATH=$SPARK_HOME/bin:$PATH

export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH

export SPARK_LOCAL_IP=LOCALHOST

now paste the same 5 codes in terminal and press enter > now it installs spark > after successful installation > to start spark type the following command in  terminal > $ pyspark > press enter > and to close spark type exit().

Word Count Program in Spark:

Open terminal in Ubuntu and type $ pyspark > and type the following commands in line by line and press enter.

Program:

text = sc.textFile(“hobbit.txt”)
print text
from operator import add
def tokenize(text):
return text.split()
words = text.flatMap(tokenize)
print words
wc = words.map(lambda x: (x,1))
print wc.toDebugString()
counts = wc.reduceByKey(add)
counts.saveAsTextFile(“output-dir”)

Word Count Program in ipython notebook:

Inside Ubuntu open terminal >  type following command which redirect you to ipython notebook in browser.

PYSPARK_PYTHON=python3 PYSPARK_DRIVER_PYTHON=ipython3 PYSPARK_DRIVER_PYTHON_OPTS=”notebook” pyspark

In ipython notebook > click new > python (conda root) > and here paste the above codes one by one and run the cell.