Choosing and setting up a Scala IDE for Spark on Windows

In choosing an IDE: Eclipse or IntelliJ?  I found more useful sites more quickly for Eclipse, and have more experience with Eclipse, so let's go that route for now.

To start, I'm following along the itversity "Develop Spark apps with Scala IDE" -- the instructions are for Linux, but were easy to translate to Windows -- with the following notes:

• Prereqs
• Set up Scala & Spark yesterday.
• Installed Scala IDE for Eclipse
• Installed sbt 
• Installed sbteclipse plugin (note to self; on Windows the home directory is the \Users\<username> directory)
• In the section "Create Spark Scala project using sbt"
• I created a project folder C:\scala-projects\simple-scala-spark with 
• The step to run sbt package creates project and target folders and downloads the dependencies defined in build.sbt
• The step to run sbt eclipse creates the project definitions needed to bring the project into eclipse
• (note to self: These would need to be run for every new project)
• In the section "Import into Scala IDE for Eclipse"
• For the first step, from the Eclipse menus, choose File > Import..., then General > Existing Projects into Workspace.  I set the search directory to C:\scala-projects and it found "simple-spark" just fine.
• When creating the new Scala object, it will direct you to the src\main\scala folder as the Source Folder.
• In the section "Run the Spark Application using Scala IDE"
• You need to select Scala Application > New configuration in the Run Configurations window.
• Type SimpleApp as the main class
• In the Program Arguments, I used local c:\temp\simpleappoutput.
• (note to self: do not create the \simpleappoutput directory in advance, otherwise SimpleApp chokes)

SUCCESS!  It runs locally.  I also now have access to an AWS account, so tomorrow I'll try to stand up an instance and do the "Run Spark Scala application on Cluster" section.

Setting up Spark Standalone on Windows

Doing this today with an eye toward Spark Scala development.  

Even though I'm running Windows 10, I followed the directions at http://nishutayaltech.blogspot.in/2015/04/how-to-run-apache-spark-on-windows7-in.html with success, with the following notes:

• I used the latest versions of Scala and Spark, and I already had Java 8 and Python 3.4 installed.
• I used a Spark prebuilt package for Hadoop; initially I was confused about the language surrounding what to do with the environment variables, but realized this meant:
• Create a new environment variable SPARK_HOME and set it to where I unzipped Spark (in this case, C:\spark-2.0.1-bin-hadoop2.7)
• Edit the PATH environment variable to add %SPARK_HOME%\bin 
• Since I have Spark prebuilt for Hadoop 2.7, I downloaded the winutils.exe for hadoop-2.7.1 at https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
• I set log4j.properties to only show WARN and above, per http://stackoverflow.com/questions/28189408/how-to-reduce-the-verbosity-of-sparks-runtime-output

And, success! running spark-shell, opening the Spark UI, and running the SparkPi example.

Up next: choosing and setting up a Scala IDE