spark-spawn a app via spark-shell VS spark-submit

  yep,u can submit a app to spark ensemble by spark-submit command ,e.g.

spark-submit  --master spark://gzsw-02:7077 --class org.apache.spark.examples.JavaWordCount --verbose --deploy-mode client ~/spark/spark-1.4.1-bin-hadoop2.4/lib/spark-examples-1.4.1-hadoop2.4.0.jar spark/spark-1.4.1-bin-hadoop2.4/RELEASE,spark/spark-1.4.1-bin-hadoop2.4/README.md

 

  but also,u can spawn a command to run a app via spark-shell,

spark-shell --jars lib/spark-examples-1.4.1-hadoop2.4.0-my.jar --master spark://gzsw-02:7077 --executor-memory 600m --total-executor-cores 16 

   note:this param total-executor-cores is a must,else u will get a java.lang.OutOfMemoryError: Java heap space exception after this log

 

15/11/25 12:08:17 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.100.13:56889/user/Executor#-1124709965]) with ID 0

 after than ,i check the master ui,i found that the app's cores shown as the max integer value(2147483647).so u can add this property to set used cores right.

 

  second,if u want to limit the executors to run ,u can specify this property to chieve that

set in spark-defaults.conf

spark.deploy.spreadOut=false

  (but with spark-class command ,this will be a side affect:it maybe cause the same effect with "java heap space" exceptions like above,so comment it is better in this case)

 

  then ,u can import the entry class of your app like this 

import class.to.your.appentry

  followed by invoking the entry method

val arr = ...//params for running this app
JavaWordCount.main(arr)

   but in this demo,the app will issue a new sparkcontext which differs with the default one initiated by spark-shell,so launcher will fail to complain :

WARN SparkContext: Multiple running SparkContexts detected in the same JVM!

  so u can add a property below in the spark-defaults.conf to ignore it

spark.driver.allowMultipleContexts=true

   after all,u will the app is launched same as command spark-submit.

 

ref:

spark-basic demo from book 'learning spark'