Installing and setting up the Routing Jar for a Spark Job - Spectrum_Routing_for_Big_Data - 5.1

Spectrum Routing Installation: Hortonworks
Product type
Software
Portfolio
Locate
Product family
Spectrum
Product
Spatial Big Data > Routing for Big Data
Version
5.1
Language
English
Product name
Spectrum Routing for Big Data
Title
Spectrum Routing Installation: Hortonworks
Copyright
2024
First publish date
2017
Last updated
2024-10-18
Published on
2024-10-18T09:54:47.614515

Perform the following steps:

  1. Deploy the Spark2 jar to the Hadoop cluster. Refer to the --jars definition below for the Spark2 jar file path and name to use based on your version of Scala.
  2. Start the Spark job using the following command:
    Note: This example assumes the product is installed to /precisely/routing/software as described in Installing the SDK.
    spark-submit --class com.example.spark.app.MyDriver 
--master yarn --deploy-mode cluster 
--jars /precisely/routing/software/spark2/sdk/lib/spectrum-bigdata-routing-sdk-spark2_scala_version-sdk_version.jar 
/path/to/custom/driver.jar driverParameter1 driverParameter2 ...

    where:

    • --jars The path to the jar file:

    Scala 2.12: 

    /precisely/routing/software/spark2/sdk/lib/spectrum-bigdata-routing-sdk-spark2_2.12-sdk_version.jar

    Scala 2.11: 

    /precisely/routing/software/spark2/sdk/lib/spectrum-bigdata-routing-sdk-spark2_2.11-sdk_version.jar