Step 1: Create Cluster
Create a new cluster as per your requirements. We support EMR versions starting from emr-6.2.0+. You can choose any version among the supported versions. We recommend using a cluster with minimum of 32 GB memory and 4 cores.
Step 2: Install SDK jar
First, you need to upload the distribution zip to an AWS S3 bucket. After your cluster has started and is in waiting state, you can copy the distribution zip from S3 to primary node of the cluster and extract it. Copy all folders inside the zip to hdfs path of your choice. We will need this path later to configure our spark session.
Step 3: Run spark-submit command for drivers with options as per your requirement
Example:
spark-submit
--class com.precisely.bigdata.li.spark.app.pointinpolygon.PointInPolygonDriver
--master local hdfs:///<hdfs-path-to-extarcted-zip>/spark3/driver/location-intelligence-bigdata-spark3drivers_2.12-0-SNAPSHOT-all.jar
--input s3a://<s3-uri-for-input-data>
--input-format=csv
--csv header=true delimiter=','
--table-file-type TAB
--table-file-path s3a://<s3-uri-of-polygon-data>
--table-file-name USZIPBDY.TAB
--latitude-column-name lat
--longitude-column-name lon
--output /home/hadoop/testfolder/output
--output-fields ZIP, Name
--include-empty-search-results
--overwrite
--download-location /home/hadoop/testfolder/downloadLocationThis example uses --download-location parameter to specify the location where you want to download the data. Make sure that the user has write permission on the download location directory.