Skip to content
File Service

AWS EMR Serverless Setup

Steps to configure

To use the Trimble Connect File System (TCFS) on AWS EMR Serverless, follow these steps:

  1. Download the TCFS JAR: The latest JAR is available as a Maven package in Trimble Artifactory.
  2. Upload the JAR to S3: Upload the downloaded JAR file to an S3 bucket that your EMR Serverless application can access.
  3. Create or Update EMR Serverless Application:
    • When creating or updating your EMR Serverless application, make sure that it has enough subnets in its VPC configuration to run jobs with more executors. And subnets should have internet access to reach Trimble Connect services
    • And make sure Application limits are set properly to run your jobs.
    • And the IAM role associated with the EMR Serverless application should have the necessary permissions to access the S3 bucket where the TCFS JAR is stored.
    • Add the EMR release version that supports your Spark version 3.4.4 and Java runtime 11. For example, EMR release version 6.14.0 or later.
  4. Set the spark properties: Make sure that these properties are set in your EMR Serverless application configuration or in spark job test class itself set these spark properties:
    • spark.jars : s3://{BUCKET-NAME}/{PATH}/trimble-hadoop-fs-tcfs-{version}.jar
    • spark.hadoop.fs.tcfs.impl : com.trimble.hadoop.fs.FileServiceFileSystem
    • spark.hadoop.fs.AbstractFileSystem.tcfs.impl : com.trimble.hadoop.fs.Tcfs
    • spark.hadoop.fs.defaultFS : tcfs://{space id}@{virtual domain} eg: tcfs://3841cdee-9c67-411a-a6f7-6056d37e057a@stage.us.fs.trimblecloud.com
    • spark.hadoop.fs.tcfs.client.id : {CLIENT-ID}
    • spark.hadoop.fs.tcfs.client.secret : {CLIENT-SECRET}
  5. Add the jars to your Spark job submission: If you are using the AWS CLI to submit your Spark job, include the —jars option with the S3 path to the TCFS JAR.
  6. Example AWS CLI command to submit a Spark job with TCFS:
aws emr-serverless start-job-run \
    --application-id 00fupapjetc19g09 \
    --execution-role-arn arn:aws:iam::409785416740:role/FSDevIAM-JobRuntimeRoleForEMRServerlessRole-vk6hT0chFFBW \
    --name "bunnydoub" \
    --job-driver '{
        "sparkSubmit":{
            "entryPoint":"s3://fs-dev-tcfs-emr-test/emr-serverless-jars/tests1.jar",
            "entryPointArguments":["-c","CLIENT_ID","-s","SECRET","-l","INFO","-r","aws","-e","20","-cr","8","-m","30g","-p","aws"],
            "sparkSubmitParameters":"--class com.trimble.seurat.LazBunnyDoubleTest
            --jars s3://fs-dev-tcfs-emr-test/emr-serverless-jars/testedJars/trimblecloud-fileservice-hdfs-1.0.0-test.jar
            --packages io.delta:delta-core_2.12:2.4.0 --conf spark.emr-serverless.driverEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64
            --conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64"
    }
}