Skip to content
File Service

Azure Synapse Workspace Setup

Steps to configure

To use the Trimble Connect File System (TCFS) on Azure Synapse, follow these steps:

  1. Download the TCFS JAR: The latest JAR is available as a Maven package in Trimble Artifactory.
  2. Workspace creation: In Microsoft Azure portal, create a Synapse workspace. There is existing workspace tcfs-hdfs-dev, under Dataocean DEV subscription. There will be workspace URL login with that.
  3. Sparkpool Creation: Create a medium apache spark pool. There is a existing Apache spark pool tcfsdevspark(8vCores, 64GB memory). Make sure that spark pool has access to test jar from Azure Blob Storage.
  4. Set the spark properties: Make sure that these properties are set in your EMR Serverless application configuration or in spark job test class itself set these spark properties:
    • spark.jars : abfss://{container-name}@{storage-account-name}.dfs.core.windows.net/{path/to/jar}
    • spark.hadoop.fs.tcfs.impl : com.trimble.hadoop.fs.FileServiceFileSystem
    • spark.hadoop.fs.AbstractFileSystem.tcfs.impl : com.trimble.hadoop.fs.Tcfs
    • spark.hadoop.fs.defaultFS : tcfs://{space id}@{virtual domain} eg: tcfs://3841cdee-9c67-411a-a6f7-6056d37e057a@stage.us.fs.trimblecloud.com
    • spark.hadoop.fs.tcfs.client.id : {CLIENT-ID}
    • spark.hadoop.fs.tcfs.client.secret : {CLIENT-SECRET}
  5. Add the jars to your Spark job submission: If you are using the Azure CLI to submit your Spark job, include the —jars option with the S3 path to the TCFS JAR.
  6. Example command to submit spark job
az synapse spark job submit --name laz_processor_test \
--workspace-name ${workspaceName} --spark-pool-name ${sparkPoolName} \
--main-definition-file "abfss://${blobName}@${storageAccountName}.dfs.core.windows.net/synapse/artifacts/tests.jar" \
--reference-files "abfss://${blobName}@${storageAccountName}.dfs.core.windows.net/synapse/artifacts/driver.jar" \
--main-class-name com.trimble.seurat.LazFileProcessorTest --executors 2 --executor-size Small