Spark Configurations Supported by HDFS Driver
The Trimble Cloud Fileservice HDFS Driver supports the following Spark configurations to enable seamless integration with Trimble Cloud File Service:
| Configuration | Value | Description |
|---|---|---|
spark.hadoop.fs.tcfs.impl | com.trimble.hadoop.fs.FileServiceFileSystem | Specifies the implementation class for the TCFS FileSystem. Required. |
spark.hadoop.fs.AbstractFileSystem.tcfs.impl | com.trimble.hadoop.fs.Tcfs | Specifies the abstract file system implementation for TCFS. Required. |
spark.hadoop.fs.defaultFS | tcfs://{space id}@{virtual domain} | Sets the default file system URI to TCFS. Required. |
spark.hadoop.fs.tcfs.client.id | {CLIENT-ID} | Client ID for authenticating with Trimble Cloud File Service. Required. |
spark.hadoop.fs.tcfs.client.secret | {CLIENT_SECRET} | Client secret for authenticating with Trimble Cloud File Service. Required. |
spark.jars | s3://{BUCKET}/{PATH}/trimble-hadoop-fs-tcfs-{VERSION}-all.jar | Path to the TCFS JAR file. Required. |
spark.hadoop.fs.tcfs.http.requestTimeout | Timeout in seconds (e.g., 60) | HTTP request timeout for TCFS operations. Default: 30 seconds. |
spark.hadoop.fs.tcfs.http.retryTimeout | Timeout in seconds (e.g., 120) | HTTP retry timeout for TCFS operations. Default: 30 seconds. |
spark.hadoop.fs.tcfs.tid.token.refresh | Interval in minutes (e.g., 50) | Token refresh interval. Default: 50 minutes. |
spark.hadoop.fs.tcfs.io.readbuffer.size | Size in bytes (e.g., 16384 for 16 KB) | Buffer size for read operations. Default: 256 MB. Min: 16 KB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.writebuffer.size | Size in bytes (e.g., 16384 for 16 KB) | Buffer size for write operations. Default: 4 MB. Min: 16 KB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.multipart.size | Size in bytes (e.g., 5242880 for 5 MB) | Multipart block size for upload operations. Default: 8 MB. Min: 5 MB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.activationTimeout | Timeout in seconds (e.g., 60) | File activation timeout. Default: 60 seconds. |
spark.hadoop.fs.tcfs.io.activationRetryDelay | Delay in seconds (e.g., .1f for 100 ms) | Delay between activation retries. Default: 0.1 seconds. |
spark.hadoop.fs.tcfs.io.writeThreads.max | Number of threads (e.g., 8) | Maximum number of threads for write operations. Default: 8. Min: 1, Max: 8. |
spark.hadoop.fs.tcfs.io.queuedWrites.max | Number of queued writes (e.g., 20) | Maximum number of queued write operations. Default: 20. Should be at least 2x write threads. |
spark.hadoop.fs.tcfs.groupAuth.enabled | true or false | Enables/disables IAM group authentication. Default: false. |