Skip to content
File Service

Trimble Cloud Fileservice HDFS Driver

Overview

The Trimble Cloud Fileservice HDFS Driver is a Hadoop-compatible FileSystem implementation that allows Spark applications to interact with the Trimble Cloud File Service using the tcfs:// URI scheme. This driver enables seamless integration with Trimble’s cloud-based file storage, allowing users to read from and write to Trimble File Service directly from their Spark jobs.

Capabilities

  • Read and write files in Trimble Cloud File Service using HDFS methods.
  • Support for common file operations such as create, delete, rename, and list.
  • Integration with Spark for distributed data processing.
  • Authentication and authorization using Trimble Cloud credentials.

Limitations

  • Currently, it supports all file operations under a single space only.

  • It is not supporting all HDFS features, here is the list of unsupported features:

    • Append method for files
    • Seek to new source when reading files

  • It is currently tested only in Trimble Cloud File Service STAGE environment in AWS US East (N. Virginia) region.

  • For write operations to function, make sure that tcfs:// related spark configurations must be broadcasted across all executors.

    For example (in Scala):

    spark.sparkContext.broadcast(new SerializableConfiguration(spark.sparkContext.hadoopConfiguration));