Configuring a Source Connector

Step 1: Download binary files

Download the latest version of the Confluent connector zip/jar binary from Artifactory{:target=“_blank”}.

Step 2: Setting up a source connector

In Confluent Cloud, a custom connector plugin refers to a custom-built connector that is developed by users to connect to an external system, in this case the Events service, with Apache Kafka®.

Prerequisites

Confluent connector binary files
A running cluster

The following procedures assume the use of the Confluent Cloud Platform’s Kafka Connect framework. Refer to the Confluent documentation{:target=“_blank”} for screen-by-screen configuration details.

Upload the source connector

After downloading the connector binary files, you can upload the zip file directly to Confluent Cloud.

To add a source connector:

Go to Connectors in your Confluent Cloud cluster. If you have one or more existing connectors, select Add Connector.
In the Connector Plugins section, select Add Plugin.
A custom connector plugin is needed to create a source connector. In the Add Custom Connector Plugin page, enter the required Connector plugin details.
- Connector plugin name: Provide a meaningful name for your connector.
- Custom plugin description: Provide a description for your connector.
- Connector class: Enter the Java class or alias for the connector. See the connector.class in the source connector properties table for more details about the value.
- Connector type: Click Source.
Select Select connector archive. To upload the archive file, select the local .zip file that you downloaded from Artifactory.
To exclude sensitive properties from being logged, select Add sensitive property. For example, by adding app.secret, the client secret is excluded from the log. Remember to identify and handle sensitive information to ensure security and compliance.
Check the box that indicates that you are responsible for the connector and then select Submit. Once you submit the archive, the source connector is deployed to Confluent Cloud.

Step 3: Configure and launch the source connector

To configure and launch the source connector:

On the Connectors page displays, select Filter by: Deployment and select custom. The uploaded custom connector is displayed.
Select the way you want to provide Kafka Cluster credentials. Select Use an existing API key.
Select Continue.
Add the following JSON configuration with appropriate values:
```
  {
     "env": "[ENVIRONMENT]",
     "app.id": "[APP_ID]",
     "app.secret": "[APP_SECRET]",
     "app.name": "[APP_NAME]",
     "events": "[EVENT_NAMESPACE]-[EVENT_NAME]-[EVENT_VERSION]"
   }
```
The parameters within square brackets must be substituted with the relevant values. For details regarding these values, refer to the Source connector configuration properties.

To enable the Datadog monitoring, refer to Monitoring Connectors.
Add Trimble Identity and Events consumer endpoints to the safe list.

Example:

https://id.trimble.com
Staging: preprd-events-ws-consumer.events.trimblecloud.com
Prod: events-ws-consumer.events.trimblecloud.com
Select the number of Tasks and select Continue.

The tasks in Kafka Connect serve as consumer threads, responsible for receiving partitions and reading data from them. Suppose you have ten partitions and set the tasks.max configuration to 5. In this case, each task will handle two partitions, effectively tracking their respective offsets.

However, it is important to note that if you set tasks.max to a value higher than the total partition count, Kafka Connect will launch a number of tasks equal to the total partitions, while the remaining tasks will remain idle and unused.

Note: Therefore, it is strongly recommended to configure the tasks.max parameter to be equal to the number of partitions present in your topics. This ensures optimal utilization of resources and efficient distribution of workload among the tasks.
The status of your connector should read Provisioning. Select the card to get the provisioning details.
Once the connector completes provisioning, the status changes to running.