Skip to main content
Version: 23.1

Create a MirrorMaker2 Checkpoint Connector

You can use the MirrorMaker2 Checkpoint connector to import consumer group offsets from other Kafka clusters.

Prerequisites

  • The external Kafka cluster is accessible.
  • A service account with read-only access to the external cluster is available.
  • The Kafka cluster topics connector is running for the same source cluster, with a matching configuration.

Create a MirrorMaker2 Checkpoint Source connector

To create the MirrorMaker2 Checkpoint Source connector:

  1. In Redpanda Cloud, click Connectors in the navigation menu, and then click Create Connector.

  2. Select Import from Kafka cluster offsets.

  3. On the Create Connector page, specify the following required connector configuration options:

    PropertyDescription
    Topics to replicateComma-separated topic names and regexes you want to replicate.
    Source cluster broker listA comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers regardless of which servers are specified here for bootstrapping.
    Source cluster security protocolThe protocol used to communicate with source brokers. The default is PLAINTEXT.
    GroupsConsumer groups to replicate. Supports comma-separated group IDs and regexes.
    Connector nameGlobally-unique name to use for this connector.
  4. Click Next. Review the connector properties specified, then click Create.

Advanced MirrorMaker2 Checkpoint Source connector configuration

In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following optional advanced connector configuration properties by selecting Show advanced options on the Create Connector page:

PropertyDescription
Source cluster SSL custom certificateTrusted certificates in the PEM format.
Source cluster SSL keystore keyPrivate key in the PEM format.
Source cluster SSL keystore certificate chainCertificate chain in the PEM format.
Topics excludeExcluded topics. Supports comma-separated topic names and regexes.
Source cluster aliasWhen using DefaultReplicationPolicy, topic names will be prefixed with it.
Replication policy classClass that defines the remote topic naming convention. Use IdentityReplicationPolicy to preserve topic names. DefaultReplicationPolicy prefixes the topic with the source cluster alias.
Emit checkpoints interval secondsFrequency of checkpoints. The default is 60.
Sync group offsets enabledSpecifies whether or not to periodically write the translated offsets to the __consumer_offsets topic in the target cluster, as long as no active consumers in that group are connected to the target cluster.
Sync group offsets interval secondsFrequency of consumer group offset sync. The default is 60.
Refresh groups interval secondsFrequency of group refreshes. The default is 600.
Offset-Syncs topic locationThe location (source or target) of the offset-syncs topic. The default is source.
Checkpoints topic replication factorReplication factor for checkpoints topic. The dfault is -1.

Test the connection

After the connector is created:

  • Ensure that there are no errors in logs and in Redpanda Console.
  • Wait for the Kafka cluster topics connector to catch up. Then check to confirm that the consumer groups are replicated.

Troubleshoot

Most MirrorMaker2 Checkpoint Source connector issues are reported as a failed task at the time of creation.

MessageAction
Connection to node -1 (invalid-url.com/103.224.182.242:9092) could not be established. Broker may not be available. / LOGS: Timed out while checking for or creating topic 'mm2-offset-syncs.target.internal'. This could indicate a connectivity issue / TimeoutException: Timed out waiting for a node assignmentMake sure broker URLs are correct and that the source cluster security protocol is correct.
SaslAuthenticationException: SASL authentication failed: security: Invalid credentialsCheck to confirm that the username and password specified are correct.
java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka configCheck to confirm that the username and password specified are correct.
Client SASL mechanism 'PLAIN' not enabled in the server, enabled mechanisms are [SCRAM-SHA-256, SCRAM-SHA-512]Check to confirm that the respective Source cluster SASL mechanism is correct.
SaslAuthenticationException: SASL authentication failed: security: Invalid credentialsMake sure the respective Source cluster SASL mechanism is correct (for example, SCRAM-SHA-256 instead of SCRAM-SHA-512).
terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issueEnable the SSL using Source cluster security protocol (specify SSL or SASL_SSL).

What do you like about this doc?




Optional: Share your email address if we can contact you about your feedback.

Let us know what we do well: