Create a MirrorMaker2 Source Connector

You can use a MirrorMaker2 Source connector to import messages from another Kafka cluster. You can also use it to:

• Replicate messages from an external Kafka or Redpanda cluster.
• Create topics on the local cluster, with a configuration matching external topics.
• Replicate topic access-control lists (ACLs).

Prerequisites

• The external Kafka cluster must be accessible.
• A service account with full access to the external cluster must be available. You can also use a service account with read-only ACLs when the Offset-Syncs topic location is set to target.

Limitations

• ACLs are copied, but service accounts are not created.
• Only topic ACLs are copied (group ACLs are not).
• Only ACLs for topics matching the connector configuration are copied (write ACLs are not copied).
• All permissions ACLs are downgraded to read-only.

Create a MirrorMaker2 Source connector

To create the MirrorMaker2 Source connector:

1. In Redpanda Cloud, click Connectors in the navigation menu, and then click Create Connector.

2. Select Import from Kafka cluster topics.

3. On the Create Connector page, specify the following required connector configuration options:

   Property	Description
   Topics to replicate	Comma-separated topic names and regexes you want to replicate.
   Source cluster broker list	A comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers regardless of which servers are specified here for bootstrapping. This list only impacts the initial hosts used to discover the full set of servers, and should be in the form host1:port1,host2:port2,.... Because these servers are only used for the initial connection to discover the full cluster membership (which may change dynamically), it need not contain the full set of servers (you may want more than one, though, in case a server is down).
   Source cluster security protocol	The protocol to use to communicate with source brokers. Default is PLAINTEXT.
   Sync topic configs enabled	Specifies whether to periodically configure remote topics to match their corresponding upstream topics.
   Sync topic ACLs enabled	Specifies whether or not to periodically configure remote topic ACLs to match their corresponding upstream topics.
   Connector name	Globally-unique name to use for this connector.

4. Click Next. Review the connector properties specified, then click Create.

Map data

The value converter does not require any schema; it copies data as bytes.

Test the connection

After the connector is created:

• Ensure that there are no errors in logs and in Redpanda Console.
• Confirm that Redpanda topics are being replicated. You should see messages coming into the topics.

Troubleshoot

Most MirrorMaker2 Source connector issues are reported as a failed task at the time of creation.

Message	Action
Connection to node -1 (invalid-url.com/103.224.182.242:9092) could not be established. Broker may not be available. / LOGS: Timed out while checking for or creating topic 'mm2-offset-syncs.target.internal'. This could indicate a connectivity issue / TimeoutException: Timed out waiting for a node assignment	Make sure broker URLs are correct and that the security.protocol is correct.
SaslAuthenticationException: SASL authentication failed: security: Invalid credentials	Confirm that the username and password specified are correct.
Terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue	Error indicates that the SSL should be enabled using Source cluster security protocol (use SSL or SASL_SSL).
RecordTooLargeException: The message is N bytes (...)	Use producer.override.max.request.size property to change max request size.
RecordTooLargeException: The request included (...)	The target server is not able to receive message because it is too large in size. Disabled compression can be a root cause. Consider enabling compression: "Compression type": "snappy",
Scheduler for MirrorSourceConnector caught exception in scheduled task: syncing topic ACLs	MirrorMaker2 requires an authorizer to be configured by the broker side, but it is not. Change the Sync topic ACLs enabled MirrorMaker2 property to false (default is true) to disable ACL syncing.