Create an HTTP Source Connector
You can use an HTTP Source connector to enable change data capture (CDC) from JSON/HTTP APIs into Redpanda. The HTTP Source connector imports data from HTTP services as batches or increments.
A request URL that is accessible by the connector
A user/password created in source so that the connector can authenticate
To create the HTTP Source connector:
In Redpanda Cloud, click Connectors in the navigation menu, and then click Create Connector.
Select Import from HTTP.
On the Create Connector page, specify the following required connector configuration options:
Name of the topic where the record will be sent.
HTTP URL to use in the request.
HTTP request params
HTTP query parameters to use in the request. Use
&separated list = separated pairs. For example,
Type of authentication. If using
Password. Default is None.
Kafka message key format
Format of the key in the Kafka topic. Default is
JSON. If choosing
AVRO, specify the
Kafka message value format.
Message key JSON contains schema
Specifies whether or not the message key contains schema in the schema field.
Maximum number of tasks to use for this connector. The default is
1. Each task replicates an exclusive set of partitions assigned to it.
Globally-unique name to use for this connector.
Click Next. Review the connector properties specified, then click Create.
After the connector is created:
Check the connector status and confirm that there are no errors in logs and in Redpanda Console.
Review the Redpanda topic to confirm that it contains the expected data.
Most HTTP Source connector issues are reported as a failed task. An invalid connection URL is reported in logs.
Additional errors and corrective actions follow.
RetriableException: java.net.UnknownHostException: invalid-url.com.pl: Name or service not known / RetriableException: java.net.SocketTimeoutException: Connect timed out
Verify that the HTTP URL is correct, the hostname and port are correct, and that the service accepts connections.
IllegalStateException: Policy failed for response code: 403
Confirm that the service URL accepts requests from the connector, with HTTP code 200.
IllegalArgumentException: No node at '/items' (unmatched part: '/items')
IllegalArgumentException: No node at '/invalid-key' (unmatched part: '/invalid-key')
Confirm that the