Create an HTTP Source Connector
You can use an HTTP Source connector to enable change data capture (CDC) from JSON/HTTP APIs into Redpanda. The HTTP Source connector imports data from HTTP services as batches or increments.
Prerequisites
-
A request URL that is accessible by the connector
-
A user/password created in source so that the connector can authenticate
Create an HTTP Source connector
To create the HTTP Source connector:
-
In Redpanda Cloud, click Connectors in the navigation menu, and then click Create Connector.
-
Select Import from HTTP.
-
On the Create Connector page, specify the following required connector configuration options:
Property name Property key Description Topic
kafka.topic
Name of the topic where the record will be sent.
HTTP URL
http.request.url
HTTP URL to use in the request.
HTTP request params
http.request.params
HTTP query parameters to use in the request. Use
&
separated list = separated pairs. For example,name=value&name2=value2
.Authentication type
http.auth.type
Type of authentication. If using
Basic
, provideUser
andPassword
. Default is None.User
http.auth.user
Basic authentication user.
Password
http.auth.password
Basic authentication password.
Kafka message key format
key.converter
Format of the key in the Redpanda topic. Default is
JSON
. If choosingAVRO
, specify theKafka message key format
.Message key JSON contains schema
key.converter.schemas.enable
Specifies whether or not the message key contains schema in the schema field.
Kafka message value format
value.converter
Format of the value in the Redpanda topic. Default is
JSON
. If choosingAVRO
, specify theKafka message value format
.Message value JSON contains schema
value.converter.schemas.enable
Specifies whether or not the message value contains schema in the schema field.
Max Tasks
tasks.max
Maximum number of tasks to use for this connector. The default is
1
. Each task replicates an exclusive set of partitions assigned to it.Connector name
name
Globally-unique name to use for this connector.
-
Click Next. Review the connector properties specified, then click Create.
Map data
Use JSON
(org.apache.kafka.connect.json.JsonConverter
) or AVRO
(io.confluent.connect.avro.AvroConverter
)
converters to map data from an HTTP endpoint to a Redpanda topic.
Test the connection
After the connector is created:
-
Check the connector status and confirm that there are no errors in logs and in Redpanda Console.
-
Review the Redpanda topic to confirm that it contains the expected data.
Troubleshoot
Most HTTP Source connector issues are reported as a failed task. An invalid connection URL is reported in logs. Select Show Logs to view error details.
Additional errors and corrective actions follow.
Message | Action |
---|---|
RetriableException: java.net.UnknownHostException: invalid-url.com.pl: Name or service not known / RetriableException: java.net.SocketTimeoutException: Connect timed out |
Verify that the HTTP URL is correct, the hostname and port are correct, and that the service accepts connections. |
IllegalStateException: Policy failed for response code: 403 |
Confirm that the service URL accepts requests from the connector, with HTTP code 200. |
IllegalArgumentException: No node at '/items' (unmatched part: '/items') |
Confirm that |
IllegalArgumentException: No node at '/invalid-key' (unmatched part: '/invalid-key') |
Confirm that the |