23.2 23.1 22.3 22.2 22.1 21.11

Create an HTTP Source Connector

You can use an HTTP Source connector to enable change data capture (CDC) from JSON/HTTP APIs into Redpanda. The HTTP Source connector imports data from HTTP services as batches or increments.

Prerequisites

  • A request URL that is accessible by the connector

  • A user/password created in source so that the connector can authenticate

Limitations

Only JSON or AVRO formats can be used as a value converter.

Create an HTTP Source connector

To create the HTTP Source connector:

  1. In Redpanda Cloud, click Connectors in the navigation menu, and then click Create Connector.

  2. Select Import from HTTP.

  3. On the Create Connector page, specify the following required connector configuration options:

    Property Description

    Topic

    Name of the topic where the record will be sent.

    HTTP URL

    HTTP URL to use in the request.

    HTTP request params

    HTTP query parameters to use in the request. Use & separated list = separated pairs. For example, name=value&name2=value2.

    Authentication type

    Type of authentication. If using Basic, provide User and Password. Default is None.

    Kafka message key format

    Format of the key in the Kafka topic. Default is JSON. If choosing AVRO, specify the Kafka message value format.

    Message key JSON contains schema

    Specifies whether or not the message key contains schema in the schema field.

    Max Tasks

    Maximum number of tasks to use for this connector. The default is 1. Each task replicates an exclusive set of partitions assigned to it.

    Connector name

    Globally-unique name to use for this connector.

  4. Click Next. Review the connector properties specified, then click Create.

Map data

Use JSON or AVRO converters to map data from an HTTP endpoint to a Redpanda topic.

Test the connection

After the connector is created:

  • Check the connector status and confirm that there are no errors in logs and in Redpanda Console.

  • Review the Redpanda topic to confirm that it contains the expected data.

Troubleshoot

Most HTTP Source connector issues are reported as a failed task. An invalid connection URL is reported in logs.

Additional errors and corrective actions follow.

Message Action

RetriableException: java.net.UnknownHostException: invalid-url.com.pl: Name or service not known / RetriableException: java.net.SocketTimeoutException: Connect timed out

Verify that the HTTP URL is correct, the hostname and port are correct, and that the service accepts connections.

IllegalStateException: Policy failed for response code: 403

Confirm that the service URL accepts requests from the connector, with HTTP code 200.

IllegalArgumentException: No node at '/items' (unmatched part: '/items')

Confirm that HTTP response list pointer is a valid pointer in the HTTP result document.

IllegalArgumentException: No node at '/invalid-key' (unmatched part: '/invalid-key')

Confirm that the HTTP response record offset pointer defines valid elements of the HTTP result document.