Connect
Self-Managed Only

sap_hana

License: This component requires an enterprise license. You can either upgrade to an Enterprise Edition license, or generate a trial license key that's valid for 30 days.

Reads rows from a SAP HANA table.

Reads rows from a SAP HANA table. Supports five modes:

  • bulk: reads all rows once then the input terminates (use with sequence for periodic re-reads).

  • incrementing: polls for rows where incrementing_column exceeds the last seen value, emitting only net-new rows.

  • query: executes a user-supplied SQL statement and emits one message per result row.

  • timestamp: polls for rows where timestamp_column falls within (last_hwm, database_now - timestamp_delay]. The bound is read from the database clock (see timestamp_clock) and the delay absorbs commit lag. The HWM advances to the window bound once the window is fully consumed, so a restart mid-window re-reads that window from its start (rows sharing a timestamp cannot be split, hence no finer checkpoint).

  • timestamp+incrementing: like timestamp but orders by (timestamp_column, incrementing_column) and resumes after the exact (timestamp, incrementing) pair of the last delivered row, so rows sharing a timestamp are neither duplicated nor missed and a restart mid-window continues from the last acknowledged batch rather than the window start.

Metadata

Messages produced in bulk, incrementing, timestamp, and timestamp+incrementing modes carry the following metadata fields (query mode attaches none):

  • table_name: The HANA table name.

  • database_schema: The configured schema_name. Only present when schema_name is set.

  • schema: Avro-compatible schema derived from SYS.TABLE_COLUMNS, suitable for use with schema_registry_encode. Column additions are detected automatically without a pipeline restart. Only present when schema_name is configured.

  • primary_key_columns: JSON array of the table’s primary-key column names in key order. Only present when schema_name is configured and the table has a primary key.

Troubleshooting

schema and primary_key_columns metadata are missing. The input reads them from SYS.TABLE_COLUMNS and SYS.INDEX_COLUMNS; when that read fails it logs a warning and carries on without them, and incrementing_initial_value falls back to a guessed bind type. Check the warning in the logs and grant the connecting user CATALOG READ (or SELECT on the table’s schema), and make sure schema_name is set.

"invalid table name" or "invalid column name" for a table that exists. Identifiers in this config are quoted when sent to HANA, so they must match the catalog’s case exactly. HANA upper-cases identifiers that were created without quotes: a table created as create table orders is ORDERS in the catalog and must be configured as such.

Batches arrive one at a time. fetch_size larger than checkpoint_limit means a batch cannot be handed on until the previous one is acknowledged (a warning is logged at startup). Raise checkpoint_limit or lower fetch_size.

Rows written around the time of a poll are missing in timestamp modes. The poll window ends at the database clock minus timestamp_delay; a row whose timestamp was assigned before that bound but committed after the poll is never seen. Set timestamp_delay longer than your longest write transaction, and pick the timestamp_clock that matches how the column is populated.

  • Common

  • Advanced

input:
  label: ""
  sap_hana:
    dsn: "" # No default (required)
    schema_name: "" # No default (optional)
    table: "" # No default (optional)
    mode: bulk
    query: "" # No default (optional)
    incrementing_column: "" # No default (optional)
    incrementing_initial_value: ""
    poll_interval: 60s
    timestamp_column: "" # No default (optional)
    timestamp_initial_value: ""
    timestamp_delay: 5s
    numeric_mapping: none
    checkpoint_cache: "" # No default (optional)
    auto_replay_nacks: true
input:
  label: ""
  sap_hana:
    dsn: "" # No default (required)
    fetch_size: 128
    schema_name: "" # No default (optional)
    table: "" # No default (optional)
    mode: bulk
    query: "" # No default (optional)
    incrementing_column: "" # No default (optional)
    incrementing_initial_value: ""
    poll_interval: 60s
    timestamp_column: "" # No default (optional)
    timestamp_initial_value: ""
    timestamp_delay: 5s
    timestamp_clock: database
    numeric_mapping: none
    max_retries: 3
    retry_backoff: 1s
    checkpoint_cache: "" # No default (optional)
    checkpoint_cache_key: sap_hana_hwm
    checkpoint_limit: 1024
    auto_replay_nacks: true

Fields

auto_replay_nacks

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Type: bool

Default: true

checkpoint_cache

Name of a cache resource to persist the high-water mark across restarts. When set, the connector resumes from where it left off rather than starting from scratch. Only effective in incrementing, timestamp, and timestamp+incrementing modes. The cache must be declared under cache_resources. Choose a durable backend (Redis, PostgreSQL) for production; in-memory caches lose state on restart.

Type: string

# Examples:
checkpoint_cache: redis_cache

checkpoint_cache_key

Key used to store the checkpoint in checkpoint_cache. Change this when multiple sap_hana inputs share the same cache resource to avoid key collisions.

Type: string

Default: sap_hana_hwm

checkpoint_limit

The maximum number of messages that can be in flight (read but not yet acknowledged) at a given time. The high-water mark is only checkpointed once every message before it has been acknowledged, preserving at-least-once delivery even when batches are acknowledged out of order. When fetch_size exceeds this limit, batches are delivered one at a time (each waits for the previous batch’s acknowledgement), so raise this alongside large fetch_size values to keep batches flowing concurrently.

Type: int

Default: 1024

dsn

SAP HANA connection DSN in hdb://user:password@host:port form.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

# Examples:
dsn: hdb://user:password@host:39017

fetch_size

Number of rows requested per FetchNext round-trip. Larger values reduce round-trips on high-latency connections.

Type: int

Default: 128

incrementing_column

Column to use as the high-water mark for incrementing mode. Must be strictly monotonically increasing with no duplicate values — a BIGINT auto-increment column is ideal. Columns that produce duplicate values (e.g. a plain TIMESTAMP) will cause rows whose value ties span a fetch_size boundary to be re-delivered. Use timestamp+incrementing mode to handle timestamp columns safely.

Type: string

incrementing_initial_value

Initial high-water mark value. When empty, all existing rows are emitted on the first run. The value is converted to the incrementing_column’s type from the catalog on connect: integers for integer columns, RFC3339 or `YYYY-MM-DD[ HH:MM:SS] for DATE/TIMESTAMP columns, hexadecimal for BINARY/VARBINARY columns, and the literal string (leading zeros preserved) for character columns. A persisted checkpoint takes precedence over this value.

Type: string

Default: ""

max_retries

Maximum number of times to retry a failed query before returning an error. Set to 0 to disable retries.

Type: int

Default: 3

mode

Operation mode.

Type: string

Default: bulk

Options: bulk, incrementing, query, timestamp, timestamp+incrementing

numeric_mapping

Controls how DECIMAL/NUMERIC columns are emitted:

  • none: emit as a canonical decimal string preserving full precision, except integer-typed columns (DECIMAL(p,0) with precision ⇐ 18) which are emitted as integers.

  • best_fit: additionally emit columns whose precision fits a double (precision ⇐ 15) as floating-point numbers; wider values fall back to canonical decimal strings.

Type: string

Default: none

Options: none, best_fit

poll_interval

How long to wait between polls in incrementing, timestamp, and timestamp+incrementing modes.

Type: string

Default: 60s

# Examples:
poll_interval: 10s

# ---

poll_interval: 5m

query

Custom SQL statement to execute. Only used when mode is query.

Type: string

retry_backoff

Base delay between query retries. The delay grows linearly with the attempt number (attempt N waits N times this value).

Type: string

Default: 1s

schema_name

Database schema for the table. When set, an Avro-compatible schema metadata field is attached to every message using data from SYS.TABLE_COLUMNS.

Type: string

table

Table to read from. Required when mode is bulk or incrementing.

Type: string

timestamp_clock

Which database clock bounds each timestamp-mode poll window. The bound is always read from HANA, never from the connector host, so it shares the clock and timezone convention of the column it is compared against:

  • database: CURRENT_TIMESTAMP (session timezone), matching columns populated by DEFAULT CURRENT_TIMESTAMP or NOW().

  • database_utc: CURRENT_UTCTIMESTAMP, for columns populated with UTC values.

Type: string

Default: database

Options: database, database_utc

timestamp_column

Column to use as the high-water mark for timestamp and timestamp+incrementing modes. Must be a TIMESTAMP or LONGDATE column.

Type: string

timestamp_delay

Commit-lag buffer for timestamp modes. The upper bound for each poll is the database clock minus timestamp_delay, so rows whose timestamp was assigned slightly before a still-uncommitted transaction finished are not missed.

Type: string

Default: 5s

# Examples:
timestamp_delay: 0s

# ---

timestamp_delay: 30s

timestamp_initial_value

Initial high-water mark in RFC3339 format (e.g. 2024-01-01T00:00:00Z). When empty, all existing rows are emitted on the first run.

Type: string

Default: ""

Examples

Incremental reads with a durable checkpoint

Polls an orders table for new rows by primary key every 30 seconds and persists the high-water mark in a file cache so a restart resumes where it left off.

input:
  sap_hana:
    dsn: hdb://user:password@hana-host:39017
    mode: incrementing
    schema_name: SALES
    table: ORDERS
    incrementing_column: ORDER_ID
    poll_interval: 30s
    checkpoint_cache: hana_checkpoints

cache_resources:
  - label: hana_checkpoints
    file:
      directory: /var/lib/redpanda-connect/sap_hana

Change tracking by timestamp with automatic Avro schema registration

Follows an updated-at column together with the primary key so rows sharing a timestamp are neither duplicated nor missed, bounds each poll by the database’s UTC clock with a commit-lag buffer, and registers the table’s schema from the emitted metadata before encoding.

input:
  sap_hana:
    dsn: hdb://user:password@hana-host:39017
    mode: timestamp+incrementing
    schema_name: SALES
    table: ORDERS
    timestamp_column: UPDATED_AT
    incrementing_column: ORDER_ID
    timestamp_clock: database_utc
    timestamp_delay: 30s
    poll_interval: 10s
    checkpoint_cache: hana_checkpoints

pipeline:
  processors:
    - schema_registry_encode:
        url: http://schema-registry:8081
        subject: sales.orders-value
        schema_metadata: schema
        format: avro
        avro:
          raw_json: true

output:
  redpanda:
    seed_brokers: [ "broker:9092" ]
    topic: sales.orders

cache_resources:
  - label: hana_checkpoints
    file:
      directory: /var/lib/redpanda-connect/sap_hana