Connect

sap_hana

License: This component requires an enterprise license. You can either upgrade to an Enterprise Edition license, or generate a trial license key that's valid for 30 days.

Reads rows from a SAP HANA table.

Reads rows from a SAP HANA table. Supports five modes:

  • bulk: reads all rows once then the input terminates (wrap it in read_until with restart_input: true for periodic re-reads).

  • incrementing: polls for rows where incrementing_column exceeds the last seen value, emitting only net-new rows. Values from an IDENTITY column or a sequence are assigned when a row is inserted, not when it commits: with concurrent writers a lower value can commit after a higher one has been read, and that row is never emitted. Use timestamp+incrementing with a timestamp_delay longer than your longest write transaction when the table has concurrent writers.

  • query: executes a user-supplied SQL statement and emits one message per result row.

  • timestamp: polls for rows where timestamp_column falls within (last_hwm, database_now - timestamp_delay]. The bound is read from the database clock (see timestamp_clock) and the delay absorbs commit lag. The HWM advances to the window bound once the window is fully consumed. Mid-window, a batch checkpoints the timestamp of its last row when the next row has a later one, so a restart re-reads only the rows of the timestamp that was in progress (rows sharing a timestamp cannot be split).

  • timestamp+incrementing: like timestamp but orders by (timestamp_column, incrementing_column) and resumes after the exact (timestamp, incrementing) pair of the last delivered row, so rows sharing a timestamp are neither duplicated nor missed and a restart mid-window continues from the last acknowledged batch rather than the window start.

Metadata

Messages produced in bulk, incrementing, timestamp, and timestamp+incrementing modes carry the following metadata fields (query mode attaches none):

  • table_name: The HANA table name.

  • database_schema: The configured schema_name. Only present when schema_name is set.

  • schema: Avro-compatible schema derived from SYS.TABLE_COLUMNS, suitable for use with schema_registry_encode. Column additions are detected automatically without a pipeline restart. Only present when schema_name is configured.

  • primary_key_columns: JSON array of the table’s primary-key column names in key order. Only present when schema_name is configured and the table has a primary key.

Permissions

The connecting user needs SELECT on the source table, or on the statement’s tables in query mode. The user also needs read access to system views in these cases:

  • When schema_name is set, the input reads SYS.TABLE_COLUMNS, SYS.INDEX_COLUMNS, and SYS.INDEXES to build the schema and primary_key_columns metadata.

  • In the timestamp and timestamp+incrementing modes, the input reads SYS.TABLE_COLUMNS (or SYS.VIEW_COLUMNS for a view) to check the type of timestamp_column when it connects. Without access, the check is skipped and a warning is logged.

  • When incrementing_initial_value is set and no checkpoint exists yet, the input reads SYS.TABLE_COLUMNS (or SYS.VIEW_COLUMNS for a view) to bind the initial value with the type of incrementing_column. Without catalog rows, it test-binds the value against the column (as a number when it parses as one, then as the literal string) and fails to connect when neither binds.

HANA lists catalog rows for the objects a user holds a privilege on, so SELECT on a table or view is enough for these reads. A view has no rows in SYS.TABLE_COLUMNS whatever the privileges, so it is read without schema and primary_key_columns metadata. For other objects the user cannot see in the catalog, grant CATALOG READ.

Known limitations

  • Polling only: deleted rows are never seen, and incrementing mode never sees updates to existing rows. The timestamp modes see an update only when the application also updates timestamp_column.

  • Rows whose incrementing_column or timestamp_column is NULL are never emitted.

  • timestamp_column must be a TIMESTAMP, LONGDATE or SECONDDATE column. Change times stored as DEC(15,0) UTC timestamps or as separate date and time character columns, as in many SAP application tables, cannot drive the timestamp modes.

  • schema and primary_key_columns metadata come from SYS.TABLE_COLUMNS, which does not describe views or calculation views, so those are read without schema metadata.

  • schema_registry_encode derives Avro field names from column names. Column names that are not valid Avro names, such as SAP’s /BIC/ namespaced columns, fail to encode; register the schema yourself and rename the columns in a mapping instead of using schema_metadata. Table names are sanitised automatically.

  • bulk and query have no position to resume from: an error part-way through a read restarts it from the first row, re-sending the rows already delivered.

  • query runs its statement once and then the input ends, like bulk, and attaches no metadata. Calculation views with input parameters can only be read this way.

  • The cursor stays open while downstream components apply backpressure. On a busy HANA system a long-open cursor holds back MVCC garbage collection, so index the incrementing_column or timestamp_column and avoid outputs that stall for long periods. The first poll of a polling mode without an initial value reads the whole table in key order; set incrementing_initial_value or timestamp_initial_value to skip history you do not need. Raise fetch_size (for example to 1000 to 5000) on high-latency links, together with checkpoint_limit.

Troubleshooting

checkpoint key …​ holds the position of …​ on connect. Two sap_hana inputs share a checkpoint_cache and checkpoint_cache_key. Give each input its own checkpoint_cache_key. The error also appears when schema_name or table changed for an existing input: the stored position is for the old reference, so delete the key to start over or set incrementing_initial_value/timestamp_initial_value under a new key.

Cannot connect to SAP HANA Cloud. HANA Cloud accepts TLS connections only, on port 443. go-hdb enables TLS when a TLS parameter is present, so add TLSServerName=<host> to the DSN (and TLSRootCAFile=<path> if the CA is not in the system trust store).

schema and primary_key_columns metadata are missing. The input reads them from SYS.TABLE_COLUMNS and SYS.INDEX_COLUMNS; when that read fails it logs a warning and carries on without them. Views never have them. For a table, check the warning in the logs, grant the connecting user SELECT on the table (or CATALOG READ), and make sure schema_name is set.

"invalid table name" or "invalid column name" for a table that exists. Identifiers in this configuration are quoted when sent to HANA, so they must match the catalog’s case exactly. HANA upper-cases identifiers that were created without quotes: a table created as create table orders is ORDERS in the catalog and must be configured as such.

Batches arrive one at a time. fetch_size larger than checkpoint_limit means a batch cannot be handed on until the previous one is acknowledged (a warning is logged at startup). Raise checkpoint_limit or lower fetch_size.

Rows written around the time of a poll are missing in timestamp modes. The poll window ends at the database clock minus timestamp_delay; a row whose timestamp was assigned before that bound but committed after the poll is never seen. Set timestamp_delay longer than your longest write transaction, and pick the timestamp_clock that matches how the column is populated.

  • Common

  • Advanced

input:
  label: ""
  sap_hana:
    dsn: "" # No default (required)
    schema_name: "" # No default (optional)
    table: "" # No default (optional)
    mode: bulk
    query: "" # No default (optional)
    incrementing_column: "" # No default (optional)
    incrementing_initial_value: ""
    poll_interval: 60s
    timestamp_column: "" # No default (optional)
    timestamp_initial_value: ""
    timestamp_delay: 5s
    numeric_mapping: none
    checkpoint_cache: "" # No default (optional)
    auto_replay_nacks: true
input:
  label: ""
  sap_hana:
    dsn: "" # No default (required)
    fetch_size: 128
    schema_name: "" # No default (optional)
    table: "" # No default (optional)
    mode: bulk
    query: "" # No default (optional)
    incrementing_column: "" # No default (optional)
    incrementing_initial_value: ""
    poll_interval: 60s
    timestamp_column: "" # No default (optional)
    timestamp_initial_value: ""
    timestamp_delay: 5s
    timestamp_clock: database
    numeric_mapping: none
    max_retries: 3
    retry_backoff: 1s
    checkpoint_cache: "" # No default (optional)
    checkpoint_cache_key: sap_hana_hwm
    checkpoint_limit: 1024
    auto_replay_nacks: true

Fields

auto_replay_nacks

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Type: bool

Default: true

checkpoint_cache

Name of a cache resource to persist the high-water mark across restarts. When set, the connector resumes from where it left off rather than starting from scratch. Only effective in incrementing, timestamp, and timestamp+incrementing modes. The cache must be declared under cache_resources. Choose a durable backend (redpanda, redis, PostgreSQL via sql) for production; memory and lru caches lose the position on every restart or redeploy, and the input then re-reads from the initial value. Without this field the position is also kept in memory only, and a warning is logged at startup.

Type: string

# Examples:
checkpoint_cache: redis_cache

checkpoint_cache_key

Key used to store the checkpoint in checkpoint_cache. Every sap_hana input sharing a cache resource needs its own key: the stored checkpoint records its table, and an input refuses to start from another table’s checkpoint.

Type: string

Default: sap_hana_hwm

checkpoint_limit

The maximum number of messages that can be in flight (read but not yet acknowledged) at a given time. The high-water mark is only checkpointed once every message before it has been acknowledged, preserving at-least-once delivery even when batches are acknowledged out of order. When fetch_size exceeds this limit, batches are delivered one at a time (each waits for the previous batch’s acknowledgement), so raise this alongside large fetch_size values to keep batches flowing concurrently.

Type: int

Default: 1024

dsn

SAP HANA connection DSN in hdb://user:password@host:port[?param=value&…​] form. URL-encode reserved characters in the user name and password (# as %23, @ as %40, % as %25). Useful parameters: databaseName selects a tenant database when connecting through the system database port, defaultSchema sets the schema unqualified names resolve against, and TLSServerName, TLSRootCAFile and TLSInsecureSkipVerify configure TLS, which SAP HANA Cloud requires.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

# Examples:
dsn: hdb://user:password@host:39017

# ---

dsn: hdb://user:password@host:30013?databaseName=HXE

# ---

dsn: hdb://user:password@<instance-id>.hana.prod-eu10.hanacloud.ondemand.com:443?TLSServerName=<instance-id>.hana.prod-eu10.hanacloud.ondemand.com

fetch_size

Number of rows requested per FetchNext round-trip. Larger values reduce round-trips on high-latency connections.

Type: int

Default: 128

incrementing_column

Column to use as the high-water mark for incrementing mode. Must be strictly monotonically increasing with no duplicate values. A BIGINT auto-increment column is ideal. Columns that produce duplicate values (for example, a plain TIMESTAMP) will cause rows whose value ties span a fetch_size boundary to be re-delivered. Use timestamp+incrementing mode to handle timestamp columns safely.

Type: string

incrementing_initial_value

Initial high-water mark value. When empty, all existing rows are emitted on the first run. The value is converted to the incrementing_column’s type from the catalog on connect: integers for integer columns, RFC3339 or `YYYY-MM-DD[ HH:MM:SS] for DATE/TIMESTAMP columns (the column’s wall time, with no offset other than Z or +00:00), hexadecimal for BINARY/VARBINARY columns, and the literal string (leading zeros preserved) for character columns. A persisted checkpoint takes precedence over this value.

Type: string

Default: ""

max_retries

Maximum number of times to retry a failed query before returning an error. Set to 0 to disable retries; negative values are rejected.

Type: int

Default: 3

mode

Operation mode.

Type: string

Default: bulk

Options: bulk, incrementing, query, timestamp, timestamp+incrementing

numeric_mapping

Controls how DECIMAL/NUMERIC columns are emitted:

  • none: emit as a canonical decimal string preserving full precision, except integer-typed columns (DECIMAL(p,0) with precision ⇐ 18) which are emitted as integers.

  • best_fit: additionally emit columns whose precision fits a double (precision ⇐ 15) as floating-point numbers; wider values fall back to canonical decimal strings.

Type: string

Default: none

Options: none, best_fit

poll_interval

How long to wait between polls in incrementing, timestamp, and timestamp+incrementing modes.

Type: string

Default: 60s

# Examples:
poll_interval: 10s

# ---

poll_interval: 5m

query

Custom SQL statement to execute. Only used when mode is query.

Type: string

retry_backoff

Base delay between query retries. The delay grows linearly with the attempt number (attempt N waits N times this value).

Type: string

Default: 1s

schema_name

Database schema for the table. When set, an Avro-compatible schema metadata field is attached to every message using data from SYS.TABLE_COLUMNS.

Type: string

table

Table to read from. Required when mode is bulk or incrementing.

Type: string

timestamp_clock

Which database clock bounds each timestamp-mode poll window. The bound is always read from HANA, never from the connector host, so it shares the clock and timezone convention of the column it is compared against:

  • database: CURRENT_TIMESTAMP (session timezone), matching columns populated by DEFAULT CURRENT_TIMESTAMP or NOW().

  • database_utc: CURRENT_UTCTIMESTAMP, for columns populated with UTC values.

Type: string

Default: database

Options: database, database_utc

timestamp_column

Column to use as the high-water mark for timestamp and timestamp+incrementing modes. Must be a TIMESTAMP or LONGDATE column.

Type: string

timestamp_delay

Commit-lag buffer for timestamp modes. The upper bound for each poll is the database clock minus timestamp_delay, so rows whose timestamp was assigned slightly before a still-uncommitted transaction finished are not missed. Set it longer than your longest write transaction: a row whose transaction commits after the poll window covering its timestamp has closed is never emitted. Negative values are rejected.

Type: string

Default: 5s

# Examples:
timestamp_delay: 0s

# ---

timestamp_delay: 30s

timestamp_initial_value

Initial high-water mark in RFC3339 format (for example, 2024-01-01T00:00:00Z). When empty, all existing rows are emitted on the first run. timestamp_column has no timezone, so write its wall-clock value as-is, tagged with a Z or +00:00 offset, rather than converting it. Any other offset is rejected when the configuration is parsed.

Type: string

Default: ""

Examples

Incremental reads with a durable checkpoint

Polls an orders table for new rows by primary key every 30 seconds and persists the high-water mark in a file cache so a restart resumes where it left off.

input:
  sap_hana:
    dsn: hdb://user:password@hana-host:39017
    mode: incrementing
    schema_name: SALES
    table: ORDERS
    incrementing_column: ORDER_ID
    poll_interval: 30s
    checkpoint_cache: hana_checkpoints
    checkpoint_cache_key: sales_orders

cache_resources:
  - label: hana_checkpoints
    file:
      directory: /var/lib/redpanda-connect/sap_hana

Change tracking by timestamp with automatic Avro schema registration

Follows an updated-at column together with the primary key so rows sharing a timestamp are neither duplicated nor missed, bounds each poll by the database’s UTC clock with a commit-lag buffer, and registers the table’s schema from the emitted metadata before encoding. The checkpoint is kept in a Redpanda topic, which survives redeploys and needs no local disk (use a compacted topic).

input:
  sap_hana:
    dsn: hdb://user:password@hana-host:39017
    mode: timestamp+incrementing
    schema_name: SALES
    table: ORDERS
    timestamp_column: UPDATED_AT
    incrementing_column: ORDER_ID
    timestamp_clock: database_utc
    timestamp_delay: 30s
    poll_interval: 10s
    checkpoint_cache: hana_checkpoints
    checkpoint_cache_key: sales_orders_changes

pipeline:
  processors:
    - schema_registry_encode:
        url: http://schema-registry:8081
        subject: sales.orders-value
        schema_metadata: schema
        format: avro
        avro:
          raw_json: true

output:
  redpanda:
    seed_brokers: [ "broker:9092" ]
    topic: sales.orders

cache_resources:
  - label: hana_checkpoints
    redpanda:
      seed_brokers: [ "broker:9092" ]
      topic: sap_hana_checkpoints

Run a custom query

Runs a SQL statement once and emits one message per result row. In query mode, table and schema_name are not required and no metadata is attached.

input:
  sap_hana:
    dsn: hdb://user:password@hana-host:39017
    mode: query
    query: |
      SELECT ORDER_ID, CUSTOMER_ID, STATUS
      FROM SALES.ORDERS
      WHERE STATUS = 'PENDING'