git
Clones a Git repository, reads its contents, then polls for new commits at a configurable interval. Any updates are emitted as new messages.
# Configuration fields, showing default values
input:
label: ""
git:
repository_url: https://github.com/username/repo.git # No default (required)
branch: main
poll_interval: 10s
include_patterns: []
exclude_patterns: []
max_file_size: 10485760
checkpoint_cache: "" # No default (optional)
checkpoint_key: git_last_commit
auth:
basic:
username: "" # No default (optional)
password: "" # No default (optional)
ssh_key:
private_key_path: "" # No default (optional)
private_key: "" # No default (optional)
passphrase: "" # No default (optional)
token:
value: "" # No default (optional)
auto_replay_nacks: true
Metadata
This input adds the following metadata fields to each message:
-
git_file_path
-
git_file_size
-
git_file_mode
-
git_file_modified
-
git_commit
-
git_mime_type
-
git_is_binary
-
git_deleted
(when a source file is deleted)
You can access these metadata fields using function interpolation.
Fields
auth.basic.password
A password to authenticate with.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
auth.ssh_key.passphrase
The passphrase for your SSH private key.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
auth.ssh_key.private_key
Your private SSH key. When using encrypted keys, you must also set a value for private_key_passphrase
.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
auth.ssh_key.private_key_path
The path to your private SSH key file. When using encrypted keys, you must also set a value for private_key_passphrase
.
Type: string
Default: ""
auth.token.value
The token value to use for token-based authentication.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
auto_replay_nacks
Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure.
Set auto_replay_nacks
to false
to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.
Type: bool
Default: true
checkpoint_cache
Specify a cache
resource to store the last processed commit hash. After a restart, Redpanda Connect can then continue processing changes from where it left off, avoiding the need to reprocess all detected updates.
Type: string
checkpoint_key
The key to use when storing the last processed commit hash in the cache.
Type: string
Default: git_last_commit
exclude_patterns[]
A list of file patterns to exclude. For example, you could choose not to read content from certain Git directories or image files: '.git/', '/*.png'
. These patterns take precedence over include_patterns
.
The following patterns are supported:
-
Glob patterns:
,
/
*/
,?
-
Character ranges:
[a-z]
. Escape any character with a special meaning using a backslash.
Type: array
Default: []
include_patterns[]
A list of file patterns to read from. For example, you could read content from only Markdown and YAML files: '*/.md', 'configs/*.yaml'
.
The following patterns are supported:
-
Glob patterns:
,
/
*/
,?
-
Character ranges:
[a-z]
. Escape any character with a special meaning using a backslash.
If this field is left empty, all files are read from.
Type: array
Default: []
max_file_size
The maximum size of files to read from (in bytes). Files that exceed this limit are skipped. Set to 0
for unlimited file sizes.
Type: int
Default: 10485760