Resolve Errors in Linux
This section describes errors or issues you might encounter while deploying Redpanda in Linux and explains how to troubleshoot them.
Deployment issues
This section addresses common deployment issues encountered during Redpanda setup or upgrades.
A Redpanda Enterprise Edition license is required
When you upgrade to a new feature release while enterprise features are in use without a valid license, the first broker that you start on the new version refuses to start. Starting in 26.3, a cluster without a license that runs more than 50 cores in total is also refused. See Community Edition Core Limit. The other brokers keep serving clients on the previous version, and the refused broker’s data is untouched.
The refused broker logs an error like the following and exits a few seconds after it starts. This example comes from a cluster that exceeds the core limit:
ERROR ... Failure during startup: A Redpanda Enterprise Edition license is required to use the currently enabled feature(s): (cluster_cores_over_limit). To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set <key>', or via Redpanda Console. To request an Enterprise license, please visit https://redpanda.com/upgrade. To try Redpanda Enterprise for 30 days, visit https://redpanda.com/try-enterprise. ...
This cluster has 51 cores across 3 brokers (0, 1, 2), more than the 50-core limit of Redpanda Community Edition. Reduce the cluster to 50 cores or fewer, or install an Enterprise license. To resolve the core limit: install a license on the running cluster with 'rpk cluster license set <key>'; or, if this broker was just added, decommission it with 'rpk cluster brokers decommission <node-id> --skip-liveness-check'; or restart it with fewer shards (--smp) so the cluster total is 50 cores or fewer. Any other feature listed in this message still needs a license. Brokers already running the new version keep running.
The features in parentheses are the ones in violation. A license clears all of them. For more ways to confirm the cause, see Confirm the cause.
To recover, choose one of these options, in order of preference:
-
Install a license. Apply a valid Redpanda Enterprise Edition license through any broker that is still running. The license takes effect on the running cluster immediately. If you don’t have a license key,
rpk generate license --applyapplies a 30-day trial license. Then start the refused broker again on the same new version, and the rolling upgrade continues. -
Roll back the refused broker. Put the previous version back on that broker and start it. The cluster returns to normal operation on the previous version with the same data.
-
Reinstall the previous version on the refused broker:
-
Debian/Ubuntu
-
Fedora/RedHat
sudo apt install redpanda=<previous-version> --allow-downgradessudo yum downgrade redpanda-<previous-version> -
-
Upgrade again after you install a license or bring the cluster within the limit.
-
-
Bring the cluster into compliance without a license. Do one of the following through any broker that is still running:
-
Disable the enterprise features that the error lists. Clusters created on Redpanda 25.3 or later run two of them by default:
partition_auto_balancing_continuous(setpartition_autobalancing_modetonode_add) andcore_balancing_continuous(setcore_balancing_continuoustofalse). -
If the error lists only
cluster_cores_over_limit, reduce the cluster to 50 cores or fewer. If the refused broker was just added, decommission it withrpk cluster brokers decommission <node-id> --skip-liveness-check. Otherwise, restart a broker with a lower--smpvalue. See Bring a cluster within the limit.Then start the refused broker again on the same new version, and the rolling upgrade continues.
-
To start the refused broker, on either version:
-
If the broker logs
Crash loop detected, delete thestartup_logfile from its data directory, or wait an hour since the last crash:sudo rm /var/lib/redpanda/data/startup_log -
Start the broker:
sudo systemctl start redpanda
If no broker is running, for example in a single-broker cluster or when you stopped every broker before the upgrade, you can’t install a license through rpk. Put the previous version back on a broker and start it, install the license, then upgrade again.
Confirm the cause
The refused broker exits with code 1. From a broker that is still running, rpk cluster upgrade status shows the refused broker as not alive (false in the ALIVE column) with no release version (- in the RELEASE-VERSION column). The cluster’s active version stays at the previous release’s logical version, for example 19 for 26.2. For details about the command’s output, see Defer Upgrade Finalization.
Earlier in its startup, the broker logs the values that the check used, which can help you confirm the cause:
Verifying enterprise license: active_version=<version>, latest_version=<version>, enterprise_features=[<features>], cluster_cores=<cores>/50, license_missing_or_expired=true
If something restarts the refused broker automatically, it can reach the crash_loop_limit, which by default allows five consecutive crashes within one hour. After that, the broker logs Crash loop detected and doesn’t start, even after you fix the license problem, until you delete the startup_log file from its data directory or an hour passes since the last crash.
On Linux, this happens when the Redpanda systemd unit uses Restart=always or Restart=on-failure. The stock unit uses Restart=on-abnormal, so a refused broker stays stopped instead.
TLS issues
This section covers common TLS errors, their causes, and solutions, including certificate issues and correct client configuration.
Invalid large response size
This error appears when your cluster is configured to use TLS, but you don’t specify that you are connecting over TLS.
unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint?
If you’re using rpk, ensure to add the -X tls.enabled flag, and any other necessary TLS flags such as the TLS certificate:
rpk cluster info -X tls.enabled=true
For all available flags, see the rpk options reference.
Malformed HTTP response
This error appears when a cluster has TLS enabled, and you try to access the admin API without passing the required TLS parameters.
Retrying POST for error: Post "http://127.0.0.1:9644/v1/security/users": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x03\x00\x02\x02"
If you’re using rpk, ensure to include the TLS flags.
For all available flags, see the rpk options reference.
x509: certificate signed by unknown authority
This error appears when the Certificate Authority (CA) that signed your certificates is not trusted by your system.
Check the following:
-
Ensure you have installed the root CA certificate correctly on your local system.
-
If using a self-signed certificate, ensure it is properly configured and included in your system’s trust store.
-
If you are using a certificate issued by a CA, ensure the issuing CA is included in your system’s trust store.
-
Check the validity of your certificates. They might have expired.
x509: certificate is not valid for any names
This error indicates that the certificate you are using is not valid for the specific domain or IP address you are trying to use it with. This error typically occurs when there is a mismatch between the certificate’s Subject Alternative Name (SAN) or Common Name (CN) field and the name being used to access the broker.
To fix this error, you may need to obtain a new certificate that is valid for the specific domain or IP address you are using. Ensure that the certificate’s SAN or CN entry matches the name being used, and that the certificate is not expired or revoked.
cannot validate certificate for 127.0.0.1
This error appears if you are using a CA certificate when you try to establish an internal connection using localhost. For example:
unable to request metadata: unable to dial: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
To fix this error, you must either specify the URL with a public domain or use self-signed certificates:
rpk cluster info \
-X brokers=<redpanda-url>:<port> \
-X tls.enabled=true
SASL issues
This section addresses errors related to SASL (Simple Authentication and Security Layer), focusing on connection and authentication problems.
Is SASL missing?
This error appears when you try to interact with a cluster that has SASL enabled without passing a user’s credentials.
unable to request metadata: broker closed the connection immediately after a request was issued, which happens when SASL is required but not provided: is SASL missing?
If you’re using rpk, ensure to specify the -X user, -X pass, and -X sasl.mechanism flags.
For all available flags, see the rpk options reference.
pattern_type is unspecified
When creating a shadow link with rpk shadow create, you may see:
Invalid cluster link configuration: pattern_type is unspecified
Ensure pattern_type values are uppercase: LITERAL, PREFIX.
broker_not_available with TLS enabled
When creating a shadow link with TLS enabled, you may see:
Cluster link unreachable, preflight check failed - { node: -1 }, { error_code: broker_not_available [8] }
The shadow cluster cannot verify the source cluster’s TLS certificate. This is the most common issue when using TLS with self-signed certificates (the default for Kubernetes deployments with tls.certs.default.caEnabled=true).
Ensure that the shadow link configuration includes the source cluster’s CA certificate.
Wrong SSL version number
When creating a shadow link, you may see in the source cluster logs:
Disconnected (applying protocol, Wrong SSL Version number: ensure client is configured to use TLS)
The source cluster requires TLS but your shadow link configuration is missing TLS settings or has tls_settings.enabled: false.
broker_not_available without TLS
When creating a shadow link without TLS, you may see:
Cluster link unreachable, preflight check failed - { node: -1 }, { error_code: broker_not_available [8] }
Verify that bootstrap_servers addresses are reachable from the shadow cluster and that ports are correct.
Connection timeout
When creating a shadow link, the command may hang or timeout without completing.
Check network connectivity between shadow and source clusters. Verify firewall rules and network policies allow traffic between the namespaces.
Topics in FAULTED state
When monitoring shadow links, you may see topics showing FAULTED state in status output.
A shadow topic enters the FAULTED state when its source topic is deleted and then recreated with the same name. Shadowing tracks each source topic by its topic ID, and it marks the shadow topic as faulted when either of the following happens on the source cluster:
-
The topic ID of the source topic changes.
-
The source topic has fewer partitions than the shadow topic.
A faulted shadow topic stops replicating and stays in the FAULTED state until you replace it.
To confirm the cause, check the shadow cluster’s broker logs for a warning that contains has changed its topic ID or has fewer partitions than expected. The warning names the faulted topic.
Deleting a source topic without recreating it does not fault the shadow topic. The shadow topic and the data it already replicated remain on the shadow cluster.
Authentication, permission, and network problems do not fault individual topics. They affect the shadow link’s tasks, which report states such as LINK_UNAVAILABLE.
If you use rpk to update the shadow link in the following steps, use rpk v26.2.2 or later, such as the version that ships with Redpanda 26.3, or rpk v26.1.15 or a later 26.1 version. rpk v26.2.1, and v26.1.14 and earlier, add the filters you edit to the existing filter list instead of replacing the list, so they cannot remove the exclude filter after you delete the topic.
To resume replication from a recreated source topic, replace the faulted shadow topic:
-
Update the shadow link to add an exclude filter that matches the name of each faulted topic exactly (
LITERALpattern type) to the auto-create topic filters. See Update an existing shadow link.Redpanda rejects the deletion of a shadow topic that matches the auto-create topic filters, so add the exclude filter before you delete the topic.
-
Delete each faulted shadow topic on the shadow cluster:
rpk topic delete <topic-name>This deletes the shadow topic and all data it replicated from the original source topic.
-
Update the shadow link again to remove the exclude filters that you added.
On its next topic metadata sync, Shadowing creates a new shadow topic for each recreated source topic.
-
Verify that the topics are replicating:
rpk shadow status <shadow-link-name>The replaced topics show the
ACTIVEstate.
High replication lag
When monitoring shadow links, you may see LAG values continuously increasing in rpk shadow status.
Check the following:
-
Check source cluster load: high produce rate may exceed replication capacity
-
Check shadow cluster resources: CPU, memory, or disk constraints
-
Check network bandwidth: verify sufficient bandwidth between clusters
To resolve:
-
Scale shadow cluster resources if constrained
-
Verify network connectivity and bandwidth
-
Review topic configuration for optimization opportunities
Task shows LINK_UNAVAILABLE
When monitoring shadow links, you may see tasks showing LINK_UNAVAILABLE state with "No brokers available" message.
Common causes include:
-
Source cluster requires SASL authentication but shadow link not configured for it
-
Source cluster unreachable from shadow cluster
-
Network policy blocking traffic between clusters
To resolve:
-
Verify SASL configuration if source cluster requires authentication
-
Test network connectivity:
kubectl execinto shadow pod and try connecting to source cluster -
Check Kubernetes NetworkPolicies and firewall rules
Application connection failures after failover
Applications may not be able to connect to the shadow cluster after failover.
Confirm authentication credentials are valid for the shadow cluster and test network connectivity from application hosts.
Consumer group offset issues after failover
After failover, consumers may start from the beginning or wrong positions.
If necessary, manually reset offsets to appropriate positions. See How to manage consumer group offsets in Redpanda for detailed reset procedures.