# Stream Jira Issues to Redpanda for Real-Time Metrics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [labs-full.txt](https://docs.redpanda.com/labs-full.txt) --- title: Stream Jira Issues to Redpanda for Real-Time Metrics latest-operator-version: v26.1.4 latest-console-tag: v3.7.3 latest-connect-version: 4.93.0 latest-redpanda-tag: v26.1.9 docname: jira-metrics-pipeline page-component-name: labs page-version: master page-component-version: master page-component-title: Labs page-relative-src-path: jira-metrics-pipeline.adoc page-edit-url: https://github.com/redpanda-data/redpanda-labs/edit/main/docs/modules/docker-compose/pages/jira-metrics-pipeline.adoc description: Build a real-time Jira metrics pipeline using Redpanda Connect and Redpanda to track development performance, SLA compliance, and team productivity. page-git-created-date: "2025-11-21" page-git-modified-date: "2025-11-21" --- This lab demonstrates a production-ready pipeline that streams Jira issues to Redpanda topics in real-time. The pipeline transforms raw Jira API responses into a normalized, consumer-friendly schema and routes issues to different topics, enabling use cases SLA monitoring and team performance analytics. ## [](#architecture)Architecture Redpanda Connect periodically queries the Jira REST API for recently updated issues in your project. Each issue is transformed to extract key fields, flatten nested objects, and compute flags like `is_high_priority` and `is_completed`. Based on these flags, issues are routed to different Redpanda topics: - `jira.issues.all`: All issues - `jira.issues.high-priority`: Issues with Highest/High priority - `jira.issues.completed`: Issues marked as Done/Closed/Resolved ## [](#use-cases)Use cases ### [](#sla-monitoring-and-alerting)SLA monitoring and alerting - Automatically detect stale high-priority issues. - Send alerts when issues haven’t been updated in a certain number of days. - Track response times and prevent SLA breaches. - Route critical issues to on-call teams. ### [](#team-performance-analytics)Team performance analytics - Calculate team velocity and throughput. - Identify bottlenecks in the development process. - Track individual contributor metrics. - Generate sprint reports automatically. ### [](#compliance-and-audit-trail)Compliance and audit trail - Maintain a complete history of issue changes. - Immutable event log in Redpanda. - Audit who changed what and when. ## [](#prerequisites)Prerequisites You must have the following installed on your host machine: - [Docker and Docker Compose](https://docs.docker.com/compose/install/) - [rpk](https://docs.redpanda.com/current/get-started/rpk-install/) (Redpanda CLI) - required to generate the Enterprise license This lab also requires: - **Jira instance** with API access - **Jira API token** from [Atlassian](https://id.atlassian.com/manage-profile/security/api-tokens) This lab is intended for Linux and macOS users. If you are using Windows, you must use the Windows Subsystem for Linux (WSL) to run the commands in this lab. ## [](#run-the-lab)Run the lab 1. Clone the repository: ```bash git clone https://github.com/redpanda-data/redpanda-labs.git ``` 2. Change into the `docker-compose/jira-metrics-pipeline/` directory: ```bash cd redpanda-labs/docker-compose/jira-metrics-pipeline ``` 3. Copy the example environment file: ```bash cp .env.example .env ``` 4. Edit the `.env` file and configure: ```bash # Versions (optional - defaults are provided) REDPANDA_VERSION=v26.1.9 REDPANDA_CONSOLE_VERSION=v3.7.3 REDPANDA_CONNECT_VERSION=4.70.0 # Jira credentials (required) JIRA_BASE_URL=https://.atlassian.net JIRA_USERNAME= JIRA_API_TOKEN= JIRA_PROJECT= ``` 5. Generate a Redpanda Enterprise trial license: ```bash rpk generate license \ --name "" \ --last-name "" \ --email "" \ --company "" ``` This creates a 30-day trial license in `./redpanda.license`. 6. Export the license as a shell environment variable: ```bash export REDPANDA_LICENSE=$(cat ./redpanda.license) ``` > 📝 **NOTE** > > The `REDPANDA_LICENSE` must be exported in your shell. It cannot be loaded from the `.env` file. The `--env-file` flag loads application variables (like `JIRA_*`), but the license must be exported separately. 7. Start the Docker containers: ```bash docker compose up -d ``` 8. Create the Kafka topics: ```bash docker compose exec redpanda rpk topic create jira.issues.all -p 3 docker compose exec redpanda rpk topic create jira.issues.high-priority -p 3 docker compose exec redpanda rpk topic create jira.issues.completed -p 3 ``` 9. Open Redpanda Console at [localhost:8080](http://localhost:8080) to view your topics and messages. 10. Monitor the Jira issues streaming into Redpanda: ```bash # Watch all issues docker compose exec redpanda rpk topic consume jira.issues.all --format json # Watch high-priority issues docker compose exec redpanda rpk topic consume jira.issues.high-priority --format json # Watch completed issues docker compose exec redpanda rpk topic consume jira.issues.completed --format json ``` ## [](#how-it-works)How it works The pipeline performs the following steps: 1. **Query Jira**: Every 30 seconds, Redpanda Connect queries Jira for recently updated issues in your project 2. **Transform data**: Each Jira issue is transformed into a normalized schema: - Flattened fields: status, priority, issue type (extracted from nested Jira objects) - Null-safe people fields: assignee and reporter (defaults to "Unassigned"/"Unknown") - Raw timestamps: created, updated, resolved (preserved in original Jira format) - Extracted component names from component objects - Computed boolean flags: `is_completed`, `is_high_priority` - Generated issue URL: `{JIRA_BASE_URL}/browse/{key}` - Pipeline processing timestamp 3. **Route messages**: Issues are intelligently routed to topics based on computed flags: - High priority (Highest/High) → `jira.issues.high-priority` - Completed (Done/Closed/Resolved) → `jira.issues.completed` - All others → `jira.issues.all` 4. **Consume and analyze**: Multiple downstream applications can consume from these topics independently: - Metrics dashboards (Grafana, Kibana) - Alert systems (Slack, PagerDuty) - Data warehouses (Snowflake, BigQuery) - Workflow automation ## [](#pipeline-configuration)Pipeline configuration The complete Redpanda Connect pipeline configuration: ```yaml # JIRA Metrics Pipeline # # This pipeline queries JIRA issues and streams them to Redpanda topics # for real-time metrics, alerting, and analytics. # # Environment variables required: # - REDPANDA_LICENSE: Enterprise license (must be exported in shell) # - JIRA_BASE_URL: https://your-domain.atlassian.net # - JIRA_USERNAME: your-email@example.com # - JIRA_API_TOKEN: your-api-token # - JIRA_PROJECT: YOUR_PROJECT_KEY # - REDPANDA_BROKERS: redpanda:9092 (set in docker-compose.yml) input: generate: mapping: | # Query JIRA for recent issues (last 7 days for testing) root.jql = "project = ${JIRA_PROJECT} AND updated >= -7d ORDER BY updated DESC" root.maxResults = 10 root.fields = [ "key", "summary", "status", "priority", "assignee", "reporter", "created", "updated", "resolutiondate", "issuetype", "labels", "components" ] # Query every 30 seconds interval: 30s pipeline: processors: # Execute JIRA query - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 100 request_timeout: 30s max_retries: 10 # Transform and enrich each issue with metrics - mapping: | # Basic issue info root.issue_key = this.key root.summary = this.fields.summary root.status = this.fields.status.name root.priority = this.fields.priority.name root.issue_type = this.fields.issuetype.name root.url = "${JIRA_BASE_URL}/browse/" + this.key # People root.assignee = if this.fields.assignee != null { this.fields.assignee.displayName } else { "Unassigned" } root.reporter = if this.fields.reporter != null { this.fields.reporter.displayName } else { "Unknown" } # Dates (keep original format for downstream processing) root.created = this.fields.created root.updated = this.fields.updated root.resolved = this.fields.resolutiondate # Labels and components root.labels = this.fields.labels root.components = this.fields.components.map_each(c -> c.name) # Flags for routing root.is_completed = this.fields.status.name.lowercase().contains("done") || this.fields.status.name.lowercase().contains("closed") || this.fields.status.name.lowercase().contains("resolved") root.is_high_priority = ["Highest", "High"].contains(this.fields.priority.name) # Note: Timestamp-based metrics (age, staleness, lead time) can be calculated # by downstream consumers using the raw `created`, `updated`, and `resolved` fields. # Add pipeline processing timestamp root.pipeline_timestamp = now() # Route to primary topic based on issue properties - mapping: | # Route based on priority and completion status meta kafka_topic = if this.is_high_priority { "jira.issues.high-priority" } else if this.is_completed { "jira.issues.completed" } else { "jira.issues.all" } output: kafka: addresses: ["${REDPANDA_BROKERS}"] topic: '${! meta("kafka_topic") }' max_in_flight: 1 batching: count: 100 period: 1s compression: snappy ``` ## [](#customize-the-pipeline)Customize the pipeline ### [](#adjust-polling-frequency)Adjust polling frequency Edit `connect-configs/jira-pipeline.yaml`: ```yaml input: generate: interval: 60s # Change from 30s to 1m, 5m, etc. ``` ### [](#modify-jql-query)Modify JQL query Target different issues by changing the JQL query: ```yaml root.jql = "project = YOUR_PROJECT AND status != Closed AND updated >= -1h" ``` ### [](#add-custom-fields)Add custom fields Include Jira custom fields in your pipeline: ```yaml root.fields = [ "key", "summary", "customfield_10001", # Add your custom field ID # ... other fields ] ``` ### [](#add-more-routing-rules)Add more routing rules Route issues to additional topics based on labels, components, or other criteria: ```yaml - mapping: | meta kafka_topic = if this.labels.contains("security") { "jira.issues.security" } else { deleted() } ``` ## [](#view-logs-and-metrics)View logs and metrics ### [](#redpanda-connect-logs)Redpanda Connect logs ```bash docker compose logs -f connect ``` ### [](#redpanda-logs)Redpanda logs ```bash docker compose logs -f redpanda ``` ### [](#redpanda-connect-metrics)Redpanda Connect metrics ```bash curl http://localhost:4195/metrics ``` ## [](#troubleshooting)Troubleshooting ### [](#license-errors)License errors If you see license-related errors in the Connect logs: 1. Verify the license is exported: ```bash echo $REDPANDA_LICENSE ``` 2. If the environment variable is empty, regenerate and export the license: ```bash rpk generate license --name "Your Name" \ --last-name "Last" \ --email "you@example.com" \ --company "Company" export REDPANDA_LICENSE=$(cat ./redpanda.license) ``` 3. Restart the containers: ```bash docker compose restart connect ``` ### [](#jira-authentication-errors)Jira authentication errors If you receive 401 Unauthorized errors: 1. Verify your API token is correct. 2. Ensure you’re using your email address as the username. 3. Check that your Jira instance URL is correct (include `https://`). 4. Test your credentials with curl: ```bash curl -u "${JIRA_USERNAME}:${JIRA_API_TOKEN}" \ "${JIRA_BASE_URL}/rest/api/3/myself" ``` ### [](#no-issues-appearing)No issues appearing If no issues are being published: 1. Check the JQL query returns results in Jira directly. 2. Verify the `JIRA_PROJECT` environment variable matches your project key. 3. Check Connect logs for errors: ```bash docker compose logs connect ``` 4. Adjust the query timeframe: ```yaml root.jql = "project = YOUR_PROJECT AND updated >= -24h" # Last 24 hours ``` ### [](#rate-limiting)Rate limiting If you hit Jira rate limits (HTTP 429): 1. Increase the polling interval in `connect-configs/jira-pipeline.yaml`: ```yaml interval: 60s # Reduce frequency to 60 seconds ``` 2. Increase retry settings: ```yaml - jira: max_retries: 20 request_timeout: 60s ``` 3. Use more specific JQL queries to reduce result sizes. ## [](#clean-up)Clean up To shut down and delete the containers along with all cluster data: ```bash docker compose down -v ``` ## [](#next-steps)Next steps After you have Jira issues streaming into Redpanda, you can extend this pipeline with Redpanda Connect outputs: ### [](#send-alerts-and-notifications)Send alerts and notifications - **Slack**: Post issue updates to channels using the [`slack_post` output](https://docs.redpanda.com/connect/components/outputs/slack_post/) - **Discord**: Send notifications with the [`discord` output](https://docs.redpanda.com/connect/components/outputs/discord/) - **HTTP/Webhooks**: Trigger PagerDuty, Opsgenie, or custom webhooks using the [`http_client` output](https://docs.redpanda.com/connect/components/outputs/http_client/) ### [](#stream-to-data-warehouses)Stream to data warehouses - **Snowflake**: Load issues into Snowflake with the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) - **BigQuery**: Stream to Google BigQuery using the [`gcp_bigquery` output](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery/) - **PostgreSQL/MySQL**: Store in relational databases with [`sql_insert`](https://docs.redpanda.com/connect/components/outputs/sql_insert/) or [`sql_raw` outputs](https://docs.redpanda.com/connect/components/outputs/sql_raw/) ### [](#calculate-metrics)Calculate metrics Use the raw timestamp fields (`created`, `updated`, `resolved`) to calculate: - **Lead time**: Average time from creation to completion - **Cycle time**: Average time from "In Progress" to "Done" - **Throughput**: Issues completed per time period - **Aging**: Distribution of issue ages - **SLA compliance**: Percentage of issues resolved within target timeframes ## [](#suggested-reading)Suggested reading - [Jira Processor Reference](https://docs.redpanda.com/connect/components/processors/jira/) - [Jira REST API Documentation](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/) - [JQL Query Guide](https://www.atlassian.com/software/jira/guides/jql) - [Redpanda Quickstart](https://docs.redpanda.com/streaming/current/get-started/quick-start/)