OpenTelemetry

An observability transformation focused on shared instrumentation and meaningful operational signals with common push-based path for logs, metrics, and traces.

ObservabilityTransformationReliability

Organization: Qualys
Role: Technical Program Manager
Industry: Cybersecurity Enterprise software
Duration: Not disclosed

Objective

OTEL is a Kubernetes-based automation pipeline for OpenTelemetry-format signals from applications to observability backends. It standardizes application telemetry, separates ingestion from processing through Kafka, and reuses existing Prometheus, ClickHouse, Grafana, and HyperDX platforms.

Business Challenge

Inconsistent telemetry patterns: Applications used varied logging, metric, and tracing mechanisms, limiting correlation and standardization.
Scale and resilience: The platform needed controlled resource use, backpressure handling, fault tolerance, self-healing, and high availability.
Data integrity: Pipeline design had to prevent loss under bursts, Kafka lag, gateway interruption, and downstream write pressure.
Enterprise query needs: ETM required ClickHouse log storage and full-text search of body and important fields.
Security: Telemetry could inadvertently include secrets, credentials, or keys.

Context

Scope: Kubernetes-based OTEL telemetry for logs, metrics, and traces. The ETM pilot is limited to Connector logs.
Projected Pilot Volume: One of the pilot team stated a worst-case projection of 2.5 million messages per day: 500 executions with 5,000 messages per execution.
Platform Direction: OTEL Collector runs as a Kubernetes DaemonSet, Kafka decouples processing, and OTel Gateway deployments scale through HPA.
Existing Backends: Prometheus stores metrics. ClickHouse stores OTEL logs and traces in otel_db. Grafana and HyperDX provide the user experience.

Responsibilities

Align scope, operating cadence, and adoption sequencing
Surface cross-team dependencies and risks
Connect technical signals with stakeholder decisions

Execution

Created common practices around instrumentation, telemetry, and operational review.

Pipeline Implementation

Deployed OTEL Collector as a Kubernetes DaemonSet with OTLP gRPC and HTTP/HTTPS receivers.
Added Kafka exporters for logs, metrics, and traces, using separate raw-signal topics.
Deployed OTel Gateways with processing, backend exporters, and HPA.
Configured Prometheus for metrics and ClickHouse for logs and traces.
Connected Grafana and HyperDX to the relevant data sources.

Performance Testing and Tuning

Tested 135 million logs in 2–3 minutes, above ETM’s projected 2.5 million messages per day.
Observed initial data loss in a 36-million-log test; changed gateway memory-limiter settings, increased exporter consumers from 10 to 20, and increased queue size from 10,000 to 100,000.
Follow-up 36-million and 90-million-log tests reported no data drop.
A continuous test reached 5 GB per minute and was stopped after 1.73 billion records because of development-pod storage constraints.

Outcomes

Common telemetry pipeline: OTEL Collector → Kafka → OTel Gateway → Prometheus / ClickHouse architecture completed.
Capacity evidence above pilot demand: 135 million logs tested in 2–3 minutes against ETM’s stated 2.5-million-message daily projection.
Data-loss remediation validated: Queue and memory tuning produced no data loss at 36 million and 90 million log tests.
Operational controls implemented: Alerting covers availability, Kafka lag, stale lag metrics, exporter failures, queue failures, and receiver refusals.
Pilot and architecture governance active: Pilot run and Architecture Review Board approval process are in progress.

Lessons Learned

Kafka is a resilience boundary: Decoupling ingestion and processing supports burst management and downstream protection.
Pilot workload is not sufficient for capacity planning: Independent testing above ETM demand was necessary.
Measured tuning matters: Gateway queue and memory adjustments eliminated observed data loss in follow-up tests.
Storage is a platform dependency: A healthy pipeline can still be constrained by test-environment storage.
Telemetry security begins at the source: Application teams must sanitize data until automated secret detection is available.
Alerting is part of the product: Availability, lag, export, queue, and receiver signals are required operational controls.

Business Challenge

Inconsistent telemetry patterns: Applications used varied logging, metric, and tracing mechanisms, limiting correlation and standardization.

Scale and resilience: The platform needed controlled resource use, backpressure handling, fault tolerance, self-healing, and high availability.

Data integrity: Pipeline design had to prevent loss under bursts, Kafka lag, gateway interruption, and downstream write pressure.

Enterprise query needs: ETM required ClickHouse log storage and full-text search of body and important fields.

Security: Telemetry could inadvertently include secrets, credentials, or keys.

Context

Scope: Kubernetes-based OTEL telemetry for logs, metrics, and traces. The ETM pilot is limited to Connector logs.

Projected Pilot Volume: One of the pilot team stated a worst-case projection of 2.5 million messages per day: 500 executions with 5,000 messages per execution.

Platform Direction: OTEL Collector runs as a Kubernetes DaemonSet, Kafka decouples processing, and OTel Gateway deployments scale through HPA.

Existing Backends: Prometheus stores metrics. ClickHouse stores OTEL logs and traces in otel_db. Grafana and HyperDX provide the user experience.

Execution

Created common practices around instrumentation, telemetry, and operational review.

Pipeline Implementation

Deployed OTEL Collector as a Kubernetes DaemonSet with OTLP gRPC and HTTP/HTTPS receivers.
Added Kafka exporters for logs, metrics, and traces, using separate raw-signal topics.
Deployed OTel Gateways with processing, backend exporters, and HPA.
Configured Prometheus for metrics and ClickHouse for logs and traces.
Connected Grafana and HyperDX to the relevant data sources.

Performance Testing and Tuning

Tested 135 million logs in 2–3 minutes, above ETM’s projected 2.5 million messages per day.
Observed initial data loss in a 36-million-log test; changed gateway memory-limiter settings, increased exporter consumers from 10 to 20, and increased queue size from 10,000 to 100,000.
Follow-up 36-million and 90-million-log tests reported no data drop.
A continuous test reached 5 GB per minute and was stopped after 1.73 billion records because of development-pod storage constraints.

Outcomes

Common telemetry pipeline: OTEL Collector → Kafka → OTel Gateway → Prometheus / ClickHouse architecture completed.

Capacity evidence above pilot demand: 135 million logs tested in 2–3 minutes against ETM’s stated 2.5-million-message daily projection.

Data-loss remediation validated: Queue and memory tuning produced no data loss at 36 million and 90 million log tests.

Operational controls implemented: Alerting covers availability, Kafka lag, stale lag metrics, exporter failures, queue failures, and receiver refusals.

Pilot and architecture governance active: Pilot run and Architecture Review Board approval process are in progress.

Lessons Learned

Kafka is a resilience boundary: Decoupling ingestion and processing supports burst management and downstream protection.

Pilot workload is not sufficient for capacity planning: Independent testing above ETM demand was necessary.

Measured tuning matters: Gateway queue and memory adjustments eliminated observed data loss in follow-up tests.

Storage is a platform dependency: A healthy pipeline can still be constrained by test-environment storage.

Telemetry security begins at the source: Application teams must sanitize data until automated secret detection is available.

Alerting is part of the product: Availability, lag, export, queue, and receiver signals are required operational controls.