OpenTelemetry Getting Started Guide

December 2, 2024 · 636 words · 3 min

Three Pillars of Observability

In modern distributed systems, observability is crucial for ensuring system health and performance. Observability typically consists of three main components:

Metrics Metrics are numerical data about system performance and health. They are typically collected periodically and provide snapshots of system state at specific points in time. Common metrics include CPU usage, memory consumption, request rate, error rate, etc. Metrics are primarily used for monitoring and alerting.

Traces Traces record the path of requests as they flow through the system, helping developers understand request lifecycles. Each trace consists of multiple spans, with each span representing the execution of a specific operation in the system. Traces are particularly useful for debugging performance issues and bottlenecks in distributed systems.

Logs Logs are event records generated by the system during runtime, typically including timestamps and event descriptions. Logs can be structured (like JSON format) or unstructured (plain text). Logs are fundamental tools for debugging and troubleshooting.

OpenTelemetry

OpenTelemetry is an open-source project aimed at providing unified standards for collecting and transmitting observability data in distributed systems. Created from the merger of OpenTracing and OpenCensus, it offers:

  • Unified APIs and SDKs: Supporting multiple programming languages for easy observability integration.
  • Automated context propagation: Automatically propagating context information in distributed systems to ensure trace completeness.
  • Multiple backend support: Supporting data transmission to various backends like Prometheus, Jaeger, Zipkin, etc.
  • Extensibility: Supporting custom data collection and processing through plugins and extensions.

Note that OpenTelemetry itself doesn’t provide data storage and analysis capabilities; it’s a standard and toolkit that needs to be used in conjunction with specific observability platforms (like Grafana, Jaeger, Prometheus, etc.). This article demonstrates using OpenObserve as the backend and OpenTelemetry as the data collection tool.

Implementing OpenTelemetry in Go Services

Starting Local OpenObserve Service

After installing Docker locally, run the following command to start the local OpenObserve service:

docker run -v $PWD/data:/data -e ZO_DATA_DIR="/data" -p 5080:5080 \
    -e ZO_ROOT_USER_EMAIL="[email protected]" -e ZO_ROOT_USER_PASSWORD="Complexpass#123" \
    public.ecr.aws/zinclabs/openobserve:latest

Open your browser and visit http://localhost:5080, login using [email protected] and Complexpass#123.

Starting OpenTelemetry Collector Service

Create a new otel-collector-config.yaml file with the following content:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlphttp/openobserve:
    endpoint: http://localhost:5080/api/default
    headers:
      Authorization: Basic bGVpQGNvbm5lY3RseS5haTphMldmRzFTWXlGWXhjUWtp
      stream-name: default

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/openobserve]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/openobserve]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/openobserve]

The OpenTelemetry Collector consists of receivers, processors, and exporters. Receivers accept data, processors handle data, and exporters send data to backends. In this example, the receiver uses the OTLP protocol, the processor uses batch processing, and the exporter sends data to OpenObserve.

Use the following configuration to start the OpenTelemetry Collector service via Docker Compose:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/log/containers:/var/log/containers:ro
    ports:
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
volumes:
  data:

At this point, both OpenTelemetry Collector and OpenObserve services are running and ready for the next step.

Go Code Instrumentation

[Code section remains the same as it’s implementation code]

Summary

This article provides a detailed guide on implementing OpenTelemetry in Go services. Through the three pillars of observability (Metrics, Traces, and Logs), we can comprehensively monitor and understand distributed system operations. OpenTelemetry, as a unified observability framework, not only provides standardized data collection methods but also supports various backend storage solutions.

In the practical section, we demonstrated:

  1. How to set up local OpenObserve and OpenTelemetry Collector environments
  2. How to initialize OpenTelemetry components in Go services
  3. How to use middleware to collect HTTP request metrics
  4. How to implement distributed tracing
  5. How to integrate structured logging

Through these practices, we can build a modern distributed system with complete observability, providing robust support for system monitoring, debugging, and optimization.

References