Skip to content

Development Environment

OpenTelemenetry and Jaeger

Local

Docker Compose

Copy the demo from https://opentelemetry.io/docs/collector/getting-started/#demo but only use Jaeger with OpenTelemetry.

cd docker-compose

vim docker-compose.yml
vim otel-collector-config.yml

docker-compose up -d

Brings up Jaeger as backend, and query UI at http://localhost:16686/search

OpenTelemetry Collector exposes the receiver ports:

  • tcp/4317 gRPC
  • tcp/4318 HTTP

Inspect with docker-compose ps to see the port mapping.

Fails with a TLS handshake because there is no TLS.

grpcurl -vv -insecure localhost:49175 helloworld.Greeter/SayHello

export OTEL_EXPORTER_OTLP_ENDPOINT="localhost:49175"


./out/binaries/gitlab-runner --debug run -c $HOME/dev/everyonecancontribute/observability/opentelemetry-gitlab-runner-playground/dev-config.toml

OpenTelemetry Dev

Follow https://opentelemetry.io/docs/collector/getting-started/#local

git clone https://github.com/open-telemetry/opentelemetry-collector-contrib.git 
git clone https://github.com/open-telemetry/opentelemetry-collector.git

cd opentelemetry-collector

make install-tools

make otelcorecol

./bin/otelcorecol_darwin_amd64 --config examples/local/otel-config.yaml

Jaeger AIO

Run it with Docker. It is meant as Tracing receiver and storage, do not care much about the backend type and so on for now.

docker run -d -p6831:6831/udp -p16686:16686 jaegertracing/all-in-one:latest

Open http://localhost:16686/search in your browser.

OpenTelemetry + Jaeger

Today I Learned Building the core OpenTelemtry does not include the Jaeger Exporter. Needs to be built with https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter

The collector deployed in the OpenTelemetry Operator uses an AIO image build.

https://github.com/open-telemetry/opentelemetry-collector-releases

TODO Build the contrib distribution with Jaeger from source for local development.

Kubernetes

Follow this guide to download Civo CLI and setup the personal API token.

Next, create a new Kubernetes cluster.

civo kubernetes create jaeger-otel

civo kubernetes config jaeger-otel --save

kubectl config use-context jaeger-otel

kubectl get node

Jaeger with ClickHouse

Follow the guide in https://github.com/jaegertracing/jaeger-clickhouse/blob/main/guide-kubernetes.md

Jaeger Operator https://www.jaegertracing.io/docs/1.33/operator/#installing-the-operator-on-kubernetes

Requires cert-manager https://cert-manager.io/docs/

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml


kubectl create namespace observability

kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.33.0/jaeger-operator.yaml -n observability

Clickhouse Operator https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml 

Jaeger ClickHouse backend

kubectl apply -f manifests/01-clickhouse.yml 
kubectl apply -f manifests/02-jaeger-clickhouse.yml
kubectl apply -f manifests/03-jaeger.yml

OpenTelemetry

Operator https://github.com/open-telemetry/opentelemetry-operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Configuration

  • Receiver: External client, GitLab Runner, writes into using gRPC or HTTP
  • Exporter: Data storage backends, Jaeger+Clickhouse on port 14250

Example in https://github.com/open-telemetry/opentelemetry-collector/blob/main/examples/k8s/otel-config.yaml

TODO: Additionally, expose port 4317 and 4318 as Service, type LoadBalancer.

kubectl get svc -n observability
kubectl apply -f manifests/04-opentelemetry-collector.yml

Meanwhile, use port-forwarding on localhost.

kubectl port-forward deployment/otel-collector 4317 -n observability
kubectl port-forward deployment/otel-collector 4318 -n observability

Jaeger Query UI

Portforwarding with the Clickhouse deployed Jaeger Service.

Inspired by https://knative.dev/blog/articles/distributed-tracing/

kubectl port-forward svc/jaeger-clickhouse-query 16686
Back to top