Development Environment¶
OpenTelemenetry and Jaeger¶
Local¶
Docker Compose¶
Copy the demo from https://opentelemetry.io/docs/collector/getting-started/#demo but only use Jaeger with OpenTelemetry.
Brings up Jaeger as backend, and query UI at http://localhost:16686/search
OpenTelemetry Collector exposes the receiver ports:
- tcp/4317 gRPC
- tcp/4318 HTTP
Inspect with docker-compose ps
to see the port mapping.
Fails with a TLS handshake because there is no TLS.
export OTEL_EXPORTER_OTLP_ENDPOINT="localhost:49175"
./out/binaries/gitlab-runner --debug run -c $HOME/dev/everyonecancontribute/observability/opentelemetry-gitlab-runner-playground/dev-config.toml
OpenTelemetry Dev¶
Follow https://opentelemetry.io/docs/collector/getting-started/#local
git clone https://github.com/open-telemetry/opentelemetry-collector-contrib.git
git clone https://github.com/open-telemetry/opentelemetry-collector.git
cd opentelemetry-collector
make install-tools
make otelcorecol
./bin/otelcorecol_darwin_amd64 --config examples/local/otel-config.yaml
Jaeger AIO¶
Run it with Docker. It is meant as Tracing receiver and storage, do not care much about the backend type and so on for now.
Open http://localhost:16686/search in your browser.
OpenTelemetry + Jaeger¶
Today I Learned Building the core OpenTelemtry does not include the Jaeger Exporter. Needs to be built with https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter
The collector deployed in the OpenTelemetry Operator uses an AIO image build.
https://github.com/open-telemetry/opentelemetry-collector-releases
TODO Build the contrib distribution with Jaeger from source for local development.
Kubernetes¶
Follow this guide to download Civo CLI and setup the personal API token.
Next, create a new Kubernetes cluster.
civo kubernetes create jaeger-otel
civo kubernetes config jaeger-otel --save
kubectl config use-context jaeger-otel
kubectl get node
Jaeger with ClickHouse¶
Follow the guide in https://github.com/jaegertracing/jaeger-clickhouse/blob/main/guide-kubernetes.md
Jaeger Operator https://www.jaegertracing.io/docs/1.33/operator/#installing-the-operator-on-kubernetes
Requires cert-manager https://cert-manager.io/docs/
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.33.0/jaeger-operator.yaml -n observability
Clickhouse Operator https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
Jaeger ClickHouse backend
kubectl apply -f manifests/01-clickhouse.yml
kubectl apply -f manifests/02-jaeger-clickhouse.yml
kubectl apply -f manifests/03-jaeger.yml
OpenTelemetry¶
Operator https://github.com/open-telemetry/opentelemetry-operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
Configuration
- Receiver: External client, GitLab Runner, writes into using gRPC or HTTP
- Exporter: Data storage backends, Jaeger+Clickhouse on port 14250
Example in https://github.com/open-telemetry/opentelemetry-collector/blob/main/examples/k8s/otel-config.yaml
TODO: Additionally, expose port 4317 and 4318 as Service, type LoadBalancer.
Meanwhile, use port-forwarding on localhost.
kubectl port-forward deployment/otel-collector 4317 -n observability
kubectl port-forward deployment/otel-collector 4318 -n observability
Jaeger Query UI¶
Portforwarding with the Clickhouse deployed Jaeger Service.
Inspired by https://knative.dev/blog/articles/distributed-tracing/