Day 23: OpenTelemetry with GitLab CI/CD | Traces & Metrics (Hands-on Project)

Welcome to Day 23 of the 30 Days of GitLab DevOps Course. In this comprehensive session, we explore OpenTelemetry (OTel)—an industry-standard observability framework—and learn how to integrate it seamlessly into a Python Flask application deployed via GitLab CI/CD. Best of all, we leverage GitLab's built-in observability backend, eliminating the need to configure, host, or maintain third-party monitoring tools like Prometheus or Jaeger manually.

1. Why Do We Need OpenTelemetry? (The Problem Statement)

In modern microservices and distributed cloud environments, understanding what happens inside a running application can feel like searching for a needle in a haystack.

The Status Quo & Challenges

• Black Box Execution: When an API request fails or slows down, developers often have to comb through raw log files across multiple servers.
• Fragmented Monitoring Tools: Traditionally, engineering teams had to install, configure, and maintain separate infrastructure for metrics (e.g., Prometheus), traces (e.g., Jaeger, Zipkin), and logs (e.g., ELK stack).
• Vendor Lock-in: Proprietary APM (Application Performance Monitoring) agents force teams to use specific vendor formats, making it difficult to switch backends.

Solution: OpenTelemetry

OpenTelemetry solves this by providing a single, open-source, vendor-agnostic standard for generating, collecting, and exporting telemetry data (Traces, Metrics, and Logs) straight from your application source code to any compatible observability backend.

2. Core Mental Model: Understanding Telemetry Data

Real-Life Analogy: The Restaurant Kitchen and Delivery Workflow

Think of an application as a busy restaurant kitchen:

• Metrics tell you the high-level statistics: How many orders came in today? What was the average wait time? How much oven capacity (CPU) is currently being utilized?
• Traces tell you the exact journey of a single customer order: Customer places order at the front register $\rightarrow$ Ticket sent to grill station $\rightarrow$ Food plated by chef $\rightarrow$ Handed to delivery driver. If an order is delayed, the trace shows precisely which station caused the bottleneck.
• Logs are the specific logbook notes written by kitchen staff along the way: Warning: Grill temperature running high, or Error: Out of burger buns.

3. The Three Pillars of OpenTelemetry Data

OpenTelemetry standardizes three distinct types of telemetry data:

4. OpenTelemetry Architecture & Components

OpenTelemetry is not a monitoring backend or dashboard itself; rather, it is a framework that instruments your code and exports standardized data.

Component Breakdown

1. OpenTelemetry SDK: Software Development Kit embedded within the application source code. It collects data, creates spans, records metrics, and captures exceptions.
2. OTLP Exporter: Takes the raw telemetry data from the SDK, packages it into the standard OpenTelemetry Protocol (OTLP), and transmits it over HTTP to a configured endpoint.
3. Observability Backend (GitLab): Receives, validates, stores, correlates, and indexes telemetry data so it can be queried instantly through GitLab's native UI.

5. Step-by-Step Hands-On Implementation Workflow

We will build, containerize, and deploy a Python Flask application on an AWS EC2 instance using a GitLab CI/CD pipeline, connecting it directly to GitLab's native observability backend.

Step 5.1: Provisioning the AWS EC2 Target Machine

1. Launch an AWS EC2 instance of type t3.medium running Ubuntu 24.04 LTS.
2. Configure storage: 25 GB EBS volume.
3. Configure Security Group: Ensure ports 22 (SSH), 5000 (Flask App), and incoming/outgoing ranges are properly configured.
4. Connect to the EC2 instance via SSH (using MobaXterm or terminal):
   BASH

   Copy

   sudo apt update -y
   

Step 5.2: Installing Docker on EC2

Run the following commands to install Docker and grant permissions to the ubuntu user:

# Install Docker dependencies and repository
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo "deb [arch=$(dpkg --arch amd64) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/source.list.d/docker.list > /dev/null

sudo apt update -y
sudo apt install docker-ce docker-ce-cli containerd.io -y

# Add ubuntu user to docker group
sudo usermod -aG docker ubuntu
newgrp docker
docker --version

6. Developing the Instrumented Flask Application

Inside our project repository, we write a Python Flask application (app.py) that initializes OpenTelemetry tracers, metrics, and exporters.

Application Code Structure (app.py)

import time
from flask import Flask, jsonify
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource

app = Flask(__name__)

# Configure Resource attributes for correlation in GitLab
resource = Resource.create({
    "service.name": "my-flask-app",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

# Setup Tracing
trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)
span_exporter = OTLPSpanExporter()  # Endpoint injected via env variable at runtime
span_processor = BatchSpanProcessor(span_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

@app.route("/")
def home():
    with tracer.start_as_current_span("home-endpoint-span"):
        return jsonify({"message": "Hello from Instrumented Flask App!"})

@app.route("/slow")
def slow_request():
    with tracer.start_as_current_span("slow-endpoint-span"):
        time.sleep(3)
        return jsonify({"message": "Completed slow request after 3s"})

@app.route("/error")
def trigger_error():
    with tracer.start_as_current_span("error-endpoint-span") as span:
        try:
            raise ValueError("Database connection failure simulation")
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            return jsonify({"error": str(e)}), 500

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Supporting Dockerfile

Create a Dockerfile in your repository root to containerize the Python application:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

Python Dependencies (requirements.txt)

flask
opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-http

7. Configuring GitLab Project & CI/CD Variables

To enable automated deployment and secure communication with our EC2 instance, we must configure CI/CD variables in GitLab.

Adding GitLab CI/CD Variables

1. Navigate to your GitLab Project -> Settings -> CI/CD -> Variables.
2. Add the following variables:
   • EC2_IP: The public IPv4 address of your EC2 instance (Masked: true).
   • SSH_PRIVATE_KEY: The content of your .pem private key file used to SSH into EC2 (Visible/Unmasked due to multiline formatting restrictions).

Enabling GitLab Observability Backend

1. Go to your GitLab Group level -> Observability -> Setup -> Click Enable.
2. Copy the generated OTLP HTTPS Endpoint URL (e.g., https://observe.gitlab.com/v1/traces?...). We will inject this URL into our deployment pipeline.

8. Writing the GitLab CI/CD Pipeline (.gitlab-ci.yml)

Our pipeline consists of two stages: Build (which packages the application into a Docker tar archive) and Deploy (which connects to our EC2 instance, loads the image, and runs the container with OTLP environment variables).

stages:
  - build
  - deploy

variables:
  IMAGE_NAME: flask-otel-app

build_image:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Building Docker image..."
    - docker build -t $IMAGE_NAME .
    - echo "Saving Docker image to tar archive..."
    - docker save $IMAGE_NAME -o ${IMAGE_NAME}.tar
  artifacts:
    paths:
      - ${IMAGE_NAME}.tar
    expire_in: 1 hour

deploy_to_ec2:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - ssh-keyscan -H "$EC2_IP" >> ~/.ssh/known_hosts
  script:
    - echo "Copying Docker image tar to EC2..."
    - scp ${IMAGE_NAME}.tar ubuntu@$EC2_IP:/home/ubuntu/
    - echo "Executing deployment commands on EC2..."
    - ssh ubuntu@$EC2_IP "
        docker load -i /home/ubuntu/${IMAGE_NAME}.tar &&
        docker stop flask-app || true &&
        docker rm flask-app || true &&
        docker run -d --name flask-app \
          -p 5000:5000 \
          -e OTEL_EXPORTER_OTLP_ENDPOINT='https://your-gitlab-observability-endpoint-url' \
          $IMAGE_NAME
      "

9. Verification, Testing & Monitoring in GitLab UI

Once the CI/CD pipeline executes successfully:

1. Access your application in the browser at http://<EC2_IP>:5000.
2. Generate synthetic traffic to populate telemetry data:
   • Hit the root endpoint (/) multiple times for standard requests.
   • Hit /slow to test latency tracking (sleeps 3 seconds).
   • Hit /error to simulate an unhandled exception.

Viewing Telemetry in GitLab

1. Navigate to your GitLab project -> Monitor -> Traces.
2. Filter by service name (my-flask-app) to inspect request timelines, span durations, and error details.
3. Navigate to Monitor -> Metrics to explore system performance charts, request counts, and latency histograms.

10. Summary and Key Takeaways

• OpenTelemetry Definition: OpenTelemetry is an open-source, vendor-agnostic observability framework that standardizes the generation and export of traces, metrics, and logs.
• Not a Backend: OTel does not store data or provide dashboards itself; its job is strictly to instrument code and export standardized payloads.
• The SDK Component: The OpenTelemetry SDK collects telemetry data inside the application runtime (e.g., spans, metrics, exceptions).
• The OTLP Exporter: Translates and transmits telemetry data using the OpenTelemetry Protocol (OTLP) over HTTP to an observability backend.
• Native GitLab Integration: GitLab provides a built-in observability backend, allowing developers to monitor applications without installing third-party tools like Prometheus or Jaeger.
• Resource Attributes: Configuring metadata such as service.name, service.version, and deployment.environment enables GitLab to correlate telemetry back to specific projects and deployments.
• Pipeline Automation: Using Docker-in-Docker (dind) alongside secure SSH artifact transfers allows seamless deployment of instrumented containers directly to production EC2 hosts.