Day 24 | GitLab CI/CD Logging & Slack Alerts: Real-Time Pipeline Notifications

1. Introduction and Overview of DevOps Observability

In modern DevOps workflows, running a CI/CD pipeline and deploying code to production is only half the battle. The other critical half is Observability—knowing instantly whether your builds succeeded, understanding why they failed, and monitoring application runtime behavior without needing manual Secure Shell (SSH) access into every server.

This chapter covers two vital pillars of observability within GitLab CI/CD:

1. Logging Mechanisms: Capturing pipeline execution logs, system-level logs, and real-time application runtime logs (from containerized services).
2. Alerts and Notifications: Integrating external communication platforms (such as Slack) via webhooks to receive real-time, event-driven pipeline notifications.

2. The Three Layers of DevOps Logging

When troubleshooting issues in a production or staging environment, logs do not exist in a vacuum. They span multiple architectural tiers.

• Pipeline Logs: Automatically generated by GitLab runners. Every script command executed within .gitlab-ci.yml is captured in the GitLab job UI.
• Application Runtime Logs: Generated by the running application itself (e.g., standard output/error streams of a Flask app running inside a Docker container). These require a custom extraction mechanism or log forwarder to view inside CI/CD.
• System Logs: Low-level operating system records stored on virtual machines (e.g., AWS EC2 instances) under /var/log/syslog or /var/log/auth.log, critical for diagnosing kernel panics, security intrusions, or daemon failures.

3. The Three Layers of Alerts and Notifications

To maintain high availability and rapid incident response, teams configure tiered alerting systems:

4. Setting Up the Target EC2 Virtual Machine

To host our demonstration Flask application, we provisioned an AWS EC2 instance.

EC2 Configuration Specifications

• Name Tag: app
• AMI (Operating System): Ubuntu 24.04 LTS
• Instance Type: t3.medium
• Security Group Rules:
  • Standard SSH (Port 22) for remote administration.
  • Custom TCP (Port 5000) open to allow inbound traffic to our Flask application.
• Root Volume: 20 GB General Purpose SSD (gp3).

Connecting and Preparing the Instance via MobileXterm / Terminal

Once launched, we connected via SSH using our private key and installed Docker, ensuring the non-root ubuntu user could execute Docker commands without sudo:

# Update local package lists
sudo apt update

# Verify Docker installation and permissions
docker ps

5. Configuring GitLab CI/CD Secure Variables

To allow our automated GitLab CI/CD pipeline to securely connect to our EC2 instance via SSH and deploy code, we stored sensitive credentials as project variables.

1. Navigate to GitLab Project -> Settings -> CI/CD -> Variables.
2. Add the following variables:
   • VM_IP: The public IPv4 address of the EC2 instance (e.g., 3.85.120.45).
   • SSH_PRIVATE_KEY: The contents of the .pem private key file used to access the EC2 instance.

6. Developing the Application Source Code

We created a lightweight Python Flask application consisting of a single route returning a status message, accompanied by dependency and containerization configurations.

app.py (Flask Application)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello from GitLab logging lab!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

requirements.txt

Flask==3.0.2

Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "app.py"]

7. Writing the Multi-Stage GitLab CI/CD Pipeline (.gitlab-ci.yml)

The pipeline is structured into three distinct stages: build, deploy, and logs.

stages:
  - build
  - deploy
  - logs

variables:
  IMAGE_NAME: flask-app

build_image:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Building Docker image..."
    - docker build -t $IMAGE_NAME .
    - docker save $IMAGE_NAME > myapp.tar
  artifacts:
    paths:
      - myapp.tar
    expire_in: 1 hour

deploy_app:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - ssh-keyscan -H "$VM_IP" >> ~/.ssh/known_hosts
  script:
    - echo "Transferring Docker tarball to EC2 VM..."
    - scp myapp.tar ubuntu@$VM_IP:/home/ubuntu/myapp.tar
    - echo "Deploying container on EC2..."
    - ssh ubuntu@$VM_IP "
        docker load -i /home/ubuntu/myapp.tar &&
        (docker stop flask-container || true) &&
        (docker rm flask-container || true) &&
        docker run -d --name flask-container -p 5000:5000 $IMAGE_NAME
      "

fetch_logs:
  stage: logs
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - ssh-keyscan -H "$VM_IP" >> ~/.ssh/known_hosts
  script:
    - echo "Fetching application runtime logs from EC2 container..."
    - ssh ubuntu@$VM_IP "docker logs flask-container" > logs.txt
    - cat logs.txt
  artifacts:
    paths:
      - logs.txt

8. Setting Up Slack Workspaces and Incoming Webhooks

To move beyond default email notifications, we integrated Slack to receive real-time pipeline status updates.

Step-by-Step Slack Configuration:

1. Create a Workspace: Navigate to Slack, click Get Started, and create a free workspace named 30-Days-Of-GitLab.
2. Create an Alert Channel: Inside the workspace, create a new public channel named #gitlab-alerts.
3. Register a Slack App:
   • Go to api.slack.com/apps and click Create New App -> From scratch.
   • App Name: GitLab-Notifier, select your workspace, and click Create App.
4. Enable Incoming Webhooks:
   • In the left sidebar, click Incoming Webhooks and toggle the switch to On.
   • Click Add New Webhook to Workspace, select the #gitlab-alerts channel, and click Allow.
   • Copy the generated Webhook URL (e.g., https://hooks.slack.com/services/T00/B00/X1Y2...).

9. Integrating GitLab Webhooks with Slack

To link GitLab pipeline events to our Slack webhook URL:

1. In your GitLab repository, navigate to Settings -> Webhooks.
2. Click Add new webhook.
3. Configure the webhook settings:
   • URL: Paste the Slack Webhook URL.
   • Trigger Events: Select Pipeline events (uncheck unnecessary events to reduce spam).
   • SSL verification: Enabled.
4. Click Add webhook.

10. End-to-End Pipeline & Alert Verification Workflow

When code is pushed to the repository, the following automated sequence executes:

11. Summary and Key Takeaways

• Multi-Layered Logging: Comprehensive observability requires capturing Pipeline logs (GitLab execution UI), Application logs (docker logs), and System logs (/var/log/).
• Artifact Passing: Using artifacts: paths: allows compiled binaries or Docker tarballs (myapp.tar) to pass seamlessly between isolated pipeline stages.
• Remote Container Orchestration: Alpine runners can securely connect to remote EC2 instances via SSH keys stored in GitLab CI/CD secure variables to manage Docker lifecycles.
• External Webhooks: Slack Incoming Webhooks transform silent CI/CD workflows into real-time collaborative event streams for engineering teams.
• Pipeline Event Filtering: Limiting webhook triggers specifically to Pipeline events ensures that Slack channels receive clean, formatted notifications covering pending, running, success, and failure states.