Day 14: DevOps Project from Absolute Zero — Project Setup, CI Pipelines Implementation & Azure DevOps

1. Overview & Project Objective

In this session, we execute a complete, end-to-end DevOps project starting from absolute zero. We take a multi-service microservices application—originally written by the Docker team—and migrate its source code and CI pipelines from GitHub to the Microsoft Azure DevOps ecosystem.

Our goal is to build an independent Continuous Integration (CI) pipeline for each custom microservice using Azure Pipelines, store our container images in Azure Container Registry (ACR), and run our builds using a self-hosted Linux agent hosted on an Azure Virtual Machine.

2. The Multi-Service Microservice Architecture

The application provided by the development team is a polling/voting application designed for container and microservices practice. It consists of five distinct components spanning three different programming languages and two data backends.

Microservice Breakdown Table

Note: Databases and in-memory stores do not require custom CI pipelines because we do not write source code for them; we simply pull and deploy their standard official Docker images.

3. Understanding Continuous Integration (CI) in Enterprise Teams

In modern software engineering, repositories often host hundreds of pull requests and commits from dozens of distributed developers. Without automation, verifying changes manually becomes impossible.

Real-Life Analogy: The Quality Control Assembly Line

Imagine an automobile factory where 33 different engineers are designing and modifying different parts of an engine simultaneously. If one engineer bolts on a new fuel injector directly without testing it, the whole car could catch fire on the highway.

Instead, factories use an automated testing assembly line. Every time a mechanic tightens a bolt, the engine is automatically put on a test bench to check pressure, electrical wiring, and fuel flow before it is ever approved for final installation. Continuous Integration (CI) is that exact automated test bench for software code.

Core Stages of a CI Pipeline

When a developer makes a commit or opens a pull request (PR), the CI system automatically executes:

1. Trigger Check: Detects changes in the repository branch or specific directory paths.
2. Unit Testing: Runs automated test suites to verify individual code functions.
3. Static Code Analysis: Scans code for syntax errors, smells, and vulnerabilities.
4. Application Build: Compiles source code or packages dependencies.
5. Docker Image Creation: Packages the application and its runtime into a container image using a Dockerfile.
6. Artifact Push: Uploads the built container image to a secure artifactory (in our case, Azure Container Registry).

4. Containerizing Microservices: Writing Dockerfiles

Before writing Azure pipelines, we must understand how each custom microservice is containerized. Each Dockerfile sits at the root of its respective microservice folder.

A. Result Microservice (Dockerfile - Node.js)

FROM node:18
RUN apt-get update && apt-get install -y tiny curl
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 80
CMD ["npm", "start"]

B. Voting Microservice (Dockerfile - Python)

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 80
CMD ["python", "app.py"]

C. Worker Microservice (Dockerfile - .NET)

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /source
COPY *.csproj .
RUN dotnet restore
COPY . .
RUN dotnet publish -c Release -o /app

5. Setting Up Azure DevOps Project & Importing Repositories

To execute our CI pipelines entirely within Azure DevOps, we perform the migration from GitHub to Azure Repos.

Step-by-Step Migration Workflow

1. Create Azure DevOps Organization & Project: Navigate to dev.azure.com, create an organization (help-devops-aspirants), and create a new private project named voting-application.
2. Import GitHub Repository:
   • Go to Repos -> Files -> Click Import a repository.
   • Paste the public GitHub repository URL (https://github.com/dockersamples/example-voting-app).
   • Click Import.
3. Fix Default Branch:
   • Azure DevOps imports branches alphabetically by default. Navigate to Repos -> Branches, locate the main branch, click the three dots (...), and select Set as default branch.

6. Creating Azure Container Registry (ACR)

We need an artifactory to store our built Docker images. We create an Azure Container Registry inside Microsoft Azure.

1. Create Resource Group:
   BASH

   Copy

   az group create --name azure-cicd --location eastus
   

2. Create Container Registry:
   BASH

   Copy

   az acr create --resource-group azure-cicd --name abishekazurecicd --sku Standard
   

7. Configuring a Self-Hosted Linux Agent on Azure VM

Azure DevOps requires compute runners to execute build pipelines. Because free-trial accounts do not include managed Microsoft-hosted agents, we provision our own Self-Hosted Linux Agent on an Azure Virtual Machine.

A. Provisioning the Virtual Machine

Create an Ubuntu Linux VM inside the same resource group (azure-cicd) and name it azure-agent:

az vm create \
  --resource-group azure-cicd \
  --name azure-agent \
  --image Ubuntu2204 \
  --admin-username azure-user \
  --generate-ssh-keys

B. Installing and Registering the Agent

1. Create Agent Pool in Azure DevOps:

   • Go to Project Settings -> Agent pools -> Add pool -> Name it Azure-Agent (Self-hosted).

2. Configure Agent on the VM via SSH: SSH into the VM using Git Bash or your terminal:

   BASH

   Copy

   ssh -i azure-agent_key azure-user@<VM-PUBLIC-IP>
   

   Run the setup commands provided by Azure DevOps:

   BASH

   Copy

   mkdir myagent && cd myagent
   wget https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-linux-x64-3.225.0.tar.gz
   tar zxvf vsts-agent-linux-x64-3.225.0.tar.gz
   ./config.sh
   

3. Provide Configuration Inputs:

   • Server URL: https://dev.azure.com/help-devops-aspirants
   • Authentication type: PAT (Personal Access Token generated from Azure DevOps user settings with full access).
   • Agent pool: Azure-Agent
   • Agent name: azure-agent

4. Install Docker on the Agent VM:

   BASH

   Copy

   sudo apt-get update
   sudo apt-get install -y docker.io
   sudo usermod -aG docker azure-user
   sudo systemctl restart docker
   

   Note: Log out and log back in via SSH so the Docker group permissions take effect. Verify Docker execution: docker pull hello-world.

5. Start the Agent:

   BASH

   Copy

   ./run.sh
   

8. Implementing Path-Based Multi-Pipeline CI in Azure DevOps

To prevent building all microservices when a change occurs in only one, we implement Path-Based Triggers. Each microservice gets its own independent YAML pipeline (azure-pipelines-result.yml, azure-pipelines-voting.yml, azure-pipelines-worker.yml).

Complete Azure Pipeline YAML (azure-pipelines-result.yml)

trigger:
  branches:
    include:
      - main
  paths:
    include:
      - result/*

pool:
  name: Azure-Agent

variables:
  dockerRegistryServiceConnection: 'abishek-acr-connection'
  imageRepository: 'resultapp'
  dockerfilePath: '$(Build.SourcesDirectory)/result/Dockerfile'
  tag: '$(Build.BuildId)'

stages:
  - stage: Build
    displayName: Build stage
    jobs:
      - job: Build
        displayName: Build Job
        steps:
          - task: Docker@2
            displayName: Build Docker Image
            inputs:
              command: build
              repository: $(imageRepository)
              dockerfile: $(dockerfilePath)
              containerRegistry: $(dockerRegistryServiceConnection)
              tags: |
                $(tag)
                latest

  - stage: Push
    displayName: Push stage
    dependsOn: Build
    jobs:
      - job: Push
        displayName: Push Job
        steps:
          - task: Docker@2
            displayName: Push Docker Image to ACR
            inputs:
              command: push
              repository: $(imageRepository)
              containerRegistry: $(dockerRegistryServiceConnection)
              tags: |
                $(tag)
                latest

9. Real-World Debugging & Troubleshooting Mindset

Common Pitfalls and Resolutions

10. Summary and Key Takeaways

• Microservices CI Isolation: Independent microservices require decoupled, independent CI pipelines so that modifying one service does not unnecessarily trigger builds for others.
• Path-Based Triggers: Using YAML paths.include filters ensures that CI pipelines execute only when files within a specific source folder change.
• Self-Hosted Agents: When free-trial accounts or security policies restrict Microsoft-hosted runners, provisioning an Azure Linux VM as a self-hosted agent gives us full control over build dependencies and tools.
• Containerization as Build Standard: Modern CI builds applications inside container images using Dockerfile instructions rather than legacy artifact packaging (.jar, .war).
• End-to-End Migration: Successfully migrating source code and pipelines from GitHub to Azure DevOps establishes foundational enterprise DevOps capabilities.
• Artifactory Integration: Connecting Azure Pipelines to Azure Container Registry (ACR) securely stores versioned container artifacts ready for downstream Continuous Delivery (CD).