Day-11 | Python For DevOps | Real-Time Use Case with Dictionaries & Sets

In modern DevOps engineering, interacting with external APIs, cloud providers, and configuration management tools is a daily requirement. This chapter explores two fundamental Python data structures—Dictionaries and Sets—focusing on how they are used to parse complex JSON payloads from APIs like GitHub, manage cloud resource configurations, and handle unique datasets.

1. The "Why" Behind Dictionaries: Limitations of Strings and Lists

To understand why dictionaries are indispensable, we must analyze the limitations of the data types we have learned so far: Strings, Integers, and Lists.

Scenario: Managing Student or Cloud Resource Data

Suppose we want to store information about a student or an AWS EC2 instance.

• Using Strings: If we only need to store a single attribute, such as an instance name, a string is sufficient:
  PYTHON

  Copy

  ec2_instance_name = "instance-001"
  

• Using Lists: If we want to store a collection of names, a list works perfectly:
  PYTHON

  Copy

  student_list = ["Abishek", "Ram", "Tom"]
  

The Problem: Storing Complex Properties

What if we need to store multiple, diverse properties of a single entity? For example, an EC2 instance has:

• An Instance ID (i-0123456789abcdef0)
• An Instance Type (t2.micro)
• A Public IP Address (54.210.23.41)
• A State (running)

The Naive Approach (Using Lists)

We could attempt to store these properties inside a standard Python list:

ec2_instance_properties = ["instance-001", "i-0123456789abcdef0", "t2.micro", "54.210.23.41"]

Why This Fails in Production

1. Lack of Context: Looking at index 1 or 2 after several days or in a large codebase makes it impossible to know what the value represents without external documentation or comments.
2. Brittle Code: If the order of elements in the list changes, any code relying on positional indexes (e.g., ec2_instance_properties[2]) will break or return incorrect data.
3. Scalability Issues: If you have 100 properties, managing them by index position is highly error-prone.

2. The Core Mental Model: Key-Value Pairs

To solve the limitations of lists for structured data, Python provides Dictionaries. Dictionaries store data in Key-Value Pairs enclosed in curly braces {}.

Real-Life Analogy

Think of a traditional physical dictionary. To find the definition (the Value), you look up the specific word (the Key). The key is a unique identifier that points directly to its associated value.

Syntax and Structure

In a dictionary, keys and values are separated by colons (:), and pairs are separated by commas (,).

# Declaring a dictionary representing student properties
student_info = {
    "name": "Abishek",
    "age": 10,
    "class": "11th"
}

Accessing Elements

Unlike lists, where elements are accessed using sequential integer indexes (0, 1, 2...), dictionary elements are accessed directly using their Keys:

# Accessing the value associated with the key "name"
print(student_info["name"])  # Output: Abishek

This approach is self-documenting. Anyone reading student_info["class"] immediately understands what data is being retrieved.

3. Advanced Structures: Lists of Dictionaries

In real-world DevOps scenarios, you rarely deal with just one resource. You will manage lists of EC2 instances, lists of Kubernetes pods, or lists of GitHub users. To represent this, we combine lists and dictionaries into a List of Dictionaries.

Code Evolution: Representing Multiple EC2 Instances

# Storing multiple EC2 instances, each with its own structured properties
ec2_instances_info = [
    {
        "id": "instance-001",
        "type": "t2.micro"
    },
    {
        "id": "instance-002",
        "type": "t2.medium"
    },
    {
        "id": "instance-003",
        "type": "t2.xlarge"
    }
]

Navigating Nested Structures

To access a specific property of a specific instance, we combine list indexing with dictionary key lookup:

# Step 1: Access the second element in the list (index 1)
second_instance = ec2_instances_info[1]  # Returns: {"id": "instance-002", "type": "t2.medium"}

# Step 2: Access the "type" key of that dictionary
instance_type = second_instance["type"]  # Returns: "t2.medium"

# Combined syntax:
print(ec2_instances_info[1]["type"])  # Output: t2.medium

4. Real-World DevOps Context: Why Dictionaries Dominate API Integrations

As a DevOps engineer, you write scripts to automate interactions with platforms like AWS, Jira, GitHub, and Slack.

Cloud/SaaS API (AWS, GitHub, Jira)
       |
       | Sends HTTP Response (JSON Payload)
       v
Python Script (requests module)
       |
       | Converts JSON to Native Python Dictionary
       v
Data Manipulation & Automation Logic

The Role of JSON

Almost all modern web APIs return data in JSON (JavaScript Object Notation) format. JSON is structurally identical to Python's dictionary format (using key-value pairs).

To perform operations on this data, Python scripts:

1. Query the API.
2. Receive the JSON payload.
3. Convert the JSON payload into a native Python dictionary (or a list of dictionaries).
4. Extract, filter, or modify the data.

5. Step-by-Step Evolution: Interacting with the GitHub API

Let's build a real-time Python script that connects to the public GitHub API to fetch and parse Pull Request (PR) information for the open-source Kubernetes repository.

The Algorithm

1. Install and Import Dependencies: Use Python's popular requests library to handle HTTP communication.
2. Define the Target URL: Identify the correct GitHub API endpoint for pull requests.
3. Execute the HTTP GET Request: Fetch the data from GitHub.
4. Convert JSON to Dictionary: Parse the raw response into a Python-readable format.
5. Extract and Process Data: Loop through the data to print specific fields (e.g., PR creator usernames).

Step 1: Install and Import the requests Library

Run the following command in your terminal to install the library:

pip install requests

Import it at the top of your Python file:

import requests

Step 2: Define the API Endpoint

The GitHub API endpoint structure for listing pull requests is: https://api.github.com/repos/{owner}/{repo}/pulls

For the Kubernetes repository, the URL is:

url = "https://api.github.com/repos/kubernetes/kubernetes/pulls"

6. Deep Dive into the Code: Parsing Nested JSON Responses

Below is the complete Python script to fetch the pull requests and print the GitHub username of each PR creator.

import requests

# Define the API endpoint for Kubernetes Pull Requests
url = "https://api.github.com/repos/kubernetes/kubernetes/pulls"

# Send an HTTP GET request to the GitHub API
response = requests.get(url)

# Verify if the request was successful (HTTP Status Code 200)
if response.status_code == 200:
    # Convert the JSON response into a Python list of dictionaries
    pull_requests = response.json()
    
    # Iterate through the list of pull requests
    for i in range(len(pull_requests)):
        # Extract the nested username: pull_requests[i] -> "user" -> "login"
        creator_username = pull_requests[i]["user"]["login"]
        pr_title = pull_requests[i]["title"]
        
        print(f"PR #{i+1}: '{pr_title}' created by {creator_username}")
else:
    print(f"Failed to fetch data. HTTP Status Code: {response.status_code}")

Understanding the Nested Traversal

The JSON response returned by GitHub is a list of dictionaries. Inside each dictionary, there is a key named "user", whose value is another dictionary containing user details. One of the keys inside the "user" dictionary is "login".

To access the username, we chain the keys:

pull_requests[i]["user"]["login"]

7. Real-World Debugging & API Response Handling

When working with APIs in production, things can fail. Understanding how to inspect and debug API responses is critical.

Common HTTP Status Codes

• 200 OK: Request succeeded.
• 403 Forbidden: Often indicates API rate limiting or missing authentication headers.
• 404 Not Found: The repository or endpoint URL is incorrect.
• 500 Internal Server Error: The remote server encountered an error.

Debugging Techniques

If your script is not returning the expected data, inspect the response metadata:

response = requests.get(url)

# 1. Check the status code
print(f"Status Code: {response.status_code}")

# 2. Check the raw response object type
print(f"Response Type: {type(response)}")  # Output: <class 'requests.models.Response'>

# 3. Convert and inspect a slice of the data
data = response.json()
if isinstance(data, list) and len(data) > 0:
    print("Sample keys in first element:", data[0].keys())

8. Introduction to Sets: The Unique Element Constraint

A Set is an unordered collection of items where every element must be unique.

Syntax

Sets are declared using curly braces {} but without key-value pairs:

# Declaring a set
my_set = {"Abishek", "Ram", "Tom", "Abishek"}

print(my_set)  # Output: {'Abishek', 'Ram', 'Tim'} (Duplicates are automatically removed)

Real-World DevOps Analogy: S3 Bucket Names

AWS S3 bucket names must be globally unique. If you are writing a script to collect and validate a list of proposed S3 bucket names before creation, storing them in a Set ensures that any accidental duplicate entries are instantly filtered out.

proposed_buckets = {"my-app-logs", "my-app-assets", "my-app-logs"}
print(proposed_buckets)  # Output: {'my-app-assets', 'my-app-logs'}

9. Mathematical Set Operations in DevOps Workflows

Sets are highly efficient for mathematical operations such as Union, Intersection, and Difference. These operations are useful when comparing configurations, firewall rules, or package lists across different environments (e.g., Dev vs. Prod).

Set A (Dev Packages)             Set B (Prod Packages)
   [ nginx, git, curl ]             [ nginx, docker, curl ]
            \                             /
             \                           /
              v                         v
               Intersection: [ nginx, curl ]

1. Intersection (Common Elements)

Find elements that exist in both sets.

dev_tools = {"nginx", "git", "curl"}
prod_tools = {"nginx", "docker", "curl"}

common_tools = dev_tools.intersection(prod_tools)
print(common_tools)  # Output: {'nginx', 'curl'}

2. Union (All Unique Elements Combined)

Combine elements from both sets, removing duplicates.

all_tools = dev_tools.union(prod_tools)
print(all_tools)  # Output: {'nginx', 'git', 'docker', 'curl'}

3. Difference (Unique to One Set)

Find elements that are in the first set but not in the second.

only_in_dev = dev_tools.difference(prod_tools)
print(only_in_dev)  # Output: {'git'}

10. Synthesizing Python Data Structures

To choose the right tool for the job, use this comparison of Python's primary collection data types:

Summary and Key Takeaways

• Dictionaries are essential for APIs: Modern cloud and SaaS APIs return JSON data, which maps directly to Python dictionaries.
• Key-Value pairs provide context: Unlike lists, which rely on positional indexes, dictionaries use descriptive keys to access data, making code self-documenting and resilient to change.
• Nested data traversal is a core skill: Accessing nested API data requires chaining indexes and keys (e.g., response_data[0]["user"]["login"]).
• The requests library simplifies HTTP: Using requests.get(url).json() handles both the network request and the JSON-to-dictionary conversion in one step.
• Sets enforce uniqueness: Sets automatically discard duplicate values, making them ideal for managing unique identifiers like S3 bucket names or UUIDs.
• Set operations simplify comparisons: Use intersection, union, and difference to quickly compare resources, configurations, or package versions across environments.