Bastion Servers Deep Dive (System Design for Beginners – Episode 47)

As cloud architectures mature, securing critical infrastructure components like databases and internal microservices requires placing them inside isolated virtual private clouds (VPCs) and private subnets. However, legitimate operational needs—such as debugging production crashes, running database migrations, or checking logs—require authorized engineers to access these isolated resources without exposing the internal network to the public internet. This chapter explores bastion servers (also known as jump hosts or jump servers), examining their architecture, operational workflows, configuration patterns, and modern zero-trust alternatives.

1. The Security Paradox of Private Subnets

When an organization moves its database or backend systems to the cloud, standard security best practices dictate that critical assets must be placed inside a Virtual Private Cloud (VPC) and isolated within a private subnet.

• No Public Exposure: The database possesses no public IP address, features no inbound or outbound routes to an internet gateway, and remains entirely invisible and unreachable from the public internet.
• Firewall Isolation: A strict security group acts as a firewall, configured exclusively to accept inbound traffic on specific ports (e.g., port 3306 for MySQL) solely from designated application servers within the same VPC.
• The Operational Dilemma: At 3:00 AM, an application crashes, a database migration is required, or a customer issue must be debugged. Because the database is completely locked down, engineers cannot connect to it directly from their laptops. Opening the database up to the public internet or office IP addresses undoes all security hardening.

2. Core Mental Model: The Bastion Host as a Single Controlled Doorway

To resolve the security paradox without sacrificing operational accessibility, architects deploy a bastion host (or jump server).

Real-Life Analogy: Think of a high-security subterranean vault containing priceless assets. You do not want the vault door exposed directly to a busy public street where anyone can walk up and hammer on it. Instead, you build a single, heavily guarded security checkpoint (the bastion host) at the perimeter. Authorized personnel must enter through this single checkpoint, verify their identity with strict multi-factor authentication, and be escorted directly to their destination inside the vault.

Public Internet (Laptops / Admins)
   |
   | (1) SSH with Key + MFA (Public IP)
   v
[ Bastion Host / Jump Server ] (Public Subnet)
   |
   | (2) Internal Private Network Route
   v
[ Private Database / App Server ] (Private Subnet, No Public IP)

3. Bastion Host vs. Normal Server: Key Differences

A bastion server is structurally and operationally distinct from standard application servers.

4. Bastion Host vs. VPN: Why Choose a Bastion?

When designing administrative access into a private cloud network, engineers often debate between using a Virtual Private Network (VPN) and a bastion host.

• The VPN Drawback: Once a VPN connects an engineer's laptop to the internal network, it typically grants broad network-level access across the entire VPC. This violates the principle of least privilege, makes fine-grained session auditing difficult, and exposes the entire internal network if a single developer laptop is compromised.
• The Bastion Advantage:
  1. Shrinks Attack Surface: Only one hardened machine is exposed to the internet rather than granting network-wide access to dozens of workstations.
  2. Simplified Control: Security policies, SSH keys, MFA requirements, and IP allow-lists are centralized onto a single target that is tractable to harden.
  3. Straightforward Auditing: Every entry point and administrative action flows through a single centralized chokepoint, making log analysis and compliance tracking trivial.

5. Infrastructure Layer Placement & Traffic Paths

In a well-designed backend architecture, resources are segregated into two distinct security zones:

Understanding the Two Separate Traffic Paths

1. User Traffic Path: External end-users hit the Load Balancer/API Gateway -> requests route to application servers -> application servers query the private database. Bastion hosts are never involved in this path.
2. Admin Access Path: Operations engineers connect to the Bastion host -> from the bastion, they connect to private servers for maintenance, debugging, and migrations.

6. Step-by-Step Bastion Workflow

1. Step 0 (Private Target Isolation): The target database sits securely inside a private subnet with no public IP.
2. Step 1 (Initial Connection): An engineer uses their laptop to initiate an SSH or RDP connection to the bastion host's public IP address.
3. Step 2 (Authentication & Authorization): The bastion validates cryptographic SSH keys and enforces Multi-Factor Authentication (MFA). IAM policies and server-side rules verify authorization (what the user is permitted to touch).
4. Step 3 (Internal Hop / Tunneling): From the bastion host, the engineer establishes a secondary connection or tunnel directly to the private target resource.
5. Step 4 (Isolated Execution): All traffic beyond the bastion rides exclusively on internal VPC routing tables, completely invisible to the public internet.

7. Operational Usage Patterns

Depending on workflow requirements, engineers use three primary connection patterns when interacting with bastion architectures.

Pattern A: Manual SSH Jump

The engineer logs into the bastion, establishes a remote shell, and then runs a second SSH command from inside the bastion to reach the private target.

• Command Flow: Laptop -> SSH -> Bastion Host -> SSH -> Private Server
• Characteristics: Requires two separate logins, two terminal windows, and two authentication challenges every time.

Pattern B: Proxy Jump Pattern (-J)

This pattern collapses the multi-step process into a single command executed directly from the local machine, while SSH transparently handles the intermediary hop in the background.

ssh -J user@bastion-host-ip user@private-target-ip

• Characteristics: Single local command invocation, secure under-the-hood tunneling, and seamless authentication without maintaining an active shell on the bastion.

Pattern C: Local Port Forwarding (GUI Tooling)

When an engineer does not want a remote terminal shell, but instead wishes to use a local graphical database client (e.g., MySQL Workbench, pgAdmin) on their laptop against a private cloud database:

ssh -L 3306:private-database-host:3306 user@bastion-host-ip

• Workflow: The local database GUI connects to localhost:3306. Under the hood, SSH tunnels the traffic securely through the bastion host directly to the private database. To the local application, it appears as though the database is running locally on the engineer's machine.

Local Laptop (MySQL Workbench)
   |
   | connects to localhost:3306
   v
SSH Tunnel via Bastion Host
   |
   | forwards packets through internal network
   v
Private Database (Port 3306)

8. Multi-Hop Enterprise Network Topologies

In complex enterprise environments, security segmentation requires chaining multiple bastion servers together:

Using the proxy jump pattern, engineers can chain multiple hosts cleanly into a single command:

ssh -J user@bastion1-ip,user@bastion2-ip user@private-target-ip

9. Clarification: Bastion Hosts vs. Load Balancers vs. API Gateways

Engineers frequently confuse edge components. Each serves an entirely different audience and protocol:

10. Real-World Enterprise Implementation: Netflix Architecture

At scale, companies like Netflix run their production workloads inside private AWS VPCs that remain completely isolated from public access.

• Zero Direct Access: Even internal employees and engineers have no direct network access to production instances.
• MFA Integration: Before opening any session, an engineer must authenticate through an identity provider like Okta for rigorous Multi-Factor Authentication.
• Role-Based Access Control: Security groups and AWS IAM policies tie access directly to employee roles.
• Session Auditing: Every session is fully logged and replayable, enforcing Zero Standing Access—the security principle where no engineer has persistent privileges until the exact moment of authorized need.

11. Modern Alternatives to Traditional Bastion Servers

While traditional SSH bastion hosts remain foundational, modern cloud ecosystems offer advanced paradigms that decouple network location from access authorization:

• AWS Systems Manager (SSM) Session Manager: Allows engineers to connect to instances that possess no public IP addresses and have port 22 (SSH) entirely closed. Connections occur securely via AWS APIs tied directly to IAM user identities.
• Google Identity-Aware Proxy (IAP): Verifies user identity and context (e.g., corporate credentials, device posture) before permitting access to resources, bypassing traditional network perimeter checks.
• Zero-Trust Platforms (e.g., Teleport): Modern open-source and enterprise access planes built explicitly to replace legacy SSH bastions with identity-based access, certificate-based authentication, and automated audit logging.

12. Summary and Key Takeaways

• Private Subnet Isolation: Placing critical assets like databases inside private subnets with no public IP addresses is a fundamental cloud security best practice.
• The Operational Paradox: Secure infrastructure still requires administrative access for debugging, migrations, and incident response without punching holes in firewalls.
• Bastion as a Chokepoint: A bastion host acts as a single, hardened, monitored entry point into private networks, dramatically shrinking the attack surface.
• Least Privilege vs. VPN: Unlike VPNs that grant broad network-level access, bastion hosts provide tightly controlled, auditable access to specific target resources.
• Traffic Segregation: Differentiate clearly between user traffic paths (Load Balancer -> App Servers) and admin access paths (Laptop -> Bastion -> Private Target).
• Flexible Access Patterns: Support multiple operational workflows ranging from manual SSH hops and proxy jumps (-J) to secure local port forwarding (-L) for GUI database tools.
• Multi-Hop Scalability: Complex enterprise networks use chained bastion hops to traverse corporate zones and project boundaries safely.
• Clear Infrastructure Boundaries: Never confuse bastion hosts (for admins) with load balancers (for users) or API gateways (for API traffic routing).
• Modern Evolution: Cloud-native tools like AWS SSM Session Manager, Google IAP, and zero-trust access planes (Teleport) are modernizing traditional bastion architectures by prioritizing identity over network location.