Logging, Monitoring and Observability

This video presents an in-depth exploration of the intertwined practices of logging, monitoring, and observability in backend application development. It highlights their individual roles, how they complement each other to enable robust system visibility, and practical implementation insights illustrated with a Go-based to-do application using New Relic.

1. Defining Logging, Monitoring, and Observability

Logging, monitoring, and observability are essential practices for managing complex, distributed backend systems running across multiple servers and regions. Together, they provide visibility into application behavior and infrastructure health, but each serves a distinct purpose:

• Logging: The practice of recording important application events, errors, and metadata throughout the application lifecycle. Logs are akin to a journal or diary of system activities.

• Monitoring: The continuous tracking of system metrics and states (e.g., CPU usage, request rates, database connections) providing near real-time aggregated insights to detect issues and understand system health trends.

• Observability: A broader discipline combining logs, metrics, and traces to allow inference of internal system states by analyzing external outputs. Observability enables not only detecting problems but understanding their root cause across distributed components.

Key Insights on Definitions

• Logging collects discrete event data, including user IDs, request latency, error codes, and relevant context.
• Monitoring delivers aggregated and time-series metrics with slight delay (typically 10-15 seconds) for practical real-time visibility without overwhelming the system.
• Observability rests on three pillars:
  1. Logs (event records),
  2. Metrics (quantitative measures),
  3. Traces (transactional tracking across components).

2. Importance and Context

Modern backend applications operate in distributed environments, often interfacing with front-end systems, load balancers, multiple service layers, and databases. This complexity demands sophisticated tools and practices to:

• Keep track of operations and health across services.
• Detect when issues occur.
• Provide detailed diagnostics to pinpoint failures rapidly.
• Support debugging and incident response effectively.

Traditional monitoring mainly alerts to the presence of a problem but lacks depth to identify causes. Observability enhances this by integrating logs, metrics, and traces to explain what went wrong and why.

3. Practical Breakdown of Logging

Logging Levels

Implementing logging efficiently requires assigning severity levels to logs, which helps in filtering and prioritization:

• Debug logs are verbose and typically disabled in production.
• Info logs capture key successful business actions.
• Warn logs record recoverable or user-caused irregularities.
• Error and Fatal logs are prioritized for immediate attention.

Structured vs Unstructured Logging

• Unstructured Logs: Human-readable plain text logs, often color-coded, ideal for development debugging in local consoles.
• Structured Logs: Logs formatted as JSON or similar, utilized in production to facilitate parsing by log management tools. This enables extracting key metadata automatically (e.g., user ID, request ID).

4. Monitoring Essentials

• Monitoring provides aggregated, time-series data reflecting system performance and operational health.
• It tracks metrics like request throughput, error rates, CPU and memory utilization, database connection counts.
• Though "real-time," monitoring generally works with slight delays (around 10–15 seconds) to prevent system overload.
• Alerts can be generated when metrics breach thresholds (e.g., error rate above 80%), triggering notifications (e.g., Slack).

5. Observability and Tracing

• A trace represents a full transaction journey through components such as frontend, service layers, repositories, and databases.
• Tracing instruments requests end-to-end, showing which components were involved and where failures occurred.
• Observability tools combine logs, metrics, and traces to deliver actionable diagnosis beyond simple alerts.

6. Integrating the Three Practices: Workflow Example

• Alerts inform about abnormal behavior (e.g., high error rate).
• Metrics dashboards show trends and aggregate system state.
• Logs provide detailed event data corresponding to alerts.
• Traces reveal transactional flow for selected problem cases to identify failure points.

This integrated approach accelerates debugging and improves understanding of complex systems.

7. Implementation Demonstration Using Go and New Relic

Setup Highlights

• The demo backend is a Go-based to-do app.
• New Relic is used as a proprietary one-stop solution for logging, monitoring, and observability.
• Middleware instruments incoming requests creating transactions attached to context, capturing attributes like request ID, user ID, IP address.
• Logging respects environment configurations:
  • Debug level and console-format logs in development.
  • Info level and JSON-structured logs in production.

Logging in Practice

• Key events logged include:
  • Starting to create a to-do item.
  • Validation checks.
  • Database operations and errors with detailed metadata.
  • Success confirmations with debug logs (visible only in development).

Monitoring and Observability

• Middleware instruments requests to collect trace data.
• Metrics like throughput, error rates, average transaction time appear on New Relic dashboards.
• Errors and logs correlate interactively, facilitating inspection and drill-down.
• Traces visualize request paths spanning multiple layers, enabling root cause analysis of issues.

Tools Overview

• OpenTelemetry is emphasized as a critical standard enabling consistent instrumentation across languages and systems.

8. Additional Context: Deployment Platform Sponsor

• The video sponsors Seala, a platform as a service (PaaS) alternative to Heroku, Vercel, and Netlify.
• Seala supports:
  • Deploying full-stack apps and databases.
  • Dockerized observability tools with internal networking.
  • Kubernetes-based infrastructure with Google Cloud and Cloudflare edge network.
  • Preview deployments on pull requests for streamlined team collaboration.
  • Multiple build options, including Nx Build Packs, Heroku compatibility, and Dockerfile-based.
• Pricing claims 33% less bandwidth cost compared to Vercel, with promotional credits offered.

9. Core Best Practices and Developer Considerations

• Logging, monitoring, and observability exist on a spectrum—no system is 100% perfect.
• Effective observability requires collective effort:
  • Developers need to implement instrumentation, logging, and context propagation.
  • DevOps must maintain infrastructure and tooling for capturing and analyzing data.
• Choosing between open source stacks (Grafana ecosystem) and proprietary solutions (New Relic, Datadog) depends on team resources and complexity.
• The process enables faster debugging, improved reliability, and better operational insights in production environments.

Summary Table: Comparison of Key Concepts

Key takeaway: Implementing a combined logging, monitoring, and observability practice tailored to your stack's needs is crucial for maintaining robust backend systems in distributed environments. These practices ensure you not only detect but also diagnose and resolve issues effectively.