Validations and transformations for backend engineers

This video provides a comprehensive overview of data validations and transformations in backend API design, emphasizing their critical roles in ensuring data integrity, security, and maintaining clean architecture layers. It explains where validations fit into the API lifecycle, different types of validations, transformation processes, and the distinction between frontend and backend validations.

1. Backend API Architecture and Validation Placement

Validations and transformations are crucial in backend API workflows to prevent unexpected states or errors caused by malformed or malicious data.

• Backend architecture is structured into three layers:

  • Repository Layer: Handles all database-related operations (queries, insertions, deletions). This layer abstractly interacts with relational or NoSQL databases and persistent storages.
  • Service Layer: Implements business logic, coordinating repository calls, triggering notifications, sending emails, or webhooks.
  • Controller Layer: Interfaces with HTTP requests, calling service methods and returning responses to clients. It focuses on HTTP-level behavior like response codes and formatting.

• Validations and transformations occur at the entry point of the controller layer, immediately after routing matches the request to a controller method, but before any business logic is executed or service layer is called.

• Reasons for this placement:

  • Ensures incoming data (JSON payload, query parameters, path parameters, headers) adheres to expected formats.
  • Prevents propagation of bad data to business logic and database.
  • Allows returning meaningful 400 Bad Request errors instead of 500 Internal Server Errors caused by unexpected data downstream.

2. Validation Objectives and Examples

Validation confirms that incoming data is correct in structure, type, and semantic meaning, based on predefined schemas or business requirements.

• Example: API expects a JSON payload with a name field:

  • Must exist (required).
  • Must be a string.
  • String length between 5 and 100 characters.

• Without validation, if the client sends, e.g., an integer instead of a string, the pipeline fails late at the database insertion stage, causing a 500 server error.

• With validation:

  • The API detects the error at the entry point.
  • Returns a 400 Bad Request error with specific message explaining why the input is invalid.
  • This enhances client experience and system robustness.

• Validation is often implemented using a middleware or reusable utility that checks JSON against schemas.

3. Types of Validation

The video categorizes validation into three primary types, each serving distinct purposes:

Key points:

• Syntactic validations confirm structure but not meaning.
• Semantic validations enforce business logic constraints.
• Type validations ensure type consistency, e.g., no string where a number is expected.

4. Transformation: Adjusting Input Data for Consistency

Transformation happens alongside validation within the same pipeline and involves modifying input data into a usable or consistent format before further processing.

• Common scenario: Query parameters are always received as strings by servers, but business logic may expect numbers.
• Example:
  • Client sends page=2 and limit=20 as query parameters (strings).
  • The server casts these strings to numbers to run range validations (page > 0 && page < 500, limit > 0 && limit < 10,000).
• Transformation can also standardize formats, e.g.:
  • Convert emails to lowercase.
  • Prefix phone numbers with + sign.
  • Reformat date strings.

Benefits of merging validation and transformation in one pipeline:

• Keeps all data integrity logic centralized and reusable.
• Avoids scattered data handling logic across the codebase.
• Ensures consistent input format for the service layer.

5. Practical API Validation Examples

• Syntactic validation example: An API expecting email, phone, date fields rejects empty or improperly formatted inputs with descriptive errors.
• Semantic validation example:
  • Rejects a date of birth set in the future.
  • Rejects an unrealistic age such as 430 years.
• Complex conditional validations:
  • Password and password confirmation fields must match.
  • If married is true, then the partner field becomes required.
• Type validation example:
  • Fields explicitly validated as number, string, boolean, or array with type checks on individual array elements.

6. Frontend vs Backend Validations

• Frontend (client-side) validation aims for user experience (UX):

  • Immediate feedback on invalid input.
  • Minimizes unnecessary API calls.
  • Does not guarantee security or data integrity.

• Backend (server-side) validation ensures security and data integrity:

  • Protects API from malicious or incorrect data from any client.
  • Handles all API clients, including tools like Postman, Insomnia, or mobile apps that have no frontend validation.
  • Always mandatory and independent of frontend validation presence.

Important takeaway:

Backend validation cannot be replaced by frontend validation and must be strict and comprehensive regardless of frontend implementation.

7. Summary of Validation and Transformation Workflow

8. Key Insights and Conclusions

• Validations and transformations safeguard APIs from invalid or malicious data inputs, improving security and reliability.
• Validation must be performed at the server entry point before executing any business logic.
• Three main types of validation—syntactic, semantic, and type—cover format correctness, logical consistency, and type accuracy respectively.
• Transformations help convert input data into formats expected downstream, avoiding common errors like treating query strings as numbers.
• Backend validation is not optional and must coexist with frontend validation for robust API design.
• Validation errors provide implicit API documentation by specifying required fields and their constraints.
• Complex conditional validation rules can be encoded in the validation pipeline to enforce relational dependencies in user data.

This structured approach to validations and transformations ensures an API is both user-friendly and resilient, providing clear error messages to clients while protecting internal services and databases from erroneous states.