Elastic Architecture

The LAMP Stack

A classic architectural pattern that is still prevalent today is the LAMP (Linux, Apache, MySQL, PHP) stack. Between applications, things like the programming language or web proxy may change, but the general design remains the same. For many use cases, this setup works reasonably well.

There are problems that typically arise in implementations of this kind of architecture. The most significant problem is that read and write operations in this system are going to compete over resource usage. The next problem is that this pattern tends to leak (as in lose) data. Consider a database column called shipment_status with a value of delayed. If that row is updated to in_transit, the fact that the shipment was delayed at some point is now lost. Another issue with the LAMP stack is that cross-cutting concerns like monitoring or analytics tend to become interwoven with the system, creating additional overhead on resources. Lastly, it's often the case that these systems don't communicate beyond their runtime language, meaning that integrating with these types of systems tends to be a cumbersome process.

A Scaled Version

Organizations working to overcome these architectural shortcomings often leverage cloud providers like AWS (Amazon Web Services) to scale the individual components of their tech stack. A common practice is to scale the application behind a load balancer using an orchestration framework such as AWS EKS, while fanning out replicas of the database in AWS RDS.

This would seem to fix whatever contention there may be between reads and writes on the system. If more read capacity is needed, the system can provision new instances of the application in the auto-scaling group and create additional database replicas.

However, despite the addition of scaling mechanisms, resource contention can still occur in this system. Replication places overhead on the primary database, and during periods of high activity, replication lag can occur as the primary database struggles to write transactions and serve the replay logs. To overcome this, the primary database must be scaled horizontally. Not only is this method of scaling extremely costly, but what is the point of scaling if the primary database will run into horizontal scaling issues anyhow? The underlying problem of coupled reads and writes was never truly resolved.

One last point to touch on with this architecture is the fact that as an application gains functionality, new components will put increased strain on the system. For teams, this means refactoring is a painful experience, requiring trade-offs in the existing system to make room for new features.

An Elastic Alternative

Rather than making incremental improvements to the LAMP architecture, teams can take advantage of resources in AWS and some well-known enterprise application design patterns to create a different kind of architecture capable of scaling to meet requirements in a cost-effective and integration-friendly manner. This post focuses on two patterns: Event Sourcing, and CQRS.

Event-Sourcing

Event sourcing is a pattern where every change to the state of an application is captured as some kind of object, and these objects themselves are stored in the order in which they were applied for the lifetime of the application. This simple concept imparts numerous benefits:

An empty application can be completely rebuilt by loading the event log.
The application state can be determined at any point in time.
Application state can be modified by replaying events from a point in time.

Git, a well-known version control system, is a familiar example of event sourcing. Git stores changes as commits, which represent immutable events in a project's history. A codebase can be rebuilt from nothing by playing all of the project's commits in the order they were applied.

CQRS

CQRS (Command Query Responsibility Segregation) is an application design pattern that separates read and write operations within an application into distinct models. This naturally lends itself to improving performance on most platforms. When paired with the event sourcing pattern, the resulting architecture allows for each component to scale independently and for storage to be optimized for either read or write activity.

Event Sourcing + CQRS

Here is one potential system that employs aspects of event sourcing and CQRS. This could also be called a "Log-Centric" architecture. The central feature of this system is the immutable event log.

The Event Log

The event log is an immutable ledger that stores each event in the order that they occur. This can be accomplished using conventional storage software like Postgres, but more often it is implemented as AWS Kinesis Data Streams, or as an AWS Managed Kafka Cluster.

Write operations sent in are written to the event log. A decoupled command processor picks up messages, processes them, and emits subsequent events back to the log. From there, consumers reading the log see the event and take some action with it, whether that be to aggregate some reports, or transform data into a materialized view.

The System Contract

Decoupling web services from business logic in the application with some kind of intermediate storage is an obvious improvement, but is only one factor in this architecture. The second is the data contract or system language.

System Language Vs. Programming Language

It's easy to think of an application in terms of the language in which it was written. This architecture, however, is capable of remaining language agnostic. The system language exists as data structures that can be created by producers or read by consumers. Here is a very simple example:

Commands & Queries

Commands (writes) and Queries (reads) are both generally known as "Actions" and declared in the present tense. Here is an example Command:

{
  "id": "af523782-8572-49cb-b7ac-4e7a090a44e2"
  "action": "create-shipment"
  "data": {
    "sender": "bob"
    "receiver": "bill"
    "packing-list": [
      "marbles"
      "string"
      "toothpick"
    ]
  }
}

and a corresponding Query:

{
  "id": "b2285ef2-a5ad-412f-a306-9db484d8e692",
  "action": "get-shipment",
  "data": {
    "shipment-id": "af523782-8572-49cb-b7ac-4e7a090a44e2"
  }
}

Events

When a command or event processor emits or receives something, an event is created. Events are declared in the past tense. Below is an example of an Event:

{
  "id": "c53afdee-3b91-4fb4-ba34-ab2aa06b23e7",
  "parent": "af523782-8572-49cb-b7ac-4e7a090a44e2",
  "action": "shipment-created",
  "data": {
    "sender": "bob",
    "receiver": "bill",
    "packing-list": [
      "marbles",
      "string",
      "toothpick"
    ],
    "estimated-arrival": "2025-12-24T00:00:00.000-00:00"
  }
}

Benefits of the System Contract

In the presented examples, a narrative is forming about a shipment. Thanks to the event sourcing aspect of this setup, each event in the lifetime of the shipment will be captured. Integrating with this system is easy. Events can be consumed from the event log using any programming language. Lastly, because everything is centralized on the event log, concerns like analytics or reporting can be performed without adding overhead to resources.

Summary

This architecture is well-suited for growing organizations and enterprises. The system contract ensures that various departments can implement business processes and integrate with each other through simple data structures rather than APIs or SDKs. Event log implementations like Kafka allow for a near infinite number of consumers seeking by offset to read the same record with near O(1) efficiency. Even for small organizations, implementing the system language contract within an application is a great way to anticipate future growth.

Want to Talk?

We've found that the event sourcing and CQRS patterns help customers grappling with the costs of scaling inflection points achieve sustainable growth. If you'd like to have a conversation about how you're approaching growth in your tech stack, or have questions, please reach out to me on LinkedIn or via my email below.