What is event-driven architecture?
Use events to trigger decoupled services and promote the production and consumption of events. Event-driven architectures contain:
- event producers: publish to the router
- event routers: filter and push to consumers, as a broker that might truncate the contents of a stream when it overflows with records
- event consumers: subscribe to the router
In fact, producers of event records often have no knowledge of who the consumers might be, nor whether consumers exist at all.
What is an event?
- An event is a change in state, or an update
- an item being placed in a shopping cart on an e-commerce website
- dispatching an order
- a product review submission
- Event:
- state: the item purchased, its price, and a delivery address
- identifiers: a notification that an order was shipped
{
"orderId": "760b5301-295f-4fec-95f8-6b303a3b824a",
"customerId": 28623823,
"productId": 31334,
"quantity": 1,
"timestamp": "2021-02-09T11:12:17+0000"
}Streams are persistent sequences of records. They are typically backed by one or more disk-based logs under the hood. Equally, streams might be backed by database tables, a distributed consensus protocol, or even a blockchain-style decentralized ledger.
What is an event broker/router?
A middleware to handle everything between producers and consumers, and it should be:
- highly performant
- fault-tolerant
Why do we implement event-driven architecture?
Scale and fail independently
- Producer and consumers are independent, so a consumer does not block a producer
- It is easy to increase/decrease the number of producers and consumers
- The router acts as an elastic buffer that accommodate surges in workloads
- Buffering and fault-tolerance while production and consumption are in different rates
Audit with ease
- Only need to define topics that support pub/sub
- Easy to define policies and permission
- Encrypt events
Develop with agility
- No need to write codes for the router
Cut costs
- On-demand
- Not paying for continuous polling
- Less network bandwidth
- Less CPU utilization
- Less idle fleet capacity
- Less SSL/TLS handshakes
What about the consumers? I suppose the consumers still require continuous polling, right?
Why don't we use event-driven architecture?
Limited to asynchronous processing
- Might take too long for request-response interactions
Introduces additional complexity
- A new broker/middleware to mediate the interactions between producers and consumers
Failure masking
- More difficult to debug
- Require monitoring and alerts, which increases complexity
How an event-driven architecture work?
When to use event-driven architecture?
Most of the reasons are similar to microservices.
Cross-account, cross-region data replication
- Distributed services/systems
Fanout and parallel processing
- Multiple inputs to one output
- One event is processed by multiple consumers
- Multiple services/systems need to respond to one event
Resource state monitoring and alerting
- Track down changes in each state
- Easy to debug/monitor when all the monitoring/alerting are in place
Integration of heterogeneous systems
- Decoupled so easy to integrate different stacks/tools
Command-query responsibility segregation (CQRS)
- Separate read and write operations
Opaque consumer ecosystem
- Producers are unaware of consumers
Should you use an event-driven architecture?
- The event source should be reliable and guarantee delivery if you need to process every single event
- The application should be able to handle async events
- You are able to dynamically track/monitor the system
- eg. the system is distributed, so you might not be able to find the event in a specific consumer when that event is consumed by others
- If you need to rebuild state, the event source should be deduplicated and ordered
Different styles of event processing
Discrete event processing
The presence of events that are generally unrelated to one another and may be handled independently, so it is called discrete.
Event stream processing
- Order matters
- Consumers need to avoid race conditions
Kafka uses record keying and partitions to preserve the order of updates.
Complex event processing (CEP)
- More involved
- Require keeping track of prior events
- Need an efficient way to query and aggregate data
An example of CEP might be monitoring a group of temperature and smoke sensors in a building to infer that a fire has broken out and to track its progress. Individual temperature changes might not be sufficient to raise an alert; however, the clustering of temperature spikes and the rate of change may provide more meaningful insights that could ultimately save lives.