Building Reactive Systems

The Reactive Manifesto

On September 16, 2014, Jonas Bonér and his colleagues published The Reactive Manifesto - a document defining the core characteristics of a Reactive system.

“

Reactive Systems are systems that are highly responsive, elastically scalable, fault-tolerant, and built on message-driven architecture.

The 4 Pillars of Reactive Systems

Relationship: Message Driven → (Resilient + Elastic) → Responsive

1. Responsive

Systems must respond in a timely manner if at all possible. Responsiveness is the foundation of usability.

Characteristics of Responsive Systems:

“

Key insight: Responsive is the result of properly applying Resilient and Elastic.

2. Resilient

Systems must remain responsive when failures occur. Any system that is not resilient will be unresponsive after failure.

Resilience is achieved through:

3. Elastic

Systems must remain responsive under varying workload. Can increase or decrease resources based on demand.

Requirements for Elasticity:

4. Message Driven

Reactive Systems rely on asynchronous message-passing to establish boundaries between components.

Benefits of Message-Driven:

• Loose coupling - Components don't depend directly on each other
• Isolation - Clear boundaries between components
• Location transparency - No need to know physical location
• Error delegation - Errors are passed as messages
• Back-pressure - Flow control when overloaded

Commands vs Events

Non-Blocking I/O with Netty

The Problem with Blocking I/O

Netty Architecture

Netty Architecture by Layer:

“

Key insight: With Netty, 1 Event Loop thread can manage thousands of connections thanks to non-blocking I/O and multiplexing.

Resilience Patterns

When building payment systems at MoMo, we applied the following patterns:

1. Retry Pattern

Purpose: Retry operations when transient failures occur.

“

Exponential Backoff: Wait time doubles after each failure

Best Practices:

• Use exponential backoff (1s → 2s → 4s → 8s)
• Limit retry attempts (max 3-5)
• Distinguish between retryable and non-retryable errors

2. Circuit Breaker Pattern

Purpose: Prevent continuous calls to a failing service.

Circuit Breaker State Flow:

3. Rate Limiter Pattern

Purpose: Control the number of requests within a time period.

Algorithms:

4. Bulkhead Pattern

Purpose: Isolate parts of the system so failures don't spread.

Like ship compartments:

• Compartments (bulkheads) are separated
• Water entering one compartment doesn't sink the entire ship

Application with Bulkhead Pattern:

“

Each pool is completely isolated - if Bank Integration is overloaded, Payment and User still work normally.

5. Fallback Pattern

Purpose: Provide alternative values when primary operation fails.

Examples:

• Return cached data when database is unavailable
• Use default values when external service is down
• Redirect to backup service

6. Timeout Pattern

Purpose: Set time limits to avoid waiting indefinitely.

Real Experience at MoMo

Applied Architecture:

1. Message-Driven Architecture:

• Apache Kafka as message broker
• Event Sourcing for transaction history
• CQRS to separate read/write operations

2. Resilience Patterns:

• Circuit Breaker for bank integrations (30+ banks)
• Retry with exponential backoff
• Bulkhead to isolate critical payment flows

3. Non-Blocking I/O:

• Vert.x for core services
• Reactive streams with back-pressure
• Connection pooling with non-blocking drivers

Results Achieved:

Key Takeaways

1. Reactive is not just a technical choice - It's an architecture decision affecting the entire system design
2. 4 pillars must be applied together - Missing one reduces overall effectiveness
3. Resilience patterns are mandatory - In distributed systems, failure is normal, not an exception
4. Message-driven is the foundation - Enables loose coupling and location transparency
5. Non-blocking I/O is the technical enabler for high throughput

References

• The Reactive Manifesto
• Release It! - Michael Nygard
• Designing Data-Intensive Applications
• Vert.x Documentation