Implement a distributed transaction in microservices software system using Saga pattern

1. Context and problem

With relational databases, we usually find the concept of transaction. Transaction is a single unit of work which may contain multiple operations that are ensured to be all occur or none occur.

Transactions should have ACID properties: atomic, consistent, isolated, and durable. Within a single service, transactions are usually easy to be ACID. When it comes to cross-service, data consistency requires a cross-service transaction management strategy.

In a distributed system/multi-services architecture:

• Atomicity is a single unit with set of operations that must all occur or none occur.
• Consistency means data should only move from one state to another expected valid state in all participants.
• Isolation ensures that concurrent transactions result in the same outcome when running sequentially.
• Durability makes the commit status of transactions persistent and be able to bear with failure related to system or power outage.

Microservices architecture usually takes the approach of database-per-microservice model. It provides many benefits with data isolation, independency and scaling ability. However, ensuring data consistency across different databases is a challenge.

Let’s take an example of the order processing logic:

• Normal flow:

1. Client: Submit checkout order

2. Order API: Save order to database

3. Account API: Create transaction and deduct account balance

4. Payment API: Save payment to database and request external payment

5. Order API: Mark order as completed

• Failures:

1. If Saving order fails

• Order API: Order will be mark as failed

2. If Creating transaction or deducting account balance fails

• Order API: Order will be mark as failed

3. If Saving payment or requesting external payment fails

• Account API: Reverse transaction and increase account balance
• Order API: Order will be mark as failed

4. If Marking order as completed fails

• Payment API: Mark payment as failed and request refund payment
• Account API: Reverse transaction and increase account balance
• Order API: Order will be mark as failed

The above workflow is quite complex and requires a distributed transaction coordination logic for the local transactions to execute and compensating actions to run in case of failures. More steps can be added by the time.

Two-phase commit (2PC) protocol, which is also a technique to create a distributed transaction, requires all participants in a transaction to commit or roll back before the transaction is considered as completed. However some participant mechanisms, e.g, NoSQL databases, message brokering, file storage, etc, don’t support this model.

Distributed transaction also has some limitations with synchronicity and availability since it needs all participants to be available. Therefore we have another candidate, the Saga pattern.

2. Saga — The solution

The Saga pattern is a transaction management strategy involves with multiple local transactions. A local transaction is the atomic work performed by a saga participant. Each local transaction makes the changes, e.g, update database, do some external actions, and publishes message or event to trigger the next transaction in the whole saga. If some local transaction fails, the saga will execute a series of specified compensating transactions that reverse/compensate the changes that were made before by the local transactions.

In Saga patterns:

• Compensable transactions: can potentially be undo by executing another transaction with the opposite effect.
• A pivot transaction: is the go/no-go point in a saga. If the pivot transaction commits, the saga runs the remaining transactions until done. We can tell the saga is successful if a pivot transaction commits. A pivot transaction can be neither compensable nor retryable, or it can be the last compensable transaction or the first retryable transaction in the saga.
• Retryable transactions: are transactions that follow the pivot transaction and are guaranteed to succeed, e.g, in ordering application, we have finished and prepared everything we need for the order such as payment, products, delivery man/vehicle, etc, and we just need to deliver products to our customers (which is retryable)
• Transaction key: each transaction should have a unique key across services in the whole system. Transaction key is used to retrieve the transaction’s status and information so coordination logic can be applied to correct targets. Entities identifier can be used as a key or they can be linked with a generated key.

There are two common saga implementation approaches: choreography and orchestration. Each has its own set of challenges and techniques to manage the workflow.

3. Set up demo projects

Introduction to the demo project:

• Full source code: trung-tran-sts/simple-saga (github.com)
• Technical: C#, .NET 6
• Libraries: RxNet
• Development tools: VS Code, Visual Studio or any IDE with .NET integration
• Branches:
• choreography-based: Choreography-based saga implementation
• orchestration-based: Orchestration-based saga implementation

Overview context:

1. There are three services with their own local transactions.

2. Client will submit action that triggers the execution of transaction 1.

3. Subsequent messages/events will be fired on status changes to trigger the other transactions.

4. There is a pivot transaction followed by a retryable transaction.

5. If there are failures, messages/events will be fired to trigger corresponding compensating transactions.

…

Read more at: https://saigontechnology.com/blog/implement-a-distributed-transaction-in-microservices-software-system-using-saga-pattern