Transactions Per Second (TPS)

Transactions Per Second (TPS)

The Corda team are involved in a number of projects globally that are looking at how they can ultimately scale their solution to perform at a level demanded by real world use cases. A key criterion in scaling up any blockchain platform is the number of transactions the system can manage. This is top of mind in our engineering team, in particular for our dedicated performance team. We’d like to share our current work and thinking in this area.

Let’s get right to it with a snapshot of throughput per node we are getting internally at R3 right now as we measure and tune our forthcoming R3 Corda distribution. There is much still to be done but we wanted to share progress to date in our test labs with two specific scenarios.

1678 TPS & 170 TPS

Defining ‘transaction’

Before we explain these figures, we’d like to put the notion of measuring transactions per second into context. What are the different ways to represent TPS?

• What a network can achieve. First generation blockchain systems are constrained at the network level because a large part of the network participates in transactions.
• What a node can achieve. For a specific configuration and transaction how many transactions can a node participate in during a single second.
• The duration of a transaction. For a single transaction how long does it take to span all nodes from inception to finality.

In stark contrast to first generation systems, the network figures for Corda will increase as you add more nodes. Corda does not require a shared consensus mechanism across a large percentage of nodes. It is a point-to-point system so as peers transact it does not affect non-participants to those transactions. Hooray!

The key constraints that we worry about on the Corda team are the nodes ability to transact in many transactions in a set duration, and the duration of a transaction.

1678 TPS. This figure has been obtained for a type of transaction that takes place on a single node demonstrating the performance the flow state machine can obtain. This transaction type is for an issuance of a new state to the ledger of the local machine.

170 TPS. This figure is for a complete transaction from the perspective of a node. A payment with two parties, as an example. We believe this is a more realistic assessment of measuring what matters in a Corda network.

These are real transactions and constitute real measurements of performance in terms of TPS. Of course, we are working feverishly to keep pushing the 1678 and 170 TPS figures higher and have begun this work in earnest following our 1.0 release.

Making comparisons

The reality is TPS is only relevant when compared against a transaction of a similar nature on an identical set of constraints. Ultimately, what we are seeking broadly is an aggregate view of how many transactions a single node is capable of participating in across a variety of transaction types.

The various distributed ledger technologies differ in how they define a transaction. For Ethereum and Fabric the transaction is measured from the submission of a contract to the network for consensus and the adding of the transaction to a block. In contrast, Corda has three characteristics that are unique and must be understood in order to determine how to effectively measure performance and, importantly, how to compare it to other platforms.

• Transaction building
• Transaction finality
• Consensus algorithm

Flows are a component of the Corda Platform not found in other systems. The purpose of the framework is to build a transaction between parties. We recognised that building the transaction is a common technical challenge that developers were faced with continually. With no consistent and agreed upon framework in place developers solve this same problem over-and-over again. Corda tackles this problem directly in the platform by abstracting away the challenges of maintaining state across systems, obtaining and verifying signatures and obtaining data in a deterministic manner.

Corda transactions are final at the time of signature by the service responsible for consensus over uniqueness of the transaction inputs. Corda networks operate this service, known as Notary Pools, as a set of distrusting entities operating in consensus. This separation is what allows Corda to to operate as a point-to-point system while retaining all the blockchain characteristics. In contrast other systems require consensus across nodes and transactions are not final until sufficient block depth is achieved.

The Notary Pools are able to use ‘pluggable’ algorithms for consensus. While a pool must consistently use a chosen algorithm, they may initially select any algorithm that has been established for use in the Corda platform. This means they may select RAFT and BFT currently, or alternate algorithms that may be offered by other vendors.

With these unique characteristics in mind we believe the following are key variables that will affect how you measure performance:

• Transaction building. To compare to other platforms the transaction would need to be built and measured from the point of submission to the Notary Pool.
• Number of parties in a transaction. If measuring with the transaction builder, t
  he complexity of the transaction increases with the number of parties involved.
• The consensus algorithm must be comparable. Among the tradeoffs in selecting an algorithm performance is the main variable.
• The transaction should be defined as validated or non-validated. This should be consistent across platforms.

Consistent variables

Unlike the more mature spaces of relational databases or virtualisation the DLT space lacks a standard definition of what a transaction is and how to compare across platforms. In the database realm this is the responsibility of an independent organisation, TPC. TPC has a standard measurement for comparing the performance of database platforms against a well-defined set of constraints including a consistent data set, a consistent query in ANSI standard SQL run against a consistent hardware and operating system configuration. Without such consistency the measurement would be unscientific in nature as it would be unable to isolate performance issues to the database engine vs. other variables.

Our internal testing is against a consistent hardware and network configuration. As a part of our performance work we’ll be publishing this specification along with the specifics of the transaction properties. We are working against a defined set of variables held consistent for accurate measurement:

• Transaction payload: for a validated transaction the definition of the payload and business logic must be consistent. This consists of a consistent set of data fields with defined types and length equating to a measurement in kilobytes.
• A consistent business logic based on the validation logic for the transaction. This should be provided in pseudocode and may be optimised on each platform.
• A consistent hardware and network environment.
• Latency between the participant and other nodes must be consistent if measuring overall network performance (vs. single node performance).
• CPU performance, memory and other hardware characteristics should be consistent.
• The operating system must be consistent, and the system should be clear of any auxiliary software (antivirus etc).

For our test noted above we used Microsoft’s Azure infrastructure running SQL Azure. Specifically, we used the VM standard D32 v3 (32 vcpus, 128 GB memory) machine and 4000 DTU database (P15). For future published benchmarking operations, we shall denote the specific hardware and environment in order to ensure we can control all variables.

Comparing Corda to other platforms

R3 believes organisations such as Hyperledger are well placed, and should take a leadership role, in working toward a consistent definition and measurement similar to the aforementioned TPC. In the early stages of evaluating this metric, however, care must be taken to ensure the comparison is done in a like-for-like manner. We offer the following standard for consideration on comparing Corda’s performance to alternate platforms. The following is how you should measure a transaction on Corda to be able to make a comparison:

• Start should commence from the point at which the party that is building the transaction (contract or other terms) does its own verification and submits the transaction to the Notary Pool for a uniqueness check. This matches platforms without a transaction building capability.
• Finish should be regarded as the point at which the transaction has been acknowledged as being added to the record which is used for assurance of double spend (the ledger).
• All variables such as the transactions, hardware and network properties should be consistent.

While we have seen published claims for TPS performance we have not discovered how those transactions are measured on alternate platforms, or what they believe constitutes a transaction. But currently Corda measures between 15 and 1678 TPS depending on how it is measured. This is contrast to other platforms:

• Bitcoin peaks out at around 7 TPS
• Ethereum is limited to around 15 TPS
• Quorum has stated a target of “dozens to hundreds of transactions per second depending on configuration” in their white paper.
• Hyperledger Fabric. As Chris Ferris noted “We haven’t published performance figures for Fabric because there isn’t a standard benchmark.”. A recent blog posting noted Fabric achieves 3500 TPS. We believe this measurement is a view of the total network transaction and incomparable as Corda does not have this constraint.

Conclusion

Organisations that are viewing TPS in isolation are doing themselves a disservice. They should view throughput as one variable in the overall performance of a system. Other characteristics such as resiliency, ability to make a trade-off in performance for assurance are as critical if not more so. A highly performant system that has little privacy and no high availability configuration would be an unbalanced trade-off that would introduce risk into the overall
system.

As we lead up to our upcoming releases of Corda, in both the open source and R3’s commercial distribution, we’ll be publishing more statistics around performance. Our vision of building a network capable of supporting some of the largest workloads that amount to billions of daily transactions puts performance as a key target for our own engineering efforts. While we continue to push Corda forward in performance we are also collaborating with IBM and others in the Hyperledger working groups to drive an industry standard for performance evaluation and general principles for performance. Look for more postings from our performance team on our blog to follow our progress.