Corda 5 “The road ahead” – Part 2 – 5 new concepts in Corda 5

In the last post we talked about some of the observations that got us to rethink aspects of the platform and finished on the objectives we set out to accomplish in defining the new major platform version.

This post is a high-level summary of the how, introducing some of the big new concepts we are working on as part of the development of Corda 5 to address those takeaways and meet the objectives in the previous post.

We’ll get a chance to dive deeper into some of these new concepts in the following posts of the series, but this one should give you the birds-eye view of the direction of 5.

Corda 5 is still in active development and we still have a lot to do on our way to our first GA with this new architecture, and I am sure plenty of opportunities to improve & tweak the direction slightly, so your feedback is welcome.

In short, this is what is new in Corda 5:

• We are introducing application networks and new application packages.
• We are improving on developer experience with new modular APIs & developer tooling.
• We are moving to an event driven architecture for availability and scalability.
• We are introducing virtual nodes for multi-network and multi-tenancy support.
• We are enabling safer application & state composability via packaging (bundles) and introducing new protocols for interoperability across independent applications networks.

Let’s have a look at each of these new concepts.

(1) Introducing application networks & new application packages

As we have seen in the previous post, a key takeaway from the adoption of 4 was to move towards a more application centric model.

In practice that means optimising the platform around a user journey that starts from the developer and puts applications at the center, rather than starting layering abstractions starting from the node and working our way up to a business network.

The approach we are taking in 5 gives developers the ability to define at build time the properties of the permissioned network that will run their application(s) in an artefact we call Installer.

An Installer contains all the CorDapps packages (in what we call this a Bundle), network information & membership attributes that govern a given application network.

An Installer becomes the single artefact to onboard a participant or to version manage in a distributed application network.

Applications networks are now simplified Corda networks. They are still permissioned and the run time responsibilities of membership management moves from a dedicated series of services (Corda Enterprise Network Manager) to a simple, extensible application running on a node — removing the need to run dedicated infrastructure to manage a network of peers.

(2) Introducing new Modular APIs & new Developer tooling

We have rewritten our APIs to be interface-based and modular.

You can think of the platform in layers that are built on top of each other:

• A network interface for secure networking and peer interactions, location-agnostic peer messaging, strong cryptographic keys.
• A workflow layer as a data agnostic, decentralised apps API to coordinate actions across parties​.
• A shared ledger that offers reliable data agreement replication across distributed parties (WYSIWIS).
• Finally, a contract layer that allows the enforcement of business logic at the edges of your network.

Our APIs will now loosely match those layers (more in an upcoming blog post), which is helpful in many ways:

• As a developer you can incrementally (bottom up) build your application by engaging with the appropriate layer when it is needed, this mental model is now more directly represented through our APIs.
• As a developer you can test specific parts of your application independently (for example, mock a workflow).
• As a developer you can decide not to use our implementation of the ledger model and go with your own persistence model etc. opening the platform up to new use cases.
• Cherry on top, is our own development velocity as this gives us the ability to develop more rapidly (behind the interfaces and in parallel), introduce new implementations of the layers down the line etc.

Finally, as mentioned in the previous post, a key metric we want to improve in 5 is reducing development time and cost. Developer experience is a key priority for us in order to meet our goals.

With the Corda 5 Developer Preview we previewed some of those initiatives, adopting OpenAPI for node interactions, introducing a new CLI tool to help you manage local machine development, as well as APIs to hook up to your CI/CD pipeline and much more. Check out more about this on our developer platform.

(3) Moving to an Event driven architecture

This is a big shift in how Corda works inside and is mostly visible on what and how you deploy it. It ties back to one of our key objectives of the re-architecture — making Corda 5 highly available and scalable to power distributed critical systems. The shift can be better summarised with “turning the Corda Client into an event driven system”.

The runtime of a node will no longer be confined to a single JVM, instead the deployment topology will look more like a cluster of workers listening to a message bus (Kafka) for events to process (in its HA configuration). A worker is an instance of a Java Virtual Machine containing a group of processing modules/services — you can think of it as a container you deploy.

Depending on the objective of your deployment you might have several different workers, each with a specific redundancy policy. For example, you may have several active versions of the same worker in your cluster.

Worker redundancy is helpful for availability objectives — if something goes wrong there is an instant failover to the next worker. But it also allows for parallelized event processing, increasing throughput in bursty scenarios.

This architecture lends itself also to dynamic scaling, as new workers can be added to the cluster with the cluster able to expand and shrink depending on the demand.

If all the above looks and feels like containers and orchestration is because it is indeed designed to be ran as such.

(4) Introducing Virtual nodes, Multi-Network and (full “node”) Multi-tenancy

A new feature we are working towards that will work nicely with the new architecture of the Corda client is Virtual nodes.

As the name implies — a virtual node is the virtual mapping between application processes, data and identity and keys to form the logical equivalent of a Corda 4 node (such as Alice in Network 1 with App A/B/C).

Alice can take advantage of the compute power of a shared worker cluster while retaining segregated or dedicated access to her key and data and ensuring her applications run in isolation of everybody else’s (yes, we are introducing sandboxing in C5!).

This means you can have several virtual nodes running independently on the same worker cluster.

This is an exciting step forward for:

• enabling true (virtual) node multi-tenancy on a shared cluster
  > this is important for managed service providers or developers looking at progressive decentralisation.
• enabling multi-network
  > Alice can join several application networks using the same “client” at no margin cost.
• reducing cost of ownership
  > particularly for operating models where previously multiple nodes were deployed
• improving development/test experience
  > particularly for production-like tests, dynamic configuration of a cluster (with several virtual nodes) allows for a much faster and simpler test set up/ run time.

Finally — as virtual nodes are just a mapping between processes, data, and keys, they can be…. portable. Which means they can be reconstructed in different “clients” if — for instance — a participant wishes to repatriate operations. With the cooperation of the original cluster operator — a virtual node can migrate its data to a different client and keys can be rotated to be safely generated in the new environment.

(5) Enabling safer composability and explicit interoperability across application networks

We have seen some of the packaging changes with the introduction of Installers earlier on. We have also seen that application processes will be now sandboxed — what is the composability story going forward?

• Application composability (the ability of two applications to share run time) becomes much easier — as application packages are loaded in their own sandboxes — you no longer run the risk of dependencies clashes with other apps or the client itself.
• State composability (the ability of an application to consume a state generated by another application) remains possible but only within the boundaries of the installer. This simplifies things quite a bit, as a developer you can bundle several applications together and take advantage of this pattern (and test it!) but also have the guarantees that no other application (outside of the installer) can consume their states.

What if bundling is not the right solution, what if you wish to access/integrate with an existing application network (or arbitrary blockchain networks)?

• This is a big area of work for us. We have showcased at CordaCon several approaches we plan to introduce in the platform to give developers explicit protocols to follow for two or more application networks to interoperate. These will introduce ways for independent application networks to reference data, reissue states or orchestrate an atomic swap with minimal upfront coordination.
• In conjunction with multi-network support with virtual nodes, this makes for a much more organic model for the formation of a network of networks and creates the regulated internet of value we are after.

Let’s recap — where are we taking Corda next and why?

Hopefully these 5 key area of focus in our development should give you a hint on where we are taking the platform next — and the previous post should give you some of the context behind these changes (particularly if you are new to Corda).

The key take away is that we are changing a lot of the properties (non-functionals) of the platform but none of its DNA:

• We aim with our Modular APIs, new tools, and packaging to make developing in Corda faster and more enjoyable.
• The new artefact — Installer — combined with simpler Application network — should make your go to market, onboarding and version management experience much simpler.
• The new architecture should give you the flexibility to evolve your operating model as you grow, with the ability to move the operations of virtual nodes across worker clusters.
• The app-based network management and virtual nodes should make Corda less expensive to run for everybody involved.
• Finally, Corda’s new event driven architecture should give us enough headroom to go after the most sophisticated enterprise multi-party use cases and power the next central bank digital currency, global payment network or critical market infrastructure.

Call to actions before you leave:

• Next on the series we’ll take a deeper look at the architectural changes and learn more about workers and how to deploy them and manage them
• In the meantime, make sure to give a go at the new APIs with the dev preview of Corda 5 and let us know what you think over in slack or discord (search for the “corda5” channel)
• Any feedback on the direction you want to give to the product team directly, let us know at [email protected]

Thanks! Gabe