Skip to main content

Centrifuge Go-Substrate-RPC Client V2

  • Team Name: k/factory (former Centrifuge Development Team)
  • Payment Address: Ethereum(USDC) - 0x2B8A956BF807E22d858dc9AD2eFc30cccc3Ea676
  • Level: 🐓 3

Project Overview 📄​

This is a follow-up grant to keep building on top of this Go Client implementation. In the past we had two grants.

The original grant through w3f:

And a maintenance Grant through Polkadot Treasury:

Overview​

Initially Go-Substrate-RPC (GSRPC for short) was designed under the premise of being a low-level, static and strongly typed library, where all the type resolution happen at compilation time.

This approach allowed almost full flexibility to developers to build their own types and events depending on the target chain/s they were connecting to.

On the other hand, since the types are statically defined, this didn’t leave room to, gracefully, deal with parsing errors when a type (defined on the chain runtime) is not defined in the GSRPC client.

Project Details​

The problem​

For a hypothetical RuntimeA, give TypeA:

type TypeA struct {
    AttrA String
    AttrB int
}

and EventA, that uses TypeA:

type EventA struct {
    Data TypeA
}

The GSRPC instance should define TypeA to be able to decode EventA. If the client is defined as such, then there will be no problem parsing it.

Now lets presume that a RuntimeUpgrade is executed and now the target chain is at version B -> RuntimeB. In this runtime ugprade attribute AttrB of struct TypeA has changed to String:

type TypeA struct {
    AttrA String
    AttrB String
}

Then when the GSRPC client is trying to parse the event, it will fail while trying to decode the scale encoded value into the EventA struct since the type that is expecting for AttrB is an int but a string has been provided.

This issue gets more critical when we are talking about processes, like bridges, that need to successfully parse every event in a block looking for a relevant event to their business logic. If due to the above reason, an event can't be parsed then the bridge relayer cannot ensure that a relevant event was or wasn't in that block. Therefore the only options are to skip that block (hoping that there was no relevant event within) or wait until the client is updated to continue parsing blocks, which disrupts the bridge operations.

Solution​

With the inclusion of Substrate Metadata 14, we now have a definition of all the types included in a specific runtime.

This means that at bootstrap time (and upon other certain situations) GSRPC can build a registry of types based on the data provided by the target chain metadata.

Load types into registry from Metadata​
Example of Metadata 14 type structure for a Balances.Transfer Event​

Pallet Name and associated type variants index

Pallets.5.Name = "Balances"
Pallets.5.Events.Type.UCompact = 28 // Events list type index

For event Balances.Transfer with index 2 in the variants list there are 3 fields, from, to & amount.

Lookup.Types.27.Variant.Variants.2.Name = "Transfer"
Lookup.Types.27.Variant.Variants.2.Fields.0.Name = "from"
Lookup.Types.27.Variant.Variants.2.Fields.0.TypeName = "T::AccountId" // Some types do not have an index reference
Lookup.Types.27.Variant.Variants.2.Fields.1.Name = "to"
Lookup.Types.27.Variant.Variants.2.Fields.1.TypeName = "T::AccountId"
Lookup.Types.27.Variant.Variants.2.Fields.2.Name = "amount"
Lookup.Types.27.Variant.Variants.2.Fields.2.TypeName = "T::Balance"
Lookup.Types.27.Variant.Variants.2.Fields.2.Type.UCompact = 6

from & to have type T::AccountId which is a Runtime Primitive, and potentially can have different implementations depending on the runtime connected to. In most of the cases, it would be an AccountId32, but that can be changed.

amount has a type reference to index 6 so we can navigate to that type:

Lookup.Types.6.Type.Def.Primitive = 7 (Si0TypeDefPrimitive)

And that tells us that is refering to Rust Primitive number 7, so according to the primitive list (found in polkadotjs/api) for type Si0TypeDefPrimitive:

0:  readonly isBool: boolean;
1:  readonly isChar: boolean;
2:  readonly isStr: boolean;
3:  readonly isU8: boolean;
4:  readonly isU16: boolean;
5:  readonly isU32: boolean;
6:  readonly isU64: boolean;
7:  readonly isU128: boolean;
8:  readonly isU256: boolean;
9:  readonly isI8: boolean;
10: readonly isI16: boolean;
11: readonly isI32: boolean;
12: readonly isI64: boolean;
13: readonly isI128: boolean;
14: readonly isI256: boolean;

The underlying type is U128

Challenges to solve:

New generic type registry encoder/decoder​

The next step is to build a generic encoder/decoder that works for all discovered types based on underlaying primitive types.

In theory, once the types are loaded, and we built generic encoders for basic primitive types, we should be able to encode & decode all types and events.

Challenges to solve:

  • Build generic encoder/decoder for any type loaded in the registry
  • Support custom encoder/decoder for irregular struct serialization and deserialization

Team 👥​

Team members​

  • Team Lead & Protocol Developer: Miguel Hervas (mikiquantum)
  • Main Protocol Developer: Cosmin Damian

Contact​

  • Registered Address: k-f dev AG, Grafenauweg 8, 6300 ZUG SWITZERLAND
  • Registered Legal Entity: k-f dev AG

Team's experience​

The k/f team has built a decentralized platform for the financial supply chain (invoices, purchase orders, warehouse receipts) comprising of a P2P network and Ethereum smart contracts over the last 2 years.

The platform allows tokenization of any digital assets. A strong team in the DeFi movement, they are now building a Centrifuge-specific chain with Substrate, integrated their existing Go application with the Substrate chain.

The k/f team also developed the Go Substrate RPC client supported by a Web3 Foundation grant.

The k/f team also developed, alongside with ChainSafe, the ChainBridge supported by Web3 Foundation grant.

The k/f team has deep knowledge in distributed/decentralized applications, libp2p, Golang, Solidity and Ethereum overall, zkSNARKs, and tokenization of assets with NFTs and has been developing with Substrate since Summer 2019.

Team Code Repos​

Team LinkedIn Profiles (if available)​

Development Status 📖​

V1 version implemented at https://github.com/centrifuge/go-substrate-rpc-client

V2 will be implemented as part of this grant

Development Roadmap 🔩​

Overview​

  • Total Costs: 54,000 USD

Milestone 1 - Dynamic Type Loader from metadata​

  • Estimated duration: 5 weeks (200 hours)
  • FTE: 1
  • Costs: 30,000 USD
NumberDeliverableSpecification
0a.LicenseApache 2.0
0b.DocumentationWe will provide both inline documentation of the code and a basic tutorial on how to load any metadata into its own registry of chain types.
0c.Testing and Testing GuideCore functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests.
0d.DockerWe will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone.
1.Metadata parsing logic into internal typesList of functions, types and structs to support parsing and internally representing any target chain Metadata

Milestone 2 - New generic type registry encoder/decoder​

  • Estimated duration: 4 weeks (160 hours)
  • FTE: 1
  • Costs: 24,000 USD
NumberDeliverableSpecification
0a.LicenseApache 2.0
0b.DocumentationWe will provide both inline documentation of the code and a basic tutorial on how to parse any substrate event and type using the new version of the library.
0c.Testing and Testing GuideCore functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests.
0d.DockerWe will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone.
1.In-Memory RegistryTarget chain types are loaded and indexed properly

Further breakdown:

  • Use the registry for parsing simple and complex structs - 80h
  • Adapt current event & storage processing to new model - 60h
  • Adapt current Types Test to decode events from the latest X blocks of popular chains - 20h

Budget Summary​

This table provides a cost breakdown based on the milestones and deliverables above.

Estimates
MSDeliverableHoursBudget
1Dynamic Type Loader200$30,000
Subtotal Milestone 1200$30,000
2Generic type registry Encoder/Decoder160$24,000
Subtotal Milestone 2160$24,000
Totals620$54,000

Future Plans​

The Centrifuge Protocol is an active user of the GSRPC library and k/f is an active member maintaining it.