Hybrid Block Explorer
- Team Name: Jonathan Brown
- Payment Address: 0x36a7401F269555916a0243E586234D3Bbf5A0c36 (DAI)
- Level: 1
Project Overview 📄​
This application is in response to two RFPs:
- Multi-chain Block Explorer https://github.com/w3f/Grants-Program/blob/master/docs/RFPs/multi-chain-block-explorer.md
- ink! smart contract block explorer. https://github.com/w3f/Grants-Program/blob/master/docs/RFPs/ink_smart_contract_block_explorer.md
Overview​
Hybrid takes a unique, partially decentralized approach that improves two major problems with current open source Substrate block explorers: centralization and huge hosting requirements.
A fully centralized block explorer typically populates an SQL database with the entirety of an archive node and stores additional data to index everything. Operating such a database reliably requires huge system resources and expense.
When querying block information, or the chain state at any block height, the Hybrid dapp will use the Substrate Connect light client from within the browser. Alternatively, these queries can be made directly to an archive node via WSS.
For event search functionality, the Hybrid indexer efficiently indexes events in all blocks so they can be found with a simple WSS query. For example, to find all events connected with a specific AccountId
.
This architecture has three main advantages:
- state queries are fully decentralized - you don't have to trust an RPC provider not to lie to you
- 100% availability - the light client doesn't depend on any centralized service that may not always be available
- the Hybrid indexer has significantly lower system requirements - it doesn't need to store all chain history
Eventually, Hybrid will use this centralized / decentralized approach as the basis for an ink! contract explorer.
Because Substrate is a federated platform, it will be possible browse multiple chains from the Hybrid dapp.
Project Details​
Hybrid will be a Substrate block explorer dapp. By default it will connect to major Substrate blockchains. Additionally it can be configured to connect to any Substrate chain.
There are two types of queries that the explorer can perform:
- Decentralized - ideally using the Substrate Connect light client, or alternatively connecting to an archival node via WSS. For example, querying blocks or state storage.
- Centralized - the Hybrid indexer for the chain will be queried via WSS. This is for event searching and watching, such as finding all events across all chains that relate to a specific
AccountId
.
Indexer​
The Hybrid indexer will be written in Rust. It can be configured to connect to any Substrate chain.
It will read events in all blocks using subxt and index these events in a Key-value database using the sled library. This is considerably more efficient than storing the index in an SQL database.
subxt currently has an issue where it is not possible to query blocks prior to V14 metadata (block #7,229,126 on Polkadot). Resolving this issue is not within the scope of the grant. Once this grant is completed a further grant application will be made that includes resolving this issue.
When decoding events, subxt needs to have the correct metadata. The metadata changes whenever a chain performs a runtime upgrade. Hybrid Indexer handles this in a very elegant way. When indexing begins it downloads the metadata for the starting block. When it encounters a decoding error it downloads the metadata for the current block and retries decoding. This means that the indexer does not have to be built with the metadata and block number of every runtime upgrade.
To index an event, it needs to be converted into a Rust type that matches the metadata. Sometimes the metadata for an event will change during a runtime upgrade. To handle this the indexer will have Rust types for current and historic versions of all events. When an event fails to be converted into a Rust type the previous type will be tried.
All events in all pallets that have identifying parameters will be indexed. For example the Transfer event in the Balances pallet is identifiable by the AccountId
of both from
and to
.
Other examples of identifying event parameters are assetId
in the Assets pallet, code_hash
in the contracts pallet, CollectionId
and ItemId
in the NFTs pallet, and MultiLocation
in the XCM pallet.
Additionally, all events are indexed by event variant.
To download a block, a query first has to be made to determine the hash for a given block number. In order to ensure throughput is as high as possible, multiple queries to the full node will be active at the same time to avoid round-trip delay. Block processing will be in a separate thread.
In the same manner that each Substrate chain is a separate Rust build that uses Substrate crates, each chain will need a separate Hybrid Indexer build that is configured to index the correct pallets.
When a chain is going to potentially perform a runtime upgrade, the Hybrid Indexer for the chain will need a new release with any updated events. If an instance of the indexer is not updated before the runtime upgrade occurs, it can be restarted with the new version at the correct block number.
WSS queries will be handled via the highly scalable tokio_tungstenite Rust library.
In addition to the identifier being searched for, queries will be able to include start block, offset, and limit to control which events are returned.
Consumers will be able to subscribe for new events that match a query.
The database keys will be constructed in such a way so that events can be found using iterators starting at a specific block number. For example, for for the AccountId keyspace:
AccountId/BlockNumber/EventIndex
Database entries will be key-only. No value will be stored. The blocknumber and event index are all that need to be returned for each event found. This reduces the size of the index database and increases decentralization. The frontend can query the chain in a decentralized manner to retrieve the event.
Dapp​
The Hybrid dapp will be a Vue dapp, using the Vuetify framework for the user interface. pnpm and Vite will be used for the build.
It will use @polkadot/api to retrieve data from the chain, either via the Substrate Connect light client, or via an RPC connection to a full archival node. The Hybrid indexer will be queried via WSS.
This grant will only include a basic dapp including:
- block browsing, showing transactions and events
- event searching, e.g. searching for all events connected to a specific
AccountId
.
A subsequent grant application will be made to develop a richer dapp experience.
Ecosystem Fit​
A major issue holding back the Substrate and Polkadot ecosystem is a high quality block explorer comparable to Etherscan. We need such a block explorer to be:
- non-proprietary
- as decentralized as possible
- not requiring massive system resources
- feature-rich
The target audience is blockchain enthusiasts and developers. Eventually end-users should not need to know about block explorers, but this depends on dapps improving their user experience.
The indexing component has value far beyond the Hybrid block explorer. Many Substrate applications will find a centralized event indexer extremely useful.