polkadot_sdk_docs/reference_docs/omni_node.rs
1//! # (Omni) Node
2//!
3//! This reference doc elaborates on what a Polkadot-SDK/Substrate node software is, and what
4//! various ways exist to run one.
5//!
6//! The node software, as denoted in [`crate::reference_docs::wasm_meta_protocol`], is everything in
7//! a blockchain other than the WASM runtime. It contains common components such as the database,
8//! networking, RPC server and consensus. Substrate-based nodes are native binaries that are
9//! compiled down from the Rust source code. The `node` folder in any of the [`templates`] are
10//! examples of this source.
11//!
12//! > Note: A typical node also contains a lot of other tools (exposed as subcommands) that are
13//! > useful for operating a blockchain, such as the ones noted in
14//! > [`polkadot_omni_node_lib::cli::Cli::subcommand`].
15//!
16//! ## Node <> Runtime Interdependence
17//!
18//! While in principle the node can be mostly independent of the runtime, for various reasons, such
19//! as the [native runtime](crate::reference_docs::wasm_meta_protocol#native-runtime), the node and
20//! runtime were historically tightly linked together. Another reason is that the node and the
21//! runtime need to be in agreement about which consensus algorithm they use, as described
22//! [below](#consensus-engine).
23//!
24//! Specifically, the node relied on the existence of a linked runtime, and *could only reliably run
25//! that runtime*. This is why if you look at any of the [`templates`], they are all composed of a
26//! node, and a runtime.
27//!
28//! Moreover, the code and API of each of these nodes was historically very advanced, and tailored
29//! towards those who wish to customize many of the node components at depth.
30//!
31//! > The notorious `service.rs` in any node template is a good example of this.
32//!
33//! A [trend](https://github.com/paritytech/polkadot-sdk/issues/62) has already been undergoing in
34//! order to de-couple the node and the runtime for a long time. The north star of this effort is
35//! twofold :
36//!
37//! 1. develop what can be described as an "omni-node": A node that can run most runtimes.
38//! 2. provide a cleaner abstraction for creating a custom node.
39//!
40//! While a single omni-node running *all possible runtimes* is not feasible, the
41//! [`polkadot-omni-node`] is an attempt at creating the former, and the [`polkadot_omni_node_lib`]
42//! is the latter.
43//!
44//! > Note: The OmniNodes are mainly focused on the development needs of **Polkadot
45//! > parachains ONLY**, not (Substrate) solo-chains. For the time being, solo-chains are not
46//! > supported by the OmniNodes. This might change in the future.
47//!
48//! ## Types of Nodes
49//!
50//! With the emergence of the OmniNodes, let's look at the various Node options available to a
51//! builder.
52//!
53//! ### [`polkadot-omni-node`]
54//!
55//! [`polkadot-omni-node`] is a white-labeled binary, released as a part of Polkadot SDK that is
56//! capable of meeting the needs of most Polkadot parachains.
57//!
58//! It can act as the collator of a parachain in production, with all the related auxillary
59//! functionalities that a normal collator node has: RPC server, archiving state, etc. Moreover, it
60//! can also run the wasm blob of the parachain locally for testing and development.
61//!
62//! ### [`polkadot_omni_node_lib`]
63//!
64//! [`polkadot_omni_node_lib`] is the library version of the above, which can be used to create a
65//! fresh parachain node, with a some limited configuration options using a lean API.
66//!
67//! ### Old School Nodes
68//!
69//! The existing node architecture, as seen in the [`templates`], is still available for those who
70//! want to have full control over the node software.
71//!
72//! ### Summary
73//!
74//! We can summarize the choices for the node software of any given user of Polkadot-SDK, wishing to
75//! deploy a parachain into 3 categories:
76//!
77//! 1. **Use the [`polkadot-omni-node`]**: This is the easiest way to get started, and is the most
78//! likely to be the best choice for most users.
79//! * can run almost any runtime with [`--dev-block-time`]
80//! 2. **Use the [`polkadot_omni_node_lib`]**: This is the best choice for those who want to have
81//! slightly more control over the node software, such as embedding a custom chain-spec.
82//! 3. **Use the old school nodes**: This is the best choice for those who want to have full control
83//! over the node software, such as changing the consensus engine, altering the transaction pool,
84//! and so on.
85//!
86//! ## _OmniTools_: User Journey
87//!
88//! All in all, the user journey of a team/builder, in the OmniNode world is as follows:
89//!
90//! * The [`templates`], most notably the [`parachain-template`] is the canonical starting point.
91//! That being said, the node code of the templates (which may be eventually
92//! removed/feature-gated) is no longer of relevance. The only focus is in the runtime, and
93//! obtaining a `.wasm` file. References:
94//! * [`crate::guides::your_first_pallet`]
95//! * [`crate::guides::your_first_runtime`]
96//! * If need be, the weights of the runtime need to be updated using `frame-omni-bencher`.
97//! References:
98//! * [`crate::reference_docs::frame_benchmarking_weight`]
99//! * Next, [`chain-spec-builder`] is used to generate a `chain_spec.json`, either for development,
100//! or for production. References:
101//! * [`crate::reference_docs::chain_spec_genesis`]
102//! * For local development, the following options are available:
103//! * `polkadot-omni-node` (notably, with [`--dev-block-time`]). References:
104//! * [`crate::guides::your_first_node`]
105//! * External tools such as `chopsticks`, `zombienet`.
106//! * See the `README.md` file of the `polkadot-sdk-parachain-template`.
107//! * For production `polkadot-omni-node` can be used out of the box.
108//! * For further customization [`polkadot_omni_node_lib`] can be used.
109//!
110//! ## Appendix
111//!
112//! This section describes how the interdependence between the node and the runtime is related to
113//! the consensus engine. This information is useful for those who want to understand the
114//! historical context of the node and the runtime.
115//!
116//! ### Consensus Engine
117//!
118//! In any given substrate-based chain, both the node and the runtime will have their own
119//! opinion/information about what consensus engine is going to be used.
120//!
121//! In practice, the majority of the implementation of any consensus engine is in the node side, but
122//! the runtime also typically needs to expose a custom runtime-api to enable the particular
123//! consensus engine to work, and that particular runtime-api is implemented by a pallet
124//! corresponding to that consensus engine.
125//!
126//! For example, taking a snippet from [`solochain_template_runtime`], the runtime has to provide
127//! this additional runtime-api (compared to [`minimal_template_runtime`]), if the node software is
128//! configured to use the Aura consensus engine:
129//!
130//! ```text
131//! impl sp_consensus_aura::AuraApi<Block, AuraId> for Runtime {
132//! fn slot_duration() -> sp_consensus_aura::SlotDuration {
133//! ...
134//! }
135//! fn authorities() -> Vec<AuraId> {
136//! ...
137//! }
138//! }
139//! ```
140//!
141//! For simplicity, we can break down "consensus" into two main parts:
142//!
143//! * Block Authoring: Deciding who gets to produce the next block.
144//! * Finality: Deciding when a block is considered final.
145//!
146//! For block authoring, there are a number of options:
147//!
148//! * [`sc_consensus_manual_seal`]: Useful for testing, where any node can produce a block at any
149//! time. This is often combined with a fixed interval at which a block is produced.
150//! * [`sc_consensus_aura`]/[`pallet_aura`]: A simple round-robin block authoring mechanism.
151//! * [`sc_consensus_babe`]/[`pallet_babe`]: A more advanced block authoring mechanism, capable of
152//! anonymizing the next block author.
153//! * [`sc_consensus_pow`]: Proof of Work block authoring.
154//!
155//! For finality, there is one main option shipped with polkadot-sdk:
156//!
157//! * [`sc_consensus_grandpa`]/[`pallet_grandpa`]: A finality gadget that uses a voting mechanism to
158//! decide when a block
159//!
160//! **The most important lesson here is that the node and the runtime must have matching consensus
161//! components.**
162//!
163//! ### Consequences for OmniNode
164//!
165//!
166//! The consequence of the above is that anyone using the OmniNode must also be aware of the
167//! consensus system used in the runtime, and be aware if it is matching that of the OmniNode or
168//! not. For the time being, [`polkadot-omni-node`] only supports:
169//!
170//! * Parachain-based Aura consensus, with 6s async-backing block-time, and before full elastic
171//! scaling). [`polkadot_omni_node_lib::cli::Cli::experimental_use_slot_based`] for fixed factor
172//! scaling (a step
173//! * Ability to run any runtime with [`--dev-block-time`] flag. This uses
174//! [`sc_consensus_manual_seal`] under the hood, and has no restrictions on the runtime's
175//! consensus.
176//!
177//! [This](https://github.com/paritytech/polkadot-sdk/issues/5565) future improvement to OmniNode
178//! aims to make such checks automatic.
179//!
180//! ### Runtime conventions
181//!
182//! The Omni Node needs to make some assumptions about the runtime. During startup, the node fetches
183//! the runtime metadata and asserts that the runtime represents a compatible parachain.
184//! The checks are best effort and will generate warning level logs in the Omni Node log file on
185//! failure.
186//!
187//! The list of checks may evolve in the future and for now only few rules are implemented:
188//! * runtimes must define a type for [`cumulus-pallet-parachain-system`], which is recommended to
189//! be named as `ParachainSystem`.
190//! * runtimes must define a type for [`frame-system`] pallet, which is recommended to be named as
191//! `System`. The configured [`block number`] here will be used by Omni Node to configure AURA
192//! accordingly.
193//!
194//! [`templates`]: crate::polkadot_sdk::templates
195//! [`parachain-template`]: https://github.com/paritytech/polkadot-sdk-parachain-template
196//! [`--dev-block-time`]: polkadot_omni_node_lib::cli::Cli::dev_block_time
197//! [`polkadot-omni-node`]: https://crates.io/crates/polkadot-omni-node
198//! [`chain-spec-builder`]: https://crates.io/crates/staging-chain-spec-builder
199//! [`cumulus-pallet-parachain-system`]: https://docs.rs/cumulus-pallet-parachain-system/latest/cumulus_pallet_parachain_system/
200//! [`frame-system`]: https://docs.rs/frame-system/latest/frame_system/
201//! [`block number`]: https://docs.rs/frame-system/latest/frame_system/pallet/storage_types/struct.Number.html