(source)

Table of Contents

• RFC-0112: Compress the State Response Message in State Sync
  • Summary
  • Motivation
  • Stakeholders
  • Explanation
  • Drawbacks
  • Testing, Security, and Privacy
  • Performance, Ergonomics, and Compatibility
    • Performance
    • Ergonomics
    • Compatibility
  • Prior Art and References
  • Unresolved Questions
  • Future Directions and Related Material

RFC-0112: Compress the State Response Message in State Sync

Summary

This RFC proposes compressing the state response message during the state syncing process to reduce the amount of data transferred.

Motivation

State syncing can require downloading several gigabytes of data, particularly for blockchains with large state sizes, such as Astar, which has a state size exceeding 5 GiB (https://github.com/AstarNetwork/Astar/issues/1110). This presents a significant challenge for nodes with slower network connections. Additionally, the current state sync implementation lacks a persistence feature (https://github.com/paritytech/polkadot-sdk/issues/4), meaning any network disruption forces the node to re-download the entire state, making the process even more difficult.

Stakeholders

This RFC benefits all projects utilizing the Substrate framework, specifically in improving the efficiency of state syncing.

• Node Operators.
• Substrate Users.

Explanation

The largest portion of the state response message consists of either CompactProof or Vec<KeyValueStateEntry>, depending on whether a proof is requested (source):

• CompactProof: When proof is requested, compression yields a lower ratio but remains beneficial, as shown in warp sync tests in the Performance section below.
• Vec<KeyValueStateEntry>: Without proof, this is theoretically compressible because the entries are generated by iterating the storage sequentially starting from an empty storage key, which means many entries in the message share the same storage prefix, making it ideal for compression.

Drawbacks

None identified.

Testing, Security, and Privacy

The code changes required for this RFC are straightforward: compress the state response on the sender side and decompress it on the receiver side. Existing sync tests should ensure functionality remains intact.

Performance, Ergonomics, and Compatibility

Performance

This RFC optimizes network bandwidth usage during state syncing, particularly for blockchains with gigabyte-sized states, while introducing negligible CPU overhead for compression and decompression. For example, compressing the state response during a recent Polkadot warp sync (around height #22076653) reduces the data transferred from 530,310,121 bytes to 352,583,455 bytes — a 33% reduction, saving approximately 169 MiB of data.

Performance data is based on this patch, with logs available here.

Ergonomics

None.

Compatibility

No compatibility issues identified.

Prior Art and References

None.

Unresolved Questions

None.

Future Directions and Related Material

None.