Orchard load and batch verification
Orchard transactions surged after NU5 and blocked Zebra sync, while zcashd cut verification time with batch verification.
Agenda summary 6
Zebra 1.0.0-beta.12 release
Zcash Foundation · Conrado
- Zebra beta.12 was released a few days ago. When the DB format changes, it resyncs from genesis instead of migrating, and it now automatically deletes old cached state to avoid wasting disk.
- Optional features such as tracing and the new metrics were left out of the default compilation and must be enabled at compile time if needed. Time notation in the config file also changed to a human-readable format such as 's'.
- There was a problem where blocks with many Orchard transactions took long to verify and prevented sync, which was resolved by adjusting timeouts and the number of parallel tasks.
- Performance improved slightly by picking up the new Halo 2 crate, and the biggest improvement, they said, will be the Halo 2 batch verification support currently in progress.
- Because of GitHub Actions' 6-hour per-job limit, the sync job that took 5 hours 55 minutes was split into several jobs. Peers on the port used by Flux are rejected and peers on the standard Zcash port are preferred.
Zcashd 5.1.0 release preview - build system and RPC improvements
Electric Coin Company · Steven
- The goal is to release zcashd 5.1.0 next week, and cxx.rs, which replaces hand-written Rust-C++ FFI, and cargo-vet, for verifying trust in Rust dependencies, were integrated into the build system.
- Wallet RPCs were changed to block until reindex and rescan finish. In one case, a partner called z_getbalance during a rescan and the balance came out as about 7 million ZEC, half of the total issuance.
- Rescan now starts only after reindex finishes, so in effect reindex includes rescan. z_gettreestate performance also improved.
- Orchard details (such as the number of actions) were added to gettransaction and decoderawtransaction. The sender, recipient and amount are not revealed.
Orchard proof batch verification
Electric Coin Company · Steven, Daira, Deirdre
- From about two and a half weeks after NU5 activation, up to 500 Orchard transactions per block came in, and verifying such a block took about 2.5 seconds.
- Optimizations such as batch verification cut verification time by about 81%, so the same amount is processed in about 0.5 seconds. The same volume of Sapling transactions would have taken about 7 seconds in the current structure.
- Daira said Sapling has not yet had batch verification applied but it is relatively easy to do. Str4d led the implementation with Daira and Sean also taking part, and a benchmark using large Orchard blocks was included in the PR.
- Deirdre asked whether zcashd uses Groth16 batch verification, which was upstreamed to Bellman, and Daira answered that it does not because they have not gotten to it yet.
Open discussion - sharing knowledge on CI and test suites
Zcash Foundation · Dodger, Marshall, Gustavo Valverde
- Dodger proposed knowledge sharing on CI, automated testing and benchmarking among ECC, ZF and grant recipients such as Ziggurat, and Daira and Steven agreed.
- Steven said that at NU5 a shared test suite that could run Zebra's basic tests against zcashd changes, and conversely some zcashd tests against Zebra changes, would have been useful.
- Marshall mentioned the idea of a shim test layer independent of the main repositories, and it was decided to connect with Gustavo, who joined last December and is overhauling the CI infrastructure, and pursue sharing in presentation format.
- Bundling in Ziggurat's multi-node, network-level tests was also mentioned.
Open discussion - Orchard verification load and DoS response
Zcash Foundation · Teor, Daira, Str4d
- Teor explained that when hundreds of blocks are verified concurrently and 500 transactions per block pile in, RAM and CPU needs are large and timeouts occur. A change to reduce the number of blocks verified concurrently will go into the next release.
- For DoS mitigation, Daira proposed a dynamic response that sets a cap on the memory a full node uses for verification and reduces parallelism when it passes a threshold. They also pointed out that a design assuming the worst-case block breaks when a worse structure is found.
- Zebra currently does not cache verification results, and they said caching keyed on the chain tip hash is possible. Daira said a cache helps the average case but has no effect against the worst-case DoS.
- Dodger proposed holding a private meeting on this topic and Str4d agreed. Marshall suggested creating a common set of monitoring rules and alerts.
Open discussion - node metrics and a network status page
Zcash Foundation · Teor, Marshall, Daira, Str4d
- Teor proposed a site, like the Tor Project's public metrics site, that shows the number of nodes, the number and types of transactions, and block fetch delays as time series.
- Marshall said past grant projects and others made similar attempts and suggested resuming the discussion, and Steven said it could become a joint project combining observed metrics and test status.
- Daira said they want to visualize Zcash network connectivity in real time. Str4d said opt-in, privacy-preserving metrics collection by full nodes has become easier than before.
- There was agreement that node counts are public information and so less of a problem, but how nodes connect to each other should be handled carefully so as not to give attackers easy information.
This summary is based on the original meeting notes; check them for exact details.