Cause of the testnet stall identified
The cause of the testnet stall was explained, and preparation for the zcashd 6.0.0 release was wrapped up.
Agenda summary 11
NU6 testnet recovery and zcashd 6.0.0 release preparation
ECC, Zcash Foundation · Daira Emma, Kris, Dodger
- The testnet is catching up at about 1 minute per minute from a state where block timestamps are far behind, and how to connect and workarounds will be shared separately.
- The zcashd 6.0.0 release branch is ready, and they decided to tag it after confirming on testnet the change that lowers the free action limit to 0.
- This change is expected to help transaction propagation on mainnet and reduce the remaining spam.
- 6.0.0 is the NU6 activation release that expires in January.
- Kris asked ZF to set up a public testnet that uses only Zebra without zcashd, and Dodger accepted since it is already under discussion.
zcashd and zebrad protocol differences and the block template bug
ECC · Kris, Daira Emma, Dodger
- ECC found a difference between the initial block download protocols of zcashd and zebrad, and a timestamp clamping input bug in the zcashd block template.
- The clamping bug will go into the release branch as a small patch.
- Daira Emma corrected that the protocol difference also affects mainnet and that zcashd sync can slow down when there are many Zebra nodes.
- On the zcashd side it is resolved by backporting some Bitcoin fixes, and on the Zebra side by removing the existing workaround code.
- Daira Emma said the P2P protocol is the least documented area, so interoperability problems easily arise.
Zebra 2.0.0 schedule and librustzcash dependencies
ECC, Zcash Foundation · Arya, Alfredo, Kris, Daira Emma
- Zebra 1.9.0 supports NU6 only on testnet, so Zebra 2.0.0, which includes the minimum protocol version and activation height for mainnet, is needed, and the final version will follow a candidate.
- Alfredo hoped to start the release work next week and finish by the weekend at the latest.
- Setting the free action limit to 0 is already reflected in Zebra, and Daira Emma agreed that it must be documented in ZIP 317.
- zcash_protocol 0.4, zcash_address 0.6, and zcash_primitives and zcash_proofs 0.19 include the mainnet activation height.
- Supporting a new network upgrade is semantically a breaking change, so the whole stack must be bumped.
- ECC discussed a zcash constants crate that splits out only the constants, and the direction is to change Zebra's transaction types into wrappers around librustzcash types.
ECC update: Sean Bowe's move, Zashi, and the note commitment tree bug
ECC · Kris, Daira Emma
- Zashi plans to ship new features around next week, including an address book, ZIP 321 integration and a UI redesign.
- They finally fixed a note commitment tree corruption bug caused by an API flaw in an old incremental merkle tree crate.
- The new shardtree crate release has a semantic change that third-party trait implementations must follow, so the release notes should be checked.
- The protocol specification and related ZIPs were updated for NU6, and the ZCG share address was also included.
- Sean Bowe will now receive funding separate from ECC and focus on Zcash scalability research.
Zebra update
Zcash Foundation · Arya, Kris, Daira Emma
- They applied part of the zcashd RPC test framework to Zebra, and made context block validation return an error instead of panicking when the parent block cannot be found.
- Items such as orphan mempool transaction validation, a fix so that sendrawtransaction returns its result after the mempool insertion or rejection, cookie-based RPC authentication, and a setting for the halving interval on custom testnets are merged or in PR state.
- They opened a PR for an index and read state request that looks up a spending transaction ID by spent output and revealed nullifier.
- They bundled the DB format upgrade into the same PR so it is done only once.
- For restoring the internal miner, review of the librustzcash equihash C code PR remains, and CI checks whether all crates can be released, which blocks merging to main.
- Local mining is possible on the restoration branch.
zcashd deprecation progress and outreach
ECC, Zcash Foundation, Zingo Labs · Dodger, Kris, Pacu
- Zingo Labs' Zaino grant was approved and they will start work soon.
- Kris and Str4d are building a repository for a minimal wallet service skeleton that uses light client code as the chain data source.
- Pacu, Pili and decentralistdan started ecosystem outreach to find where zcashd is used, collecting requirements so that no use case is dropped at deprecation.
Debate over the scope of zcashd wallet RPC compatibility
ECC, Zcash Foundation · Kris, Daira Emma, Alfredo, Dodger, Pacu, Arya
- Kris took a weak position of not providing compatibility with the existing zcashd wallet RPCs and changing APIs whose semantics differ as breaking changes, and said requirements should be collected as business needs.
- Daira Emma and Alfredo proposed mapping the existing API onto an account model with new semantics and documenting it.
- Kris said that if needed a compatibility layer can be added during the 16-week adoption period that follows NU7 implementation, audit and one month of testnet.
- Kris proposed making full protocol support the default with transparent-only as secondary, but Dodger objected that it could be delisted from more than 95% of exchanges and that this is a scope expansion.
- Dodger suggested asking exchanges which standard they prefer as an interface, such as the Mesh (formerly Rosetta) API that Coinbase pointed to.
- Pacu suggested putting an RPC translation module in Zaino, and Daira Emma suggested encouraging the move to shielded by adding features such as FROST and hardware wallets.
Trailing Finality update
Shielded Labs · Zooko
- Shielded Labs is currently interviewing and hiring, and asked people to refer anyone interested in the protocol and the Zcash mission.
- At the next Arborist call they will gather final input on the first deployment's requirements, and until then they take input through a shared Google document or individual contact.
Open discussion: NU6 testnet recovery status
Pacu, Daira Emma, Kris, Arya, Alfredo
- They expect the testnet to work normally once it catches up to the chain tip within a few hours, and decided to post related guidance in #nodedev on the R&D Discord.
- Arya explained that if Zebra followed the wrong fork, the copy-state command can copy only the state of the correct fork, and the method is in the Zebra 1.9.0 release notes.
- Alfredo proposed a document covering the cause and fix, and Daira Emma said they would hold a retrospective and write it down.
Cause of the testnet halt: header sync and max tip age
ECC · Daira Emma
- zcashd wasted several times the sync bandwidth because of a duplicate getheaders request bug that came in from Bitcoin Core in 2015.
- To compensate, Zebra sent fewer headers than the maximum (158), but when Zebra was the authoritative chain zcashd wrongly concluded it had reached the chain end, and sync slowed greatly.
- The max tip age rule, which halts a node if there is no block for more than 24 hours, made it worse this time because there was no mining for 7 days.
- Daira Emma worked around it by raising this constant to 10 days.
Testnet recovery process and lessons
ECC, Zcash Foundation · Kris, Dodger, Daira Emma, Arya
- They temporarily fixed zcashd's block timestamp clamping bug, and built a connected network by manually finding the best chain and using add-node.
- Kris confirmed that zcashd syncs even without the max tip age setting, so the testnet was recovered.
- Dodger pointed out that the trigger was the single mining node for the faucet going down and mining stopping, and Daira Emma summarized that it was not any particular node's fault but a design problem in header sync.
- Daira Emma pointed out the lesson to avoid changes meant to compensate for bugs in other software.
- Kris added that ECC and ZF were not collaborating as closely at that time as they are now.
This summary is based on the original meeting notes; check them for exact details.