Replace usages of streamMilestoneBlocks on production code paths: RunnerBuilder and TransactionPoolFactory.
These usages were also using getBlockByNumber which is ultimately what we're trying to remove.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
<!-- Thanks for sending a pull request! Please check out our
contribution guidelines: -->
<!-- https://github.com/hyperledger/besu/blob/main/CONTRIBUTING.md -->
## PR description
There is an issue when restarting Besu when a backward sync session is
running, since after the restart it is possible that the Consensus
client sends a FcU or a NewPayload for a block that is present in the
backward sync storage, but not yet imported, so not on the main chain,
but still the backward sync thinks it should not do anything with that
block, so it returns like it has completed the sync, but since the sync
is not done actually then the internal error that the finalize block is
not present.
The solution is to persist the backward sync status, so in case of a
restart, it can resume from where it was interrupted.
## Fixed Issue(s)
<!-- Please link to fixed issue(s) here using format: fixes #<issue
number> -->
<!-- Example: "fixes #2" -->
fixes#5053
## Documentation
- [x] I thought about documentation and added the `doc-change-required`
label to this PR if
[updates are
required](https://wiki.hyperledger.org/display/BESU/Documentation).
## Acceptance Tests (Non Mainnet)
- [x] I have considered running `./gradlew acceptanceTestNonMainnet`
locally if my PR affects non-mainnet modules.
## Changelog
- [x] I thought about the changelog and included a [changelog update if
required](https://wiki.hyperledger.org/display/BESU/Changelog).
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Pull up and simplify data passed to CheckpointSource, which then allows checkpointBlockHeader to be used by protocolSchedule.getByBlockHeader.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Fix empty body concept after shanghai
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Check protocolSchedule to create empty block and add withdrawals to empty block check
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add unit tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Extract block creation into a private method
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Nit changes
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Refactor test names
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
* Use the copy during prepareTrieLog instead of saveTrieLog
* add final flag on BonsaiWorldStateUpdater
* Use a copy of BonsaiInMemoryWorldState inside prepareTrieLog
* add link to persisted worldstate storage
* fix tests
* Make a copy of the worldstate after committing changes to the trielog
* spotless + remove maybeUnSubscribe in setNextWorldView
* subscribe storage on layered worldstate
* fix null issue
* not close layered worldstate during getAccount
* clean code
* Add changelog entry
---------
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Karim TAAM <karim.t2am@gmail.com>
* Replace getByBlockNumber by getByBlockHeader where trivial
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Rolledback change since tests will need non trivial changes
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Rollback replace
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Change other 2 trivial tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Pass ProcessableBlockHeader to the mocks
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Replace getByBlockNumber by getByBlockHeader
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add mock of blockHeader
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Use any() for the tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add unit test for headBlockHeader()
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* bonsai layered worldstate subscription
* unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState
* minor txpool logging improvements
Signed-off-by: garyschulte <garyschulte@gmail.com>
* add withdrawalRoot computation
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* include withdrawalsRoot and withdrawals when creating block
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* validate withdrawals when validation mainnet blocks
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Fix invalid withdrawals root during backwards sync
Add withdrawalsRoot as part of BodyIdentifier
Add comment to avoid new issues when adding new fields to BlockBody
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Include withdrawalsRoot when creating new header in EngineNewPayload
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Use empty trie hash in genesis for withdrawalsRoot if shanghai is enabled at genesis
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Revert testing changes to AbstractEngineNewPayload
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Fix BesuController tests failing after GenesisState change
Signed-off-by: Jason Frame <jason.frame@consensys.net>
* Missing withdrawals wiring in BlockProcessor
Add empty withdrawals in GenesisState for shanghai at genesis
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Jason Frame <jason.frame@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
- Added missing javadocs so that javadoc doclint passes against JDK 17 (invoke by Besu gradle build).
- Exclude following packages from javadoc lint:
org.hyperledger.besu.privacy.contracts.generated
org.hyperledger.besu.tests.acceptance.*
- Temporarily exclude ethereum and evm submodule for doc lint checks.
- Run the javadoc task using GitHub actions (use Java 17) to report any javadoc errors during the PR builds
- Updating plugin-api build.gradle with new hash as javadoc comments caused it to change
Signed-off-by: Usman Saleem <usman@usmans.info>
* use debug rpc endpoint to resync worldstate
* Reset transaction pool state every time the initial sync is done
Signed-off-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Modify reattemptPendingPeerRequests implementation to not iterate on pending request. This should avoid a lot of unnecessary computation
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Fix failed unit tests.
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Fix failed unit tests
Iterate over pending requests only if there is a peer with available request capacity
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Fix stream implementation (use the same stream several times)
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Spotless
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Undo reducing method visibility as it is no longer used
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* use one buffered for each peer and wait to have capacity before sending request
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* fix tests
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Karim TAAM <karim.t2am@gmail.com>
The idea behind this commit is to preload asynchronously account nodes and storage nodes from the database during the transaction processing to use these nodes during the calculate root hash step.
We've created two caches, one for account nodes and one for storage nodes. The size of these caches is 100k for accounts and 200k for storage. We've tested other values but this configuration is the one that works better.
We also use exporter cache metrics as Prometheus metrics to check cache efficiency.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Ameziane H <ameziane.hamlat@consensys.net>
Stefan Pingel
Sally MacFarlane
garyschulte
Simon Dudley
matkt
Fabio Di Fabio
Gabriel Fukushima
Gabriel-Trintinalia
Diego López León
ahamlat
Jason Frame
Justin Florentine
Usman Saleem
mark-terry
lukelee-sl