* Refactor to make RocksDBColumnarKeyValueStorage abstract and extend it to optimistic and pessimistic. Atm Forest has a concurrency level that does not cope well with the OptimisticTransactionDB and RocksDB ends up raising a lot of Busy when committing RocksDBTransactions.
A TransactionDB will do conflict checking for all write operations (Put, Delete and Merge), including writes performed outside a Transaction according to RocksDB. This does impact the times we see when syncing so likely not the final solution for Forest.
Bonsai should continue using OptimisticTransactionDB.
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Changing bonsai-max-back-layers-to-load to --bonsai-historical-block-limit
Signed-off-by: Matt Nelson <monels11@gmail.com>
* remove a couple other places where snapshot option was configured
Signed-off-by: garyschulte <garyschulte@gmail.com>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/stable/DataStorageOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
---------
Signed-off-by: Matt Nelson <monels11@gmail.com>
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Add new sepolia bootnodes from EF
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Parithosh Jayanthi <parithosh@indenwolken.xyz>
* removed separate decoding logic
* run non-mainnet ATs to make sure
* remove goQuorum flag from everywhere
* remove GOQUORUM api group
* remove GoQuorum enclave, privacy params, RPCs and other related config
* removed Goquorum related error codes, rpcMethod codes, privateBloom
* removed v from transaction
* removed private GoQuorum storage provider
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Add 8 metrics to evaluate the state of the flat database for accounts and storage slots
* Total number of calls (for both accounts and storage)
* Number of account or storage entries found in the flat database
* Number of storage slots located in the merkle trie but not found in the flat database
* Number of account or storage entries not found in either the flat database or the merkle trie
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* remove controller and config
* remove ibftlegacy module
* remove further config options
* run non-mainnet ATs just to be sure
* remove temp change to circle config
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
* Fix ethstats retry logic. Attempt to connect with and without ssl. If --ethstats is specified without a port, use port 443 (for wss://) and port 80 (for ws://) as defaults instead of 3000.
* Introduced optional --ethstats-cacert-file to specify root CA for ethstats server.
Signed-off-by: Usman Saleem <usman@usmans.info>
While syncing, it is possible that we access the cached header's number before the block itself has had chance to be imported. In other words there is a mismatch between cached blockNumber and non-cached blockHeader that is being retrieved for the RPC.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Override the transactionSimulator's default TransactionValidationParams with one that allows for exceeding the account balance (which effectively zeros the baseFee).
This mimics the way that eth_estimateGas and eth_call are implemented.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Changes and config to support using GraalVM AOT to compile and
execute evmTool as an alternate configuration.
While not as long-term performant the startup time
makes filling reference tests with Besu reasonable.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
<!-- Thanks for sending a pull request! Please check out our
contribution guidelines: -->
<!-- https://github.com/hyperledger/besu/blob/main/CONTRIBUTING.md -->
## PR description
There is an issue when restarting Besu when a backward sync session is
running, since after the restart it is possible that the Consensus
client sends a FcU or a NewPayload for a block that is present in the
backward sync storage, but not yet imported, so not on the main chain,
but still the backward sync thinks it should not do anything with that
block, so it returns like it has completed the sync, but since the sync
is not done actually then the internal error that the finalize block is
not present.
The solution is to persist the backward sync status, so in case of a
restart, it can resume from where it was interrupted.
## Fixed Issue(s)
<!-- Please link to fixed issue(s) here using format: fixes #<issue
number> -->
<!-- Example: "fixes #2" -->
fixes#5053
## Documentation
- [x] I thought about documentation and added the `doc-change-required`
label to this PR if
[updates are
required](https://wiki.hyperledger.org/display/BESU/Documentation).
## Acceptance Tests (Non Mainnet)
- [x] I have considered running `./gradlew acceptanceTestNonMainnet`
locally if my PR affects non-mainnet modules.
## Changelog
- [x] I thought about the changelog and included a [changelog update if
required](https://wiki.hyperledger.org/display/BESU/Changelog).
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* If a PoS block creation repetition takes less than a configurable duration, then waits before next repetition
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add unit test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/MiningOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Use the copy during prepareTrieLog instead of saveTrieLog
* add final flag on BonsaiWorldStateUpdater
* Use a copy of BonsaiInMemoryWorldState inside prepareTrieLog
* add link to persisted worldstate storage
* fix tests
* Make a copy of the worldstate after committing changes to the trielog
* spotless + remove maybeUnSubscribe in setNextWorldView
* subscribe storage on layered worldstate
* fix null issue
* not close layered worldstate during getAccount
* clean code
* Add changelog entry
---------
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Karim TAAM <karim.t2am@gmail.com>
* Add engine get payload body methods and test
Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add header
Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com>
* Update result struct & add test
Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com>
* Change constant to use upper case
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add PayloadBody class and withdrawals to response of methods
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add unit tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add changelog
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add check to prevent returning trailing null results past the head
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add test to check trailing null post head scenario
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Split tests into pre and post shanghai
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Rename methods
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Use getName() to log method name
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Rename variable
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Call constructor directly
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix ByHash json parsing
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Fix json parsing again
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Add check to prevent unnecessary queries
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Refactor method
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add new error code for EngineGetPayloadBodies methods
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add return error for request above the API limit
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add constructor for empty response
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* add check for number of blocks requested and for requests of post head
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add test to check error code when request exceeds API limits
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* add constant for max blocks allowed per request
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix some nits
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add invalid params check
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add tests for invalid params check
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix test and spotless
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Revert "Fix json parsing again"
This reverts commit 558d325bf3.
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Revert "Fix ByHash json parsing Signed-off-by: Simon Dudley <simon.dudley@consensys.net>"
This reverts commit 663e11e2
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Use UnsignedLongParameter to cast params of the request
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add optional withdrawals to the NewPayload log (#5021)
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* kubernetes and errorprone - update versions (#5013)
* update errorprone and kubernetes versions
* fixed errorprone issues in prod cod
* fixed errorprone issues in test code
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
* Rename JsonRpcService to EngineJsonRpcService (#5036)
* rename JsonRpcService to EngineJsonRpcService
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
* Params should be single item of array type, not outer array of strings (#5037)
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Add EIP-2537 (BLS12-381 curve precompiles) to Cancun (#5017)
Add the BLS curve precompiles into the registry for cancun. All of the
curve precompiles have been here since berlin, so this is just wiring
them in.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Use UnsignedLongParameter to cast params of the request
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add optional withdrawals to the NewPayload log (#5021)
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* kubernetes and errorprone - update versions (#5013)
* update errorprone and kubernetes versions
* fixed errorprone issues in prod cod
* fixed errorprone issues in test code
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
* Rename JsonRpcService to EngineJsonRpcService (#5036)
* rename JsonRpcService to EngineJsonRpcService
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
* Params should be single item of array type, not outer array of strings (#5037)
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Add EIP-2537 (BLS12-381 curve precompiles) to Cancun (#5017)
Add the BLS curve precompiles into the registry for cancun. All of the
curve precompiles have been here since berlin, so this is just wiring
them in.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Convert start and count from hex to match JSON-RPC Spec standard
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Co-authored-by: Zhenyang Shi <wcgcyx@gmail.com>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: Daniel Lehrner <daniel.lehrner@consensys.net>
Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Convert blockCount from hex to match JSON-RPC Specification
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Fix tests
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Add changelog
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* bonsai layered worldstate subscription
* unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState
* minor txpool logging improvements
Signed-off-by: garyschulte <garyschulte@gmail.com>
Update following sections:
Gradle build, CircleCI, GitHub Actions, Docker
The devp2p TLS mutual authentication unit tests dealing with PKCS11+nss and QBFT+PKCS11+nss ATs were failing on Java 17 when dealing with RSA keys. The unit tests and AT are updated to use EC keys instead. The keystores, truststores, CRL etc. involved in those ATs are regenerated and detailed instructions are provided on how they are created.
Signed-off-by: Usman Saleem <usman@usmans.info>
* Do not check state on failed TXes
* Exclude EIPTests from test set
* Include withdrawals in block for blockchain reference test cases
* remove outdated test exclusions.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Jason Frame <jason.frame@consensys.net>
Co-authored-by: Jason Frame <jason.frame@consensys.net>
* Add option to limit requests in a single batch
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
* Change changelog
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
* Set default max batch size to one
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
* Update changelog
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
* Fix max rpc batch size for unit tests
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
* Change variable name
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@gmail.com>
Gabriel Fukushima
Sally MacFarlane
Fabio Di Fabio
Matt Nelson
Danno Ferrin
ahamlat
Simon Dudley
Usman Saleem
Jason Frame
Håvard Anda Estensen
garyschulte
Gabriel-Trintinalia
Justin Florentine