The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* update to 2.4.1
* update use of DNS daemon with Vertx
* fix issue with Bytes.repeat
* update antlr version
* fix dns tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
---------
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* logging cleanup, giant new payload.
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
---------
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
Add support for withdrawals in GraphQL, including needed changes to
testing infrastructure for shanghai-era blocks.
Also align existing adapters with graphql schema optionality.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Allow scheme (ws or wss) to be specified in ethstats url
* Start ethstat service after main ethereum loop is up
Signed-off-by: Usman Saleem <usman@usmans.info>
* Fix the number of RocksDB log files to 5 with 100 MB size for each LOG file
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Fix build issue
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* spotless and add a changelog entry
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* keep 1 week log files
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
* Update change log and fix the number of files
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
---------
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: ahamlat <ameziane.hamlat@consensys.net>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Introduce experimental layered transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* new Xlayered-tx-pool flag to enabled the new tx pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move pending transaction sorter tests in the sorter folder
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unit tests for new and old transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: do not decrease size when promoting ready txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: remove tx from orderByFee when replaced
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: decrease size when removing confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: always recreate orderByFee for London fee market
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: transaction removal counter when txs added to block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: update expected nonce when demoting a prioritized transaction
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: correctly remove expected nonce entry when removing the last prioritized transaction for the sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix NullPointerException when the replaced tx is not prioritized
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Replace postponed with spare transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge from main
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixed most tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix more tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename and reorg some classes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More renaming and code clean up
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Refactor transaction pool metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Stats log refined
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Cleanup unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve stats log
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove unnecessary test parameters
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/MainnetTransactionValidator.java
* Cancel older block creation tasks upon receiving a new one
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Fixes to expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix promotion filter and use synchronized methods instead of blocks
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix metrics concurrent access issue
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Configuration options for the layered txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use long instead of Instant for PendingTransaction add time
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move layered txpool clasess in a dedicated package
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove confirmed transaction from sparse set too
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fill gap on added tx
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix eviction on sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from ready layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for block added and confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes to sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix the filling of the gap when adding transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Layered pending transactions test and fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Distinguish between layer and comulative space used
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* unit tests for expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adding test for transaction selection
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Re-enable prioritized transaction tests and more fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* log stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* syncronized some methods, metrics update and dump transactions for replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Test that replay tx and fix for tx replacement across layers
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing copyright and fix replay test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add consistency check asserts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix ready internalRemove
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Metrics tests improvements
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP: Transaction memory size estimation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Complete pending transaction memory used computation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename to specify that the limit is per layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update metric names in tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adjust tx layer max capacity according to new tx memory size calculation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix legacy transaction expiration tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix IndexOutOfBoundsException in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unique senders metric
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Ignore ReplayTest by default, fix logging of stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log for replay renamings
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Document howto generate txpool replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Reduce max layer capacity
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* exclude transaction replay resource
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve compareByFee when effectivePriorityFee is 0
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More debug logs during transaction selection for a block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use only one thread for building blocks so there is no risk of overlapping
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve transaction trace log making wei human readable
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use List instead of Set when getting all pending transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More detailed log on adding remote txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Execute transaction broadcast aysnc after adding them to the txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log time taken to add remote txs before their broadcast
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing header
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add CHANGELOG entry
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete unneeded file
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename some layered txpool metrics to avoid conflict with existing metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix pushing to next layers txs following an invalid one
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* In case of an unexpected error, log more data and do a consistency check
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix null check on wrong var
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix some codeql alerts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix sparse gap calculation when invalidating
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Apply suggestions from doce review
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Only trigger consistency check if trace log is enable in case of unexpected error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix replay of blocks with no transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for negative gap when there is a reorg
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Implement code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for a case when deleting tx with zero gap in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete redoundant tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG.md
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/ReadyTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/layered/AbstractPrioritizedTransactionsTestBase.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/SparseTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve logSender when there are no sparse txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix off by one error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/TransactionsLayer.java
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename fix
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Simplify the way reorgs are handled, by detecting a negative gap and deleting and readding all the txs for that sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Do not run consistency check on internal error since it is too expensive,
instead force a reorg of the sender txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove invalid txs after the selection is complete to avoid ConcurrentModificationException
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update txpool defaults
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Tune default
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
* Transaction pool flag to disable special handling of locally submitted transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix options and more tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Move the implementation of "big" eof into Future EIPs, out of cancun.
Also, create space for Prague, Osaka, and Bogota forks in the EVM code.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Use binary search to eth_estimateGas more accurately
eth_estimateGas currently conservatively estimates gas this change
will take that estimate and use it as a hi bound and will use
gasUsed as lo bound to hone in on the most true gas limit required
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Performance improvements
- Reduce the use of UInt256 in operations, replacing them with Bytes as
appropriate.
- Use Java17 expression switch
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Refactor to make RocksDBColumnarKeyValueStorage abstract and extend it to optimistic and pessimistic. Atm Forest has a concurrency level that does not cope well with the OptimisticTransactionDB and RocksDB ends up raising a lot of Busy when committing RocksDBTransactions.
A TransactionDB will do conflict checking for all write operations (Put, Delete and Merge), including writes performed outside a Transaction according to RocksDB. This does impact the times we see when syncing so likely not the final solution for Forest.
Bonsai should continue using OptimisticTransactionDB.
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Changing bonsai-max-back-layers-to-load to --bonsai-historical-block-limit
Signed-off-by: Matt Nelson <monels11@gmail.com>
* remove a couple other places where snapshot option was configured
Signed-off-by: garyschulte <garyschulte@gmail.com>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/stable/DataStorageOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
---------
Signed-off-by: Matt Nelson <monels11@gmail.com>
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Add new sepolia bootnodes from EF
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Parithosh Jayanthi <parithosh@indenwolken.xyz>
* removed separate decoding logic
* run non-mainnet ATs to make sure
* remove goQuorum flag from everywhere
* remove GOQUORUM api group
* remove GoQuorum enclave, privacy params, RPCs and other related config
* removed Goquorum related error codes, rpcMethod codes, privateBloom
* removed v from transaction
* removed private GoQuorum storage provider
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Add 8 metrics to evaluate the state of the flat database for accounts and storage slots
* Total number of calls (for both accounts and storage)
* Number of account or storage entries found in the flat database
* Number of storage slots located in the merkle trie but not found in the flat database
* Number of account or storage entries not found in either the flat database or the merkle trie
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* remove controller and config
* remove ibftlegacy module
* remove further config options
* run non-mainnet ATs just to be sure
* remove temp change to circle config
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
* Fix ethstats retry logic. Attempt to connect with and without ssl. If --ethstats is specified without a port, use port 443 (for wss://) and port 80 (for ws://) as defaults instead of 3000.
* Introduced optional --ethstats-cacert-file to specify root CA for ethstats server.
Signed-off-by: Usman Saleem <usman@usmans.info>
While syncing, it is possible that we access the cached header's number before the block itself has had chance to be imported. In other words there is a mismatch between cached blockNumber and non-cached blockHeader that is being retrieved for the RPC.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Override the transactionSimulator's default TransactionValidationParams with one that allows for exceeding the account balance (which effectively zeros the baseFee).
This mimics the way that eth_estimateGas and eth_call are implemented.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
matkt
Sally MacFarlane
Antoine Toulme
Fabio Di Fabio
garyschulte
Matt Nelson
George Tebrean
Danno Ferrin
Usman Saleem
Simon Dudley
Adrian Sutton
Gabriel-Trintinalia
Jason Frame
ahamlat
Sebastian Bathke
Danno Ferrin
Justin Florentine
Antony Denyer
Diego López León
Nischal Sharma
Gabriel Fukushima