Its usage is not clear and it's causing besu to rewind the head to it (a large reorg) when combined with nimbus backwards sync.
Add more backwards sync logging.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
As per a recent EIP update the opcode numbers for Transient Storage
have been updated. EOF opcodes were also updated.
Update the code and test cases for the new numbers.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Improve log of p2p messages
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* [skip ci] Update ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/rlpx/wire/AbstractMessageData.java
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use a more flexible way to trim logged data
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Sanitize user input before printing it in logs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Fix code and storage retrieve for bonsai getting a new world state snapshot to fetch code/storage from in the AccountAdapter
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* update to 2.4.1
* update use of DNS daemon with Vertx
* fix issue with Bytes.repeat
* update antlr version
* fix dns tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
---------
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Load code when WS snapshot still available
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Handle invalid account address
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* logging cleanup, giant new payload.
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
---------
Signed-off-by: Matt Nelson <85905982+non-fungible-nelson@users.noreply.github.com>
Add support for withdrawals in GraphQL, including needed changes to
testing infrastructure for shanghai-era blocks.
Also align existing adapters with graphql schema optionality.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Allow scheme (ws or wss) to be specified in ethstats url
* Start ethstat service after main ethereum loop is up
Signed-off-by: Usman Saleem <usman@usmans.info>
This API fulfils the basic requirements, but will probably be extended in the near future.
Signed-off-by: Stefan <stefan.pingel@consensys.net>
Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Log access denied error separately to other IO exceptions so cause is clearer
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix Java spotless errors
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
---------
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Co-authored-by: Matthew Whitehead <matthew1001@gmail.com>
* Add info log to indicate Besu is waiting on CL to be synced
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Introduce experimental layered transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* new Xlayered-tx-pool flag to enabled the new tx pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move pending transaction sorter tests in the sorter folder
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unit tests for new and old transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: do not decrease size when promoting ready txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: remove tx from orderByFee when replaced
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: decrease size when removing confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: always recreate orderByFee for London fee market
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: transaction removal counter when txs added to block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: update expected nonce when demoting a prioritized transaction
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: correctly remove expected nonce entry when removing the last prioritized transaction for the sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix NullPointerException when the replaced tx is not prioritized
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Replace postponed with spare transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge from main
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixed most tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix more tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename and reorg some classes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More renaming and code clean up
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Refactor transaction pool metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Stats log refined
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Cleanup unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve stats log
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove unnecessary test parameters
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/MainnetTransactionValidator.java
* Cancel older block creation tasks upon receiving a new one
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Fixes to expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix promotion filter and use synchronized methods instead of blocks
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix metrics concurrent access issue
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Configuration options for the layered txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use long instead of Instant for PendingTransaction add time
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move layered txpool clasess in a dedicated package
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove confirmed transaction from sparse set too
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fill gap on added tx
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix eviction on sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from ready layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for block added and confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes to sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix the filling of the gap when adding transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Layered pending transactions test and fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Distinguish between layer and comulative space used
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* unit tests for expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adding test for transaction selection
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Re-enable prioritized transaction tests and more fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* log stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* syncronized some methods, metrics update and dump transactions for replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Test that replay tx and fix for tx replacement across layers
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing copyright and fix replay test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add consistency check asserts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix ready internalRemove
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Metrics tests improvements
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP: Transaction memory size estimation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Complete pending transaction memory used computation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename to specify that the limit is per layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update metric names in tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adjust tx layer max capacity according to new tx memory size calculation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix legacy transaction expiration tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix IndexOutOfBoundsException in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unique senders metric
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Ignore ReplayTest by default, fix logging of stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log for replay renamings
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Document howto generate txpool replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Reduce max layer capacity
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* exclude transaction replay resource
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve compareByFee when effectivePriorityFee is 0
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More debug logs during transaction selection for a block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use only one thread for building blocks so there is no risk of overlapping
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve transaction trace log making wei human readable
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use List instead of Set when getting all pending transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More detailed log on adding remote txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Execute transaction broadcast aysnc after adding them to the txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log time taken to add remote txs before their broadcast
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing header
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add CHANGELOG entry
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete unneeded file
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename some layered txpool metrics to avoid conflict with existing metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix pushing to next layers txs following an invalid one
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* In case of an unexpected error, log more data and do a consistency check
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix null check on wrong var
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix some codeql alerts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix sparse gap calculation when invalidating
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Apply suggestions from doce review
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Only trigger consistency check if trace log is enable in case of unexpected error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix replay of blocks with no transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for negative gap when there is a reorg
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Implement code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for a case when deleting tx with zero gap in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete redoundant tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG.md
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/ReadyTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/layered/AbstractPrioritizedTransactionsTestBase.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/SparseTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve logSender when there are no sparse txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix off by one error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/TransactionsLayer.java
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename fix
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Simplify the way reorgs are handled, by detecting a negative gap and deleting and readding all the txs for that sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Do not run consistency check on internal error since it is too expensive,
instead force a reorg of the sender txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove invalid txs after the selection is complete to avoid ConcurrentModificationException
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update txpool defaults
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Tune default
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
* Refactor to remove duplicated code that was handling RLP read/write. This logic doesn't change that ofter having the read and writes spread accross the code base has tripped some devs. We are now handling all the Body readfrom rlp and write to rlp using methods of the BlockBody class.
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
* Transaction pool flag to disable special handling of locally submitted transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix options and more tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
matkt
Simon Dudley
Sally MacFarlane
Danno Ferrin
Fabio Di Fabio
Gabriel Fukushima
Antoine Toulme
garyschulte
Matt Nelson
Stefan Pingel
Usman Saleem
Gabriel-Trintinalia
Adrian Sutton
Matt Whitehead
Jason Frame
Sebastian Bathke
Camilo Reis Dotto