The currently way Besu has to spot if a transaction has already been process, is to look if the transaction is present in the transaction pool, that by default is 4K, while the amount of pending transactions on the mainnet, is much more, the order of hundred of thousands, so basically even if a transaction has been already processed, the chances that it gets reprocessed is very high, with the result of doing a lot of useless work, that affects Besu performance.
A trivial solution could be to just raise the transaction pool size, but that is not always advisable, because it is critical for block production to keep it fast, and incresing its size could negatively affect the perfomance of the strategy choosen to select transactions to include in the block.
A better option, implemented here, is to leverage data that we already have, and that keeps the history of the transactions exchanged with other peers. This data is just a collection of transaction hashes that we have received or seen, and in any case if a transaction is in that collection, it means that it has already been processed by Besu, so it is possible to directly skip it.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Currently Besu has a limited support for sending NewPooledTransactionHashes messages, and other aspect related to reduce transactions synchronization traffic, described in the Ethereum Wire Protocol version 66.
Specifically:
Besu only uses NewPooledTransactionHashes for new local transactions, while it could be extended to any transaction added to the transaction pool
Besu does not limit the sending of the full transaction messages to a small fraction of the connected peers, and sends the new transaction hashes to all the remaining peers
This PR, extends eth/66 support and does some code refactoring, to remove some reduntant code and rename some classes to identify they are related to the NewPooledTransactionHashes message.
The main changes are:
Do not have a separate tracker for transaction hashes, since for them we can reuse PeerTransactionTracker, that tracks full transactions exchange history and sending queue with a peer. So PeerPendingTransactionTracker has been removed. --tx-pool-hashes-max-size is now deprecated and has no more effect and it will be removed in a future release.
When a new peer connects, if it support eth/6[56] then we send all the transaction hashes we have in the pool, otherwise we send the full transactions.
When new transactions are added to the pool, we send full transactions to peers without eth/6[56] support, or to a small fractions of all peers, and then we send only transaction hashes to the remaining peer that support eth/6[56]. Both transactions and transaction hashes are only sent if not already exchanged with that specific peer.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Tune transaction worker queue capacity to adapt to mainnet
Currently Besu handles hundreds of transaction message per seconds,
so having a queue of 1M messages result in a lot of expired messages,
make sense to reduce the capacity to drop incoming messages that could not
be handled anyway.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Tune the number transactions seen by peer to adapt to mainnet traffic
Currently on mainnet there are many thousand of pending transactions
exchanged between peers, but Besu has a short memory of what has been
exchanged with a specific peer, with the result that the same transaction
is often exchanged back and forth with the same peer, expecially when the
peer is another Besu.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve trace log of transaction exchanges with other peers
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Allow precompiled contracts to return richer information instead of
results/fail. System/precompile contracts can now revert, fail, refund
gas, etc. instead of just succeed or fail.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* refactored auth to allow variant JWT rules to be adopted by different stacks
* configures JWT secret from a file, or creates a temporary one
* jwt auth is optional for engine apis
* test coverage of new cli options
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* Json RPC endpoint and test files for trace_rawTransaction
* cache the block number to make sure we are using the same block
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Co-authored-by: Frank Li <b439988l@gmail.com>
Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>
* Refactor TransactionPool to accept MiningParameters
* Check for zero GasPrice Frontier Transactions
* if you are not mining your node could fill up with pending transactions.
* make low-or-no-gas transactions viable for local transactions
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Co-authored-by: garyschulte <garyschulte@gmail.com>
* some unit tests
Signed-off-by: Frank Li <b439988l@gmail.com>
* CLI parsing tests
Signed-off-by: Frank Li <b439988l@gmail.com>
* regression tests
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix failing at
Signed-off-by: Frank Li <b439988l@gmail.com>
* refactor -> api to api methods
Signed-off-by: Frank Li <b439988l@gmail.com>
* functionality in
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix override test
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix duplicates test
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix duplicates test
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix failing test and remove unnecessary code
Signed-off-by: Frank Li <b439988l@gmail.com>
* add entry to changelog
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix typo and NPE
Signed-off-by: Frank Li <b439988l@gmail.com>
* action on some items
Signed-off-by: Frank Li <b439988l@gmail.com>
* some refactoring + more tests + implement auth skip for web sockets
Signed-off-by: Frank Li <b439988l@gmail.com>
* refactor unused method
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix test failing
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix wrong variable used
Signed-off-by: Frank Li <b439988l@gmail.com>
List of components: controller builder, protocol schedule,
coordinator, block creator and processor.
Code originally written by garyschulte on the merge branch.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add PostMergeContext that will stop syncing after the swith to PoS
Code originally written by garyschulte and gezero on the merge branch.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add PostMergeContext that will stop syncing after the swith to PoS
Code originally written by garyschulte and gezero on the merge branch.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unit test for PostMergeContext
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add CHANGELOG entry
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Addressing comments
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More PostMergeContext unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Rename CONSTANTS_PREFIX to VARIABLES_PREFIX,
since values stored there are not fixed and change over time.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Extend validateAndProcessBlock to return an error message in case of failure
This message can then be used to send a more detailed response to the RPC
caller of ExecutePayload API.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Merge: extend block creation and mining to support The Merge
Code originally written by garyschulte on the merge branch.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
When asking for the forkId for the chain head sometimes we may return
null. In those cases return an empty list.
Fixes#3343
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* add a websocket max frame size configuration in support of nimbus EL/CL interop (#3379)
* add changelog
Signed-off-by: garyschulte <garyschulte@gmail.com>
* Refactor to async retrieve blocks, and change peer when retrying to get a block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Apply suggested changes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove deprecated class
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add more logs arond block synchronizer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* First try do download the block from the peer that announced it
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Avoid redownload non annunced blocks
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Conditionally log at trace level
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use max number of peers when retrying to download a block to try all peers
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use the shared Slf4jLambdaHelper, instead of custom helper
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Bump SLF4J version
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace log4j2 API with SLF4j API
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace usage of LogManager#getFormatterLogger
This is for keeping compatibility with SLF4J. If neccesary, a specific formatter can be created for the RlpBlockImporter class
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Unset the default logging value for the retesteth
This is because it's not possible to resolve the root logger level into a Log4J2 field
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Prevent creation of Logger context outside SLF4J
org.hyperledger.besu.cli.BesuCommand#setAllLevels was taken from
https://github.com/apache/logging-log4j2/blob/rel%2F2.17.1/log4j-core/src/main/java/org/apache/logging/log4j/core/config/Configurator.java#L309
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add FATAL level deprecation message
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S2139
Exceptions should be either logged or rethrown but not both
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S3457
Printf-style format strings should be used correctly
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add changelog
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Do not exit ChainDownloader in case of a generic CancellationException
There right way of halting a ChainDownloader is only via the cancel method,
no special action should be taken on a generic CancellationException, that
could be triggered by peer tasks that run in the pipeline, since in that
case the error is transient and the download should restart.
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
When picking up the fast sync after previous incomplete run of fast sync we have to clear the old worldstate data to be sure children are persisted before parents.
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
Simon Dudley
Lucas Saldanha
Fabio Di Fabio
Sally MacFarlane
Frank Li
garyschulte
Danno Ferrin
mbaxter
Justin Florentine
Stefan Pingel
Antony Denyer
fab-10
Diego López León
Jiri Peinlich