Refactor some transaction calculations, including moving upfront
overflow checks from the constructor to the validator and optimizing
some RLP calculations. Also fix all lint errors.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Fix the flakeiness in EthGetTransactionByHashTest as well as some other
sonar identified cleanup.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Introduce experimental layered transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* new Xlayered-tx-pool flag to enabled the new tx pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move pending transaction sorter tests in the sorter folder
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unit tests for new and old transaction pool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: do not decrease size when promoting ready txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: remove tx from orderByFee when replaced
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: decrease size when removing confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: always recreate orderByFee for London fee market
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: transaction removal counter when txs added to block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: update expected nonce when demoting a prioritized transaction
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix: correctly remove expected nonce entry when removing the last prioritized transaction for the sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix NullPointerException when the replaced tx is not prioritized
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Replace postponed with spare transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge from main
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixed most tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix more tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename and reorg some classes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More renaming and code clean up
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Refactor transaction pool metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Stats log refined
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Cleanup unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve stats log
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove unnecessary test parameters
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java
# ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/MainnetTransactionValidator.java
* Cancel older block creation tasks upon receiving a new one
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Fixes to expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix promotion filter and use synchronized methods instead of blocks
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix metrics concurrent access issue
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Configuration options for the layered txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use long instead of Instant for PendingTransaction add time
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move layered txpool clasess in a dedicated package
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove confirmed transaction from sparse set too
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fill gap on added tx
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix eviction on sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from ready layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix remove from sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for block added and confirmed txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fixes to sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix the filling of the gap when adding transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Layered pending transactions test and fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Distinguish between layer and comulative space used
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* unit tests for expected next nonce for sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adding test for transaction selection
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Re-enable prioritized transaction tests and more fixes
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* log stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* syncronized some methods, metrics update and dump transactions for replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Test that replay tx and fix for tx replacement across layers
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing copyright and fix replay test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add consistency check asserts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix ready internalRemove
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Metrics tests improvements
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* WIP: Transaction memory size estimation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Complete pending transaction memory used computation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename to specify that the limit is per layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update metric names in tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adjust tx layer max capacity according to new tx memory size calculation
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix legacy transaction expiration tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix IndexOutOfBoundsException in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Unique senders metric
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Ignore ReplayTest by default, fix logging of stats
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log for replay renamings
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Document howto generate txpool replay
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Reduce max layer capacity
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* exclude transaction replay resource
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve compareByFee when effectivePriorityFee is 0
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More debug logs during transaction selection for a block
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use only one thread for building blocks so there is no risk of overlapping
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve transaction trace log making wei human readable
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Use List instead of Set when getting all pending transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* More detailed log on adding remote txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Execute transaction broadcast aysnc after adding them to the txpool
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log time taken to add remote txs before their broadcast
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix test
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add missing header
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix unit tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Add CHANGELOG entry
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete unneeded file
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename some layered txpool metrics to avoid conflict with existing metrics
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix pushing to next layers txs following an invalid one
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* In case of an unexpected error, log more data and do a consistency check
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix null check on wrong var
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix some codeql alerts
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix sparse gap calculation when invalidating
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Apply suggestions from doce review
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Only trigger consistency check if trace log is enable in case of unexpected error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix replay of blocks with no transactions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for negative gap when there is a reorg
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Implement code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix for a case when deleting tx with zero gap in sparse layer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Delete redoundant tests
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update CHANGELOG.md
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPool.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolMetrics.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/ReadyTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/layered/AbstractPrioritizedTransactionsTestBase.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/SparseTransactions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/TransactionPoolOptions.java
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Improve logSender when there are no sparse txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix off by one error
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/layered/TransactionsLayer.java
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Address code review suggestions
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Rename fix
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Simplify the way reorgs are handled, by detecting a negative gap and deleting and readding all the txs for that sender
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Do not run consistency check on internal error since it is too expensive,
instead force a reorg of the sender txs
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove invalid txs after the selection is complete to avoid ConcurrentModificationException
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Update txpool defaults
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Tune default
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix merge
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
---------
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Due to subtle differences between Bytes32 and UInt256 the changes failed
burn-in testing and are being reverted.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
The current internal storage APIs implement UInt256 rather than Bytes32,
which is an accident of prior EVM design. This migrates all internal
handling of storage addresses and values to Bytes32.
The main performance gain is reduced GC pressure, in that UInt256
stores the data as an array of ints, whereas Bytes32 keeps a sliced or
wrapped array of bytes. Since database APIs are all byte array based
this should remove unneeded conversions when going to/from the DB.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* interprete an empty string (0x80) as an unsigned byte of 0
Signed-off-by: Stefan <stefan.pingel@consensys.net>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Remove deprecated cleanup task
* Move util class into test-support
This is because this class can be used outside the scope of the junit tests but it'll not be part of main
* Move referenceTests into its own scope
* Migrate referenceTests to Junit5
I inlined `o.h.b.e.vm.AbstractRetryingTest` into `o.h.b.e.vm.VMReferenceTest` because there was a single usage of the abstraction and also because it wasn't portable to how Junit5 parameters work
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Danno Ferrin <danno.ferrin@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
10.4 revision of reference tests has notable changes
* Nonce can be up to 2^64-1, with some opcode and validity interactions specced in eip-2681
* Wei fields can be up to 2^256, tests check for rollover
* VM Tests were removed
* Legacy Tests were removed
* Reference tests make it clear that transactions with a maxPriorityFee of zero are valid. Perhaps useless, but still valid.
* Adding validation hooks in the out-of-chain test execution. These validations are caught in full-chain execution, just not in conveniently places for integration testing.
* This does not transaction tests support for the "merge" fork, as the release notes marked it as an "upcoming" feature.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
Co-authored-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Daniel Lehrner <daniel.lehrner@consensys.net>
* Fix two operand stack tests
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Remove unreachable assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace expected annotation field with assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Extract variables to let only a single call for assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Broad reaching optimizations to speed up EVM calculations
* Generally speaking, use int and long where it is more appropriate than UInt256 (memory indexes mostly)
* Move the internal stack to Bytes from UInt256
* Re-work the flow of many operations to account for the above
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Upgrade to Apache Tuweni 2.0
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove intermediate repository
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove all occurrences of toBytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Migrate to tuweni-bytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* add changelog
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* correct reference tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Initial API changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* more changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Change APIs for VM ops
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Use constant UInt256.ONE
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Optimize a bit address <> word transformation
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* spotless
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Implement max message size rather then cap with fixed number of transactions
Adding transactions to the RLP until the message size exceeds the limit and then send that.
* fix final variables
* Update AbstractRLPOutput.java
add javadoc
* pr discussion
put this factory method on LimitedTransactionsMessages rather than TransactionsMessage since it returns a LimitedTransactionsMessages.
* SpotlessApply
* fix PR discussion
- simplify design
- remove useless code
* Update LimitedTransactionsMessages.java
* fix PR discussion
- simplify logic
- add tests
* Update AbstractRLPOutput.java
* Update ethereum/eth/src/main/java/tech/pegasys/pantheon/ethereum/eth/messages/LimitedTransactionsMessages.java
Co-Authored-By: abdelhamidbakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
* Update Transaction.java
* fix PR discussion
* fix PR discussion
- add tests
* Update BlockDataGenerator.java
* Update LimitedTransactionsMessagesTest.java
fix PR unit test
* Update LimitedTransactionsMessagesTest.java
* Update LimitedTransactionsMessagesTest.java
Use LinkedHashSet to preserve order.
* Update LimitedTransactionsMessagesTest.java
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Received New Round messages require significant validation to ensure their content is consistent and valid with regard the current state of the system.
This commit includes a class (and tests) which ensures illegally constructed NewRound messages are identified.
This does not include the action to be taken upon detection of an illegal New Round message.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Danno Ferrin
Fabio Di Fabio
Nischal Sharma
Sally MacFarlane
Usman Saleem
lukelee-sl
Danno Ferrin
mark-terry
Stefan Pingel
mbaxter
Diego López León
fab-10
Ratan (Rai) Sur
Antoine Toulme
Jason Frame
David Mechler
Danno Ferrin
Antoine Toulme
Joshua Fernandes
mbaxter
Abdelhamid Bakhta
tmohay
Adrian Sutton
Christopher Mckay
PegaSys Admin