Update Bonsai Tries persistence so they can calculate intermediate world
state for pre-byzantium transactions.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Finish internal ConstantinopleFix->Petersburg transition, including
class names. Only backwards compatibility and retesteth use of the
name will remain.
* Remove four unused config fields from json configs. These are fields
with no code uses of any sort. All are implied by other fields.
eip150Hash, eip155Block, eip160Block, and daoForkSupport.
* Remove redundant fork block fields where their value is implied
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
intermediate values of the base fee can be higher than maximum value encoded by a long. Now using BigInteger to compute intermediate fee delta.
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Two updates to Bonsai Tries
* Log Rolling is implemented on top of the existing Persisted head. When
Besu is at chain head and the new best head makes the current head an
orphan branch, the Bonsai TrieLogs are used to roll back to a common
block and roll forward to the needed base block. Goerli is known to
maintain sync. There are still some issues with frontier era block
receipts.
* Non-mutable reads can be done off of the persisted block. These are
accurate for all reads that were performed in the block. If a read is
not known it proceeds through a fallback series of calls to prior
layers until it hits the persisted block. These layered reads are
driven off of the TrieLogs.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Two refactorings to make the next Bonsai PR less noisy.
Prefer WorldState when we used to ask for MutableWorldState.
Persist takes an optional block header instead of a block hash.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Bonsai tries will require storing state by block hash rather than by
state root. To accommodate both forest mode and bonsai mode all state
queries will pass in both the block hash and state root. This also
permits parallel forest/bonsai modes for private state with bonsai
public state.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Remove all but 4 log4j2.xml config files
* The main config for the besu CLI app
* The config for the evmTool CLI app
* The config for acceptance tests
* A config in testUtil
If any tests depend on a log4j file they should import testutil.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Add tracing support for internals and JSON-RPC
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove rocksdb tracing as it slows down execution too much
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add B3 headers extraction on JSON-RPC requests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove traces around trie tree as they slow down syncing significantly
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing to fast sync pipeline
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing for all pipelines
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Address code review
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add acceptance tests and break out the shaded dependency
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Fix tracer id
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Revert changes to trie
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Upgrade otel to latest, remove old tracing
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review comments
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Fixes two bugs related to tracing APIs
- invalid trace when calling a precompiled contract
- invalid trace when a create fails with a depth > 0
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Move the block header validation errors logging levels from trace and
debug to info. Also, include a standard prefix "Invalid block header: "
in each of the log lines.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Since the EIP-1559 transition is going away, simply use a set of
accepted transactions for transaction validator that we'll check
against.
Don't assume that there are two transaction types and instead check
what type the transactions are.
Use guessType when we're deserializing from json.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The change from create to DEFAULT was something that I saw was needed
to keep the reference tests performant in the upcoming EIP-2718 changes.
The change from reading the whole mainnet genesis to reading just the
config portion came from OutOfMemoryErrors in the reference tests when
running normalizeKeys.
Also moved them to ProtocolTestFixture since they're only used by tests.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Adding the Log4j "jul" (java.util.logging) adapter resulted in many
messages like this at startup:
`main INFO Registered Log4j as the java.util.logging.LogManager.`
These come from the Log4j status logger. We can get rid of those by
setting the status attribute on all configurations to a higher logging
level. WARN is the next higher level.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* modifying transaction RLP encode/decode to handle GoQuorum private tx value
* added static flag to be read by TransactionRLPDecoder so that we can still allow chainId=1
* ensure chainId and v not both provided
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
With future storage needs in mind change `createPruningStorage` to a
more generic `getStorageBySegmentIdentifier` method where the segment
identifier is the argument. For now that's just pruning storage as
world state needs a specific return type.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* expose epochCalculator to MiningCoordinator
Signed-off-by: Edward Mack <ed@edwardmack.com>
* replace epochCalculator Function with interface
Signed-off-by: Edward Mack <ed@edwardmack.com>
* apply spotless, clean-up tests
Signed-off-by: Edward Mack <ed@edwardmack.com>
* cleanup comments, fix more tests
Signed-off-by: Edward Mack <ed@edwardmack.com>
* fixing failures from testing
Signed-off-by: Edward Mack <ed@edwardmack.com>
* modify dagSeed calculator to use epochCalculator
epochStartBlock to determine epoch start block based on current epoch
length.
Signed-off-by: Edward Mack <ed@edwardmack.com>
* add epochCalculator to StratumConnectionTest
Signed-off-by: Edward Mack <ed@edwardmack.com>
* apply spotless
Signed-off-by: Edward Mack <ed@edwardmack.com>
* remove unnecessary comments
Signed-off-by: Edward Mack <ed@edwardmack.com>
* remove activation block from Ecip1099EpochCalculator
Signed-off-by: Edward Mack <ed@edwardmack.com>
* add messages to IllegalArgumentExceptions, init epochCalculator
Signed-off-by: Edward Mack <ed@edwardmack.com>
* un-wrap private dagSeed method
Signed-off-by: Edward Mack <ed@edwardmack.com>
* fix typo
Signed-off-by: Edward Mack <ed@edwardmack.com>
* add unit tests for epochStartBlock functions
Signed-off-by: Edward Mack <ed@edwardmack.com>
* remove unused field
Signed-off-by: Edward Mack <ed@edwardmack.com>
* Add OTLP metrics support
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Don't automatically push to grpc. Only allow if pushEnabled is set to true
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Missed refactoring
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add missing header
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Don't make otel depend on the push enabled flag
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Expose JUL logs to log4j2
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review fixes and make sure not to start push gateway if not set to prometheus
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* spotless
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Fix unit test
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* code review feedback
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add changelog entry
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Moves TransactionInvalidReason to top level so it can be accessed by
privacy code while also allowing the TransactionValidator to be
collapsed as it's only used on the permissionless side.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* #1066 Switched to use unprefixed hex strings for memory and stack values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Disable flaky tests per Ben Burns(Yeti) request
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Revert last commit and enable ignored tests.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1157 - updated to create 2 agents so that proper bonding can occur
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1162 - Updated test to mock the local peer PING packet creation so that the hash can be managed.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove p2p network code per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Updates from PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Spotless Apply fixes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Admin force cache refresh when called through end point per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Pr updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update changelog for 1.5.1
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove check for 0x prefix on addresses to match expectations
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update graphql pending to allow for sorting of transactions
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Added tests for new miner endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 - PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 - Added tests for precompiled contract and for byzantium and berlin gas values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 - Added tests for precompiled contract and for byzantium and berlin gas values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 Initial changes for EIP-2565
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 - Added tests for precompiled contract and for byzantium and berlin gas values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1086 - PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
Co-authored-by: David Mechler <davemec@users.noreply.github.com>
* #1066 Switched to use unprefixed hex strings for memory and stack values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Disable flaky tests per Ben Burns(Yeti) request
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Revert last commit and enable ignored tests.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1157 - updated to create 2 agents so that proper bonding can occur
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1162 - Updated test to mock the local peer PING packet creation so that the hash can be managed.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove p2p network code per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Updates from PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Spotless Apply fixes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Admin force cache refresh when called through end point per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Pr updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update changelog for 1.5.1
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove check for 0x prefix on addresses to match expectations
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update graphql pending to allow for sorting of transactions
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Added tests for new miner endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 - PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
Co-authored-by: David Mechler <davemec@users.noreply.github.com>
Make `TargetingGasLimitCalculator` respect the rule that a gas _delta_ can be no more than floor(previous_gas/1024)
Signed-off-by: Atkins <atkinschang@gmail.com>
Update the besu-native libraries to 0.3.0 to gain access to faster BN128
and secp256k1 implementations.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
The parallel task production is by sub-trie, so calling `visitAll` on a
root node will eventually spawn up to 16 tasks (for a hexary trie).
If we marked each sub-trie in its own thread, with no common queue of
tasks, our mark speed would be limited by the sub-trie with the maximum
number of nodes. In practice for the Ethereum mainnet, we see a large
imbalance in sub-trie size so without a common task pool the time in
which there is only 1 thread left marking its big sub-trie would be
substantial.
If we were to leave all threads to produce mark tasks before starting
to mark, we would run out of memory quickly.
If we were to have a constant number of threads producing the mark
tasks with the others consuming them, we would have to optimize the
production/consumption balance.
To get the best of both worlds, the marking executor has a
ThreadPoolExecutor.CallerRunsPolicy which causes the producing tasks to
essentially consume their own mark task immediately when the task queue
is full. The resulting behavior is threads that mark their own sub-trie
until they finish that sub-trie, at which point they switch to marking
the sub-trie tasks produced by another thread.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The warm address/slot data was not returned to the calling messageFrame
for CREATE[2] calls as it was for CALL series calls.
Also, improve performance by using more efficient copy constructors and
reusing objects were relevant.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
If a peer exceeds the authorized number of pending blocks, Besu will replace the lowest priority block in the cache from this peer by this new one until the local node sync a new block and maybe purges one of the blocks of this peer.
The highest priority blocks are those that are lowest in block height and then higher priority if they were sent more recently.
Other peers will not be impacted and will be able to continue sending pending blocks.
The cache size limit is the distance between the minimum and maximum value of the BlockPropagationRange parameter. Besu automatically purges blocks outside this range.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Danno Ferrin
Trent Mohay
Abdelhamid Bakhta
Ratan (Rai) Sur
Antoine Toulme
Stefan Pingel
matkt
mark-terry
Lucas Saldanha
Sally MacFarlane
Edward Mack
David Mechler
Atkins