* Include Commit-Hash in manifests
* Add commit method in BesuInfo
* Use 8 character hash in EngineGetClientVersionV1 and revert calculateVersion back to original spec
Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Co-authored-by: garyschulte <garyschulte@gmail.com>
* Create a BFT-specific pivot block handler
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Change visibility
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor sync-peer-count internal variable to match name, add experimental flag to enabled snap + BFT
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Merge with main
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix uppercase
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Address synchronization issue with trie pruning. Create BFT-specific account range handler. Add pipeline name and logs
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove debug log
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* fixing snapsync for empty state
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
* Don't queue up events we can't handle
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix timing window where a validator with an empty data dir sometimes falls back to full sync if peer status isn't received quickly enough
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove BFT-specific account request class. Not needed
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor some more 'fast' sync variables that are common to all pivot-based sync types
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* In FULL sync mode, disable bonsai-limit-trie-logs-enabled instead of failing to start
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Add javadoc comments, clarify overriding bonsai-limit-trie-logs-enabled
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Add BFT pivot block selector tests
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix failure error message
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove the unnamed Pipe constructor and update tests to set a pipe name
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Revert some info logs back to debug given the feedback on noise in the logs syncing with holesky
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor fastSyncPivotDistance to syncPivotDistance
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Incomplete refactoring
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Update BFT event queueing tests
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Event queue test fixes
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove automatic setting of bonsai-limit-trie-logs-enabled to false if sync-mode = FULL (moving to another PR)
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
---------
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>
Co-authored-by: Karim Taam <karim.t2am@gmail.com>
Upgrade spotless to 1.22.0 and reformat.
This is required for Java21 support.
Signed-off-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Move Besu custom error-prone checks into its own repository and use it as an external dependency. This allows to move to a newer version of Google errorprone checks as well while cleaning up build.gradle file.
Key changes resulted due to this change:
* String toLowerCase and toUpperCase to use Locale.ROOT as argument
* Use interface such as List,Map or NavigatableMap instead of concrete class where appropriate.
* Simplify StringBuilder to plain String
* Suppress warnings where appropriate.
-----
Signed-off-by: Usman Saleem <usman@usmans.info>
The purpose of this commit is to fix some bugs detected in the snap sync. This should allow for greater stability with the snap sync and the healing of the flat DB.
---------
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
Upgrade bouncy castle to v1.75. This involved a change in maven
coordinates for other modules.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* update to 2.4.1
* update use of DNS daemon with Vertx
* fix issue with Bytes.repeat
* update antlr version
* fix dns tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
---------
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Refactor crypto, datatypes, and plugin-api to reduce plugin APIs in the EVM.
Split crypto into crypto services and crypto algorithms
Reverse the dependency between datatyps and plugin-api.
Remove plugin Hash and Address types (use datatypes)
Move PublicKey and Quantity into datatypes.
Lots of changes to imports and build files, and some fromPlugin calls removed.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@shemnon.com>
Co-authored-by: Justin Florentine <justin+github@florentine.us>
Due to subtle differences between Bytes32 and UInt256 the changes failed
burn-in testing and are being reverted.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Make the ethereum/trie and services/kvstore modules as slim as possible
by dropping dependencies.
* The ':util' api dependency is removed from kvstore and added to
eth module
* Move tries's kvstore dependency to test
* Other dependencies are removed as unused from trie and kvstore
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Remove deprecated cleanup task
* Move util class into test-support
This is because this class can be used outside the scope of the junit tests but it'll not be part of main
* Move referenceTests into its own scope
* Migrate referenceTests to Junit5
I inlined `o.h.b.e.vm.AbstractRetryingTest` into `o.h.b.e.vm.VMReferenceTest` because there was a single usage of the abstraction and also because it wasn't portable to how Junit5 parameters work
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Danno Ferrin <danno.ferrin@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Upgrade Errorprone
Upgrade errorprone to 2.13.1. Like all errorprone upgrades there are
new checks requiring code changes.
* Unused methods now cause compilation errors
* fields must be static and final
* Effectively constant booleans must now be returned as true/false.
* longs should not auto-cast to double.
* turn off errorprone javadocs
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Fix two operand stack tests
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Remove unreachable assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace expected annotation field with assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Extract variables to let only a single call for assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* avoid the edge case of long minimum value which gives a negative value when passed into abs()
* changed some Longs to long where they were statically intialized
* replace Lists.emptyList() with Collections.emptyList()
* removed unnecessary eq() invocation and resulting code smell
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Change the unit test execution to use the Junit5 JUnitPlatform. This
allows for a mix of junit 4 and junit 5 tests and for a gradual
migration to junit 5 instead of a big bang. One class depended on
junit 4 exceptions and was updated. Two tests depending on
native libraries fail gracefully on mac (and only mac).
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
PR that adds the different request tasks necessary for the snapsync as well as a utility to manage the ranges of requests
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* lots of errorprone fixes
* some license updates
* some mockito updates
* upgrade the rocksdb version
* Prometheus left at 0.9.0 as 0.10.0+ introduces OpenMetrics
related changes that break unit tests.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Upgrade to Apache Tuweni 2.0
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove intermediate repository
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove all occurrences of toBytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Migrate to tuweni-bytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* add changelog
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* correct reference tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Initial API changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* more changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Change APIs for VM ops
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Use constant UInt256.ONE
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Optimize a bit address <> word transformation
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* spotless
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Update Bonsai Tries persistence so they can calculate intermediate world
state for pre-byzantium transactions.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Add tracing support for internals and JSON-RPC
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove rocksdb tracing as it slows down execution too much
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add B3 headers extraction on JSON-RPC requests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove traces around trie tree as they slow down syncing significantly
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing to fast sync pipeline
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing for all pipelines
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Address code review
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add acceptance tests and break out the shaded dependency
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Fix tracer id
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Revert changes to trie
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Upgrade otel to latest, remove old tracing
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review comments
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The current formulation of the decode can hide exceptions if the finally
block does not complete. Since exceptions are thrown mid-rlp processing
this typically results in the list not being fully consumed and the real
exception is hidden by a "not at end of list" exception.
This more verbose form prevents such exceptions from being overruled by
rlp processing errors.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
The parallel task production is by sub-trie, so calling `visitAll` on a
root node will eventually spawn up to 16 tasks (for a hexary trie).
If we marked each sub-trie in its own thread, with no common queue of
tasks, our mark speed would be limited by the sub-trie with the maximum
number of nodes. In practice for the Ethereum mainnet, we see a large
imbalance in sub-trie size so without a common task pool the time in
which there is only 1 thread left marking its big sub-trie would be
substantial.
If we were to leave all threads to produce mark tasks before starting
to mark, we would run out of memory quickly.
If we were to have a constant number of threads producing the mark
tasks with the others consuming them, we would have to optimize the
production/consumption balance.
To get the best of both worlds, the marking executor has a
ThreadPoolExecutor.CallerRunsPolicy which causes the producing tasks to
essentially consume their own mark task immediately when the task queue
is full. The resulting behavior is threads that mark their own sub-trie
until they finish that sub-trie, at which point they switch to marking
the sub-trie tasks produced by another thread.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Experimental backup and restore via `operator x-backup-state` and `operator x-restore-state` CLI commands. Besu needs to be in a non-operational state for the backup and restore to occur. Restore has only been tested into an empty database, not on top of an existing database.
This feature is not advised for production use at the moment.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Removes as many Gradle 7.0 compatibility issues as possible
* `baseName` -> `archiveBaseName`
* `extension` -> `archiveExtension`
* `destinationDir` -> `destinationDirectory`
* `runtime` -> `runtimeOnly`
* Change some log4j-api and log4j-core dependencies
* Remove an unneeded and outdated plugin (`net.ltgt.apt`)
* tweak the plugin-api change detector's property annotations.
Warnings still exist with one external plugin used for license file
checking that we do not control the source code for.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Update dependencies to most current version
- except picocli which is a major version update
Alphabetize dependencies
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Matilda-Clerke
Sally MacFarlane
Matt Whitehead
garyschulte
Usman Saleem
Danno Ferrin
Karim TAAM
Nischal Sharma
Antoine Toulme
Danno Ferrin
lukelee-sl
Justin Traglia
Diego López León
Sally MacFarlane
Antoine Toulme
Ratan (Rai) Sur
Joshua Fernandes