The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Refactor crypto, datatypes, and plugin-api to reduce plugin APIs in the EVM.
Split crypto into crypto services and crypto algorithms
Reverse the dependency between datatyps and plugin-api.
Remove plugin Hash and Address types (use datatypes)
Move PublicKey and Quantity into datatypes.
Lots of changes to imports and build files, and some fromPlugin calls removed.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@shemnon.com>
Co-authored-by: Justin Florentine <justin+github@florentine.us>
Due to subtle differences between Bytes32 and UInt256 the changes failed
burn-in testing and are being reverted.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* Remove deprecated cleanup task
* Move util class into test-support
This is because this class can be used outside the scope of the junit tests but it'll not be part of main
* Move referenceTests into its own scope
* Migrate referenceTests to Junit5
I inlined `o.h.b.e.vm.AbstractRetryingTest` into `o.h.b.e.vm.VMReferenceTest` because there was a single usage of the abstraction and also because it wasn't portable to how Junit5 parameters work
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Danno Ferrin <danno.ferrin@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Upgrade Errorprone
Upgrade errorprone to 2.13.1. Like all errorprone upgrades there are
new checks requiring code changes.
* Unused methods now cause compilation errors
* fields must be static and final
* Effectively constant booleans must now be returned as true/false.
* longs should not auto-cast to double.
* turn off errorprone javadocs
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Fix two operand stack tests
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Remove unreachable assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace expected annotation field with assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Extract variables to let only a single call for assert
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* avoid the edge case of long minimum value which gives a negative value when passed into abs()
* changed some Longs to long where they were statically intialized
* replace Lists.emptyList() with Collections.emptyList()
* removed unnecessary eq() invocation and resulting code smell
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
PR that adds the different request tasks necessary for the snapsync as well as a utility to manage the ranges of requests
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* lots of errorprone fixes
* some license updates
* some mockito updates
* upgrade the rocksdb version
* Prometheus left at 0.9.0 as 0.10.0+ introduces OpenMetrics
related changes that break unit tests.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Upgrade to Apache Tuweni 2.0
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove intermediate repository
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove all occurrences of toBytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Migrate to tuweni-bytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* add changelog
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* correct reference tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Initial API changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* more changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Change APIs for VM ops
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Use constant UInt256.ONE
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Optimize a bit address <> word transformation
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* spotless
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Update Bonsai Tries persistence so they can calculate intermediate world
state for pre-byzantium transactions.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The current formulation of the decode can hide exceptions if the finally
block does not complete. Since exceptions are thrown mid-rlp processing
this typically results in the list not being fully consumed and the real
exception is hidden by a "not at end of list" exception.
This more verbose form prevents such exceptions from being overruled by
rlp processing errors.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
The parallel task production is by sub-trie, so calling `visitAll` on a
root node will eventually spawn up to 16 tasks (for a hexary trie).
If we marked each sub-trie in its own thread, with no common queue of
tasks, our mark speed would be limited by the sub-trie with the maximum
number of nodes. In practice for the Ethereum mainnet, we see a large
imbalance in sub-trie size so without a common task pool the time in
which there is only 1 thread left marking its big sub-trie would be
substantial.
If we were to leave all threads to produce mark tasks before starting
to mark, we would run out of memory quickly.
If we were to have a constant number of threads producing the mark
tasks with the others consuming them, we would have to optimize the
production/consumption balance.
To get the best of both worlds, the marking executor has a
ThreadPoolExecutor.CallerRunsPolicy which causes the producing tasks to
essentially consume their own mark task immediately when the task queue
is full. The resulting behavior is threads that mark their own sub-trie
until they finish that sub-trie, at which point they switch to marking
the sub-trie tasks produced by another thread.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Experimental backup and restore via `operator x-backup-state` and `operator x-restore-state` CLI commands. Besu needs to be in a non-operational state for the backup and restore to occur. Restore has only been tested into an empty database, not on top of an existing database.
This feature is not advised for production use at the moment.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Mark Sweep Pruner
* add `unload` method on Node interface, which is a noop everywhere but on `StoredNode`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Put and remove are currently available on the KeyValueStorage interface
as non-transactional calls. These are used nowhere other than test
code, all our production write calls are to the transactional interface.
* Remove the non-transactional mutation APIs from the interface
* Update the tests to use transactional writes
* Update getStartTransaction to just be startTransaction since it is
not a property but a method.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Upgrade errorprone
* Upgrade errorprone from 2.3.1 to 2.3.2
* Upgrade Jenkinsfile so that CI will use Java 11
* Suppress these new rules:
* EqualsGetClass
* ImmutableEnumChecker
* UnnecessaryParentheses
* Change code to conform to these new rules:
* BadImport
* BadInstanceof
* InconsistentHashCode
* LockNotBeforeTry
* MathAbsoluteRandom
* ModifiedButNotUsed
* UndefinedEquals
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Nischal Sharma
matkt
Danno Ferrin
garyschulte
lukelee-sl
Justin Traglia
Danno Ferrin
Diego López León
Sally MacFarlane
Antoine Toulme
Ratan (Rai) Sur
Antoine Toulme
Joshua Fernandes
CJ Hare
Karim T
mbaxter
Ivaylo Kirilov
Danno Ferrin
Adrian Sutton
mark-terry
Christopher Mckay
PegaSys Admin