Upgrade spotless to 1.22.0 and reformat.
This is required for Java21 support.
Signed-off-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Improve the high spec flag, limit it to few column families
* Update changelog
* spotless
* Update the plugin API hash as one of the interfaces was changed\
* Fix failing unit tests
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* Update reference tests to 12.4
Update reference tests to 12.4
* Some in-memory storage changed to ConcurrentMap
* exclude cancun from all EIP tests, EIP-4788 still in flux
* Add new fields to ReferenceTestEnv, and re-order the reflected
constructor for clarity and ease of development.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* spotless
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
---------
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
* create a bonsai based reference test worldstate
-> getOrCreate in BonsaiWorldStateUpdateAccumulator - do not throw if we discover an empty account in a non-null BonsaiValue<Account>
-> add curentStateRoot to t8n
-> storageEntriesFrom and streamAccounts implemented in BonsaiWorldStateKeyValueStorage
-> add endKey version of streamFromKey
* bonsai fix for self-destruct and create2 at the same address and same block
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Karim TAAM <karim.t2am@gmail.com>
* services to junit5
* removed some junit4 engine imports
* updated some plugins test since these extend from testutil KV storage
* one more form of EPL v2
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
* Refactor to make RocksDBColumnarKeyValueStorage abstract and extend it to optimistic and pessimistic. Atm Forest has a concurrency level that does not cope well with the OptimisticTransactionDB and RocksDB ends up raising a lot of Busy when committing RocksDBTransactions.
A TransactionDB will do conflict checking for all write operations (Put, Delete and Merge), including writes performed outside a Transaction according to RocksDB. This does impact the times we see when syncing so likely not the final solution for Forest.
Bonsai should continue using OptimisticTransactionDB.
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
---------
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
- Added missing javadocs so that javadoc doclint passes against JDK 17 (invoke by Besu gradle build).
- Exclude following packages from javadoc lint:
org.hyperledger.besu.privacy.contracts.generated
org.hyperledger.besu.tests.acceptance.*
- Temporarily exclude ethereum and evm submodule for doc lint checks.
- Run the javadoc task using GitHub actions (use Java 17) to report any javadoc errors during the PR builds
- Updating plugin-api build.gradle with new hash as javadoc comments caused it to change
Signed-off-by: Usman Saleem <usman@usmans.info>
* use optimistictransactiondb for mutable isolated snapshots
* plumbing necessary to have a snapshot specific updater.
* snapshot rolling working
* implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots
* add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate
Signed-off-by: garyschulte <garyschulte@gmail.com>
* update rocksdbjni to v6.29.5 for compatibility with arm64 CPU
Signed-off-by: Pedro Novais <jpvnovais@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
* Bump SLF4J version
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace log4j2 API with SLF4j API
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace usage of LogManager#getFormatterLogger
This is for keeping compatibility with SLF4J. If neccesary, a specific formatter can be created for the RlpBlockImporter class
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Unset the default logging value for the retesteth
This is because it's not possible to resolve the root logger level into a Log4J2 field
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Prevent creation of Logger context outside SLF4J
org.hyperledger.besu.cli.BesuCommand#setAllLevels was taken from
https://github.com/apache/logging-log4j2/blob/rel%2F2.17.1/log4j-core/src/main/java/org/apache/logging/log4j/core/config/Configurator.java#L309
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add FATAL level deprecation message
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S2139
Exceptions should be either logged or rethrown but not both
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S3457
Printf-style format strings should be used correctly
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add changelog
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Remove all but 4 log4j2.xml config files
* The main config for the besu CLI app
* The config for the evmTool CLI app
* The config for acceptance tests
* A config in testUtil
If any tests depend on a log4j file they should import testutil.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Adding the Log4j "jul" (java.util.logging) adapter resulted in many
messages like this at startup:
`main INFO Registered Log4j as the java.util.logging.LogManager.`
These come from the Log4j status logger. We can get rid of those by
setting the status attribute on all configurations to a higher logging
level. WARN is the next higher level.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Add `memory` as an option for `--key-value-storage`. This is useful in
small network synchronization tests as memory is faster and is easier to
inspect via a debugger.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* add doomed key check (busy-waiting for now)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* optional and logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* sleeping and hardening
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename segments
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move away from atomic references to regular vars
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove hardened segment parameter
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* increase sleep
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* spotless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary interface
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move commit waiting outside of timer
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "rename segments"
This reverts commit 184eefaaa0ccc857b0caff2b382f8338ff225d5d.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add documentation
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* bump up sleep to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* (POC) Add lock to ensure that we don't prune while comitting
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary persist (#569)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* flesh out @mbaxter's idea and remove my code
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* iterator changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* hybridize with doomed key
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move doomed key unset to after node added listener
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* update instead of getting and setting doomedKeyRef in commit
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* invert condition
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove locks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove `removeAllKeysUnless`
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* more remove removeAllKeysUnless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* reuse streamKeys
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove test
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert back to locks instead of doomedkey
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* change delete to not guarantee deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* plugin hash
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "change delete to not guarantee deletion"
This reverts commit 2289bb34cf.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* skip key deletion on timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* clear in rollback
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "fix jmh compilation error"
This reverts commit b64ecf8656.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "add default lock timeout to tests"
This reverts commit aff6aa6065.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "set default lock timeout to 1ms"
This reverts commit 267fe0a642.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use noSlowDown write option instead of global timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* close tryDeleteOptions
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary lock
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move increment inside try
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use StorageException subclass instead of field
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert accidental deletion in javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tryDelete javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add trace for skipping key deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* merge catch and finally try blocks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* switch from exception to boolean return value
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tweak
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* changelog changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add api back
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back throws javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Co-authored-by: Meredith Baxter <meredith.baxter@consensys.net>
Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>
Adds integration tests to catch bug where the mark storage was being cleared before a mark began. Instead, the mark storage is now cleared upon preparing the mark sweep pruner
Fixes bug where the pending marks where not being checked while pruning was occurring. By removing the flush in sweepBefore, the existing tests catch the bug.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* sweep state roots before child nodes
* Adds long removeAccountStateTrieNode(key) method to WorldStateStorage.Updater
* remove assertj assertions from `AbstractKeyValueStorageTest`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Database Versioning: The behavior is to load the database at the existing version if it
already exists or create the newest version if it doesn't
* multi-column by default: This makes the separated world state storage column required by mark sweep on by default
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Mark Sweep Pruner
* add `unload` method on Node interface, which is a noop everywhere but on `StoredNode`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PAN-2853] Create a metric tracking DB size
- create a Prometheus long gauge to track the database size
- `rocksdb.live-sst-files-size` : 872a261ffc/include/rocksdb/db.h (L685)
- use `rocksdb.live-sst-files-size` property rather than `rocksdb.total-sst-files-size` (performance purpose: see warning 872a261ffc/include/rocksdb/db.h (L680))
- add tests to check creation of metrics
* fix wildcard import
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Columnated storage to allow for iteration over world state
* change MetricsCategory to PantheonMetricsCategory
* consistency renaming of kvstores
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Gabriel-Trintinalia
Karim Taam
Usman Saleem
Danno Ferrin
garyschulte
ahamlat
Danno Ferrin
Sally MacFarlane
Gabriel Fukushima
Fabio Di Fabio
Pedro Novais
Diego López León
Yehia Tarek
Ratan Rai Sur
Antoine Toulme
Ivaylo Kirilov
Joshua Fernandes
CJ Hare
mark-terry
Jason Frame
mbaxter
Abdelhamid Bakhta