Adding the Log4j "jul" (java.util.logging) adapter resulted in many
messages like this at startup:
`main INFO Registered Log4j as the java.util.logging.LogManager.`
These come from the Log4j status logger. We can get rid of those by
setting the status attribute on all configurations to a higher logging
level. WARN is the next higher level.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Add `memory` as an option for `--key-value-storage`. This is useful in
small network synchronization tests as memory is faster and is easier to
inspect via a debugger.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* add doomed key check (busy-waiting for now)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* optional and logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* sleeping and hardening
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename segments
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move away from atomic references to regular vars
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove hardened segment parameter
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* increase sleep
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* spotless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary interface
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move commit waiting outside of timer
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "rename segments"
This reverts commit 184eefaaa0ccc857b0caff2b382f8338ff225d5d.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add documentation
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* bump up sleep to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* (POC) Add lock to ensure that we don't prune while comitting
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary persist (#569)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* flesh out @mbaxter's idea and remove my code
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* iterator changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* hybridize with doomed key
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move doomed key unset to after node added listener
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* update instead of getting and setting doomedKeyRef in commit
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* invert condition
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove locks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove `removeAllKeysUnless`
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* more remove removeAllKeysUnless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* reuse streamKeys
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove test
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert back to locks instead of doomedkey
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* change delete to not guarantee deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* plugin hash
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "change delete to not guarantee deletion"
This reverts commit 2289bb34cf.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* skip key deletion on timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* clear in rollback
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "fix jmh compilation error"
This reverts commit b64ecf8656.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "add default lock timeout to tests"
This reverts commit aff6aa6065.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "set default lock timeout to 1ms"
This reverts commit 267fe0a642.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use noSlowDown write option instead of global timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* close tryDeleteOptions
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary lock
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move increment inside try
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use StorageException subclass instead of field
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert accidental deletion in javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tryDelete javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add trace for skipping key deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* merge catch and finally try blocks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* switch from exception to boolean return value
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tweak
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* changelog changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add api back
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back throws javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Co-authored-by: Meredith Baxter <meredith.baxter@consensys.net>
Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>
Removes as many Gradle 7.0 compatibility issues as possible
* `baseName` -> `archiveBaseName`
* `extension` -> `archiveExtension`
* `destinationDir` -> `destinationDirectory`
* `runtime` -> `runtimeOnly`
* Change some log4j-api and log4j-core dependencies
* Remove an unneeded and outdated plugin (`net.ltgt.apt`)
* tweak the plugin-api change detector's property annotations.
Warnings still exist with one external plugin used for license file
checking that we do not control the source code for.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Update dependencies to most current version
- except picocli which is a major version update
Alphabetize dependencies
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Adds integration tests to catch bug where the mark storage was being cleared before a mark began. Instead, the mark storage is now cleared upon preparing the mark sweep pruner
Fixes bug where the pending marks where not being checked while pruning was occurring. By removing the flush in sweepBefore, the existing tests catch the bug.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Ensure `plugin-api` module gets published at the correct maven path
* Move `plugins` to `plugin-api`
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Return plugin-api to the main repo
* Spotless
* Migrate all external plugin-api references to the project in this repo
* Add licence header
* Update repo reference for publish, even if commented
* Use real configuration for publishing plugin-api
This was tested with the
`:plugins:publishMavenJavaPublicationToMavenLocal` task and checking the
local Maven repo to make sure it was using the correct paths
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Rich Data for Events Plugin
* plugin-api -> api
* use BinaryData as a root and add getValue to UInt256Value
* bring rick data changes in line with proposed APIs.
Undo orthagonal naming changes.
* add size
* use log and transaction interfaces from rich data api
* update to released plugin api version and add new event listener.
* add tests
* fix acceptance tests
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* sweep state roots before child nodes
* Adds long removeAccountStateTrieNode(key) method to WorldStateStorage.Updater
* remove assertj assertions from `AbstractKeyValueStorageTest`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Database Versioning: The behavior is to load the database at the existing version if it
already exists or create the newest version if it doesn't
* multi-column by default: This makes the separated world state storage column required by mark sweep on by default
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Mark Sweep Pruner
* add `unload` method on Node interface, which is a noop everywhere but on `StoredNode`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PAN-2853] Create a metric tracking DB size
- create a Prometheus long gauge to track the database size
- `rocksdb.live-sst-files-size` : 872a261ffc/include/rocksdb/db.h (L685)
- use `rocksdb.live-sst-files-size` property rather than `rocksdb.total-sst-files-size` (performance purpose: see warning 872a261ffc/include/rocksdb/db.h (L680))
- add tests to check creation of metrics
* fix wildcard import
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Columnated storage to allow for iteration over world state
* change MetricsCategory to PantheonMetricsCategory
* consistency renaming of kvstores
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Default values are currently unchanged but increasing both these values to 4 appears to improve sync performance for both ropsten and Mainnet fast syncs.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Use labelled timer to differentiate between metrics of public and private db
* Update the label names
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds handling for options that need to be configurable for tuning and performance but in general are not long-term supported configuration options.
Options that are added via a mixin, are hidden, and that start with `--X` are considered 'Unstable Options.' These options do not show up via normal help facilities but are enumerated via a `XHelp` CLI invocation. These options are unstable, unsupported, and undocumented. They may be added, removed, or modified without prior warning or notice. Use them at your own risk.
For example, the `--Xrocksdb-max-open-files` flag can be set via CLI and can increase the number of max open files RocsDB uses. On a 8GB memory system the default is appropriate, but on a 128GB memory system a much larger number may improve performance. Normal operation scenarios will not need to modify this flag.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Biggest change is that UnusedVariable and UnusedMethod went to WARN by
default. Since our build is a no warning build this means we either need
to turn them off or fix them. I mostly opted for the latter. Test code
was mostly fixed, unused loggers were deleted, and other shipped code
was mostly suppressed.
Two less noisy fixes to not use `SortedSet` and to use zero based
comparable results instead of -1, 0, and 1. Also a compiler nit in
errorprone was suppressed, per the description it won't affect us.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Two changes:
* Stream chains now take up less vertical lines, only breaking on
stream operations.
* Long annotations that span multiple lines no longer have a dangling
parentesis and indent 4 spaces.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Put and remove are currently available on the KeyValueStorage interface
as non-transactional calls. These are used nowhere other than test
code, all our production write calls are to the transactional interface.
* Remove the non-transactional mutation APIs from the interface
* Update the tests to use transactional writes
* Update getStartTransaction to just be startTransaction since it is
not a property but a method.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Upgrade errorprone
* Upgrade errorprone from 2.3.1 to 2.3.2
* Upgrade Jenkinsfile so that CI will use Java 11
* Suppress these new rules:
* EqualsGetClass
* ImmutableEnumChecker
* UnnecessaryParentheses
* Change code to conform to these new rules:
* BadImport
* BadInstanceof
* InconsistentHashCode
* LockNotBeforeTry
* MathAbsoluteRandom
* ModifiedButNotUsed
* UndefinedEquals
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Upgrade dependency versions
* All version information is merged into one place
* Upgrade most version to their latest non-test version.
* dependencies are now sorted
Two dependencies were not upgraded:
errorprone - There are new checks that require build or source changes.
vertex - The license to eclipse 2.0 which is not in our current approved license list.
The change is trivial but for tracability it should be done alone.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Danno Ferrin
Antoine Toulme
Yehia Tarek
Ratan Rai Sur
Ivaylo Kirilov
Joshua Fernandes
CJ Hare
Edward
mark-terry
Jason Frame
mbaxter
Abdelhamid Bakhta
Adrian Sutton
Adrian Sutton
Danno Ferrin
Puneetha Karamsetty