Adds integration tests to catch bug where the mark storage was being cleared before a mark began. Instead, the mark storage is now cleared upon preparing the mark sweep pruner
Fixes bug where the pending marks where not being checked while pruning was occurring. By removing the flush in sweepBefore, the existing tests catch the bug.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Ensure `plugin-api` module gets published at the correct maven path
* Move `plugins` to `plugin-api`
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Return plugin-api to the main repo
* Spotless
* Migrate all external plugin-api references to the project in this repo
* Add licence header
* Update repo reference for publish, even if commented
* Use real configuration for publishing plugin-api
This was tested with the
`:plugins:publishMavenJavaPublicationToMavenLocal` task and checking the
local Maven repo to make sure it was using the correct paths
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Rich Data for Events Plugin
* plugin-api -> api
* use BinaryData as a root and add getValue to UInt256Value
* bring rick data changes in line with proposed APIs.
Undo orthagonal naming changes.
* add size
* use log and transaction interfaces from rich data api
* update to released plugin api version and add new event listener.
* add tests
* fix acceptance tests
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* sweep state roots before child nodes
* Adds long removeAccountStateTrieNode(key) method to WorldStateStorage.Updater
* remove assertj assertions from `AbstractKeyValueStorageTest`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Database Versioning: The behavior is to load the database at the existing version if it
already exists or create the newest version if it doesn't
* multi-column by default: This makes the separated world state storage column required by mark sweep on by default
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Mark Sweep Pruner
* add `unload` method on Node interface, which is a noop everywhere but on `StoredNode`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PAN-2853] Create a metric tracking DB size
- create a Prometheus long gauge to track the database size
- `rocksdb.live-sst-files-size` : 872a261ffc/include/rocksdb/db.h (L685)
- use `rocksdb.live-sst-files-size` property rather than `rocksdb.total-sst-files-size` (performance purpose: see warning 872a261ffc/include/rocksdb/db.h (L680))
- add tests to check creation of metrics
* fix wildcard import
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Columnated storage to allow for iteration over world state
* change MetricsCategory to PantheonMetricsCategory
* consistency renaming of kvstores
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Default values are currently unchanged but increasing both these values to 4 appears to improve sync performance for both ropsten and Mainnet fast syncs.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Use labelled timer to differentiate between metrics of public and private db
* Update the label names
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds handling for options that need to be configurable for tuning and performance but in general are not long-term supported configuration options.
Options that are added via a mixin, are hidden, and that start with `--X` are considered 'Unstable Options.' These options do not show up via normal help facilities but are enumerated via a `XHelp` CLI invocation. These options are unstable, unsupported, and undocumented. They may be added, removed, or modified without prior warning or notice. Use them at your own risk.
For example, the `--Xrocksdb-max-open-files` flag can be set via CLI and can increase the number of max open files RocsDB uses. On a 8GB memory system the default is appropriate, but on a 128GB memory system a much larger number may improve performance. Normal operation scenarios will not need to modify this flag.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Biggest change is that UnusedVariable and UnusedMethod went to WARN by
default. Since our build is a no warning build this means we either need
to turn them off or fix them. I mostly opted for the latter. Test code
was mostly fixed, unused loggers were deleted, and other shipped code
was mostly suppressed.
Two less noisy fixes to not use `SortedSet` and to use zero based
comparable results instead of -1, 0, and 1. Also a compiler nit in
errorprone was suppressed, per the description it won't affect us.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Two changes:
* Stream chains now take up less vertical lines, only breaking on
stream operations.
* Long annotations that span multiple lines no longer have a dangling
parentesis and indent 4 spaces.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Put and remove are currently available on the KeyValueStorage interface
as non-transactional calls. These are used nowhere other than test
code, all our production write calls are to the transactional interface.
* Remove the non-transactional mutation APIs from the interface
* Update the tests to use transactional writes
* Update getStartTransaction to just be startTransaction since it is
not a property but a method.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Upgrade errorprone
* Upgrade errorprone from 2.3.1 to 2.3.2
* Upgrade Jenkinsfile so that CI will use Java 11
* Suppress these new rules:
* EqualsGetClass
* ImmutableEnumChecker
* UnnecessaryParentheses
* Change code to conform to these new rules:
* BadImport
* BadInstanceof
* InconsistentHashCode
* LockNotBeforeTry
* MathAbsoluteRandom
* ModifiedButNotUsed
* UndefinedEquals
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Upgrade dependency versions
* All version information is merged into one place
* Upgrade most version to their latest non-test version.
* dependencies are now sorted
Two dependencies were not upgraded:
errorprone - There are new checks that require build or source changes.
vertex - The license to eclipse 2.0 which is not in our current approved license list.
The change is trivial but for tracability it should be done alone.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Update value returned by web3_clientVersion to be ethstats friendly
* Version part starts with a v
* SNAPSHOT builds report 32 bits of the git hash
* OS and architecture are sniffed out and reported
* JVM version and branding are sniffed out and reported
Example values (not all real):
pantheon/v0.9.0-dev-f800a0b1/osx-x86_64/oracle-java-1.8
pantheon/v0.9.0-dev-27960b57/osx-x86_64/zulu-java-11
pantheon/v0.9.0/linux-arm64/openjdk-java-12
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Replace explicit usages of RocksDbKeyValueStorage with the more generic but now just as capable KeyValueStorage.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
- Remove custom Rocksdbjni and use upgraded version from rocksdb
- Remove custom picocli lib and use the upgraded version from pico
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Ivaylo Kirilov
Ratan Rai Sur
Joshua Fernandes
CJ Hare
Edward
mark-terry
Jason Frame
Danno Ferrin
mbaxter
Abdelhamid Bakhta
Adrian Sutton
Adrian Sutton
Danno Ferrin
Puneetha Karamsetty
Christopher Mckay
Eric Kellstrand
S. Matthew English
PegaSys Admin