It was identified that when both KV_ROCKSDB metrics and Privacy were enabled,
that Besu failed to start due to a naming collision in Metrics Collators - this was ultimately due to Besu having 2 keyvalue stores - one for public state, and another for private state - and _both_ using the same metrics.
To overcome this issue, the metrics used in the private kv store are prefixed with the word "private".
Signed-off-by: Trent Mohay <trent.mohay@consensys.net>
[PIE-1858] Added functionality to register custom metrics categories and exposed some PoA data for metrics.
Signed-off-by: Mark Terry <mark.terry@consensys.net>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Remove metrics from plugin registration. Doing it this way here always results in a
NoOpsMetrics being inited because CLI options have not been parsed.
Add an acceptance test for metrics. This is what should have been in place and
would have detected metrics not working earlier.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Ensure `plugin-api` module gets published at the correct maven path
* Move `plugins` to `plugin-api`
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Return plugin-api to the main repo
* Spotless
* Migrate all external plugin-api references to the project in this repo
* Add licence header
* Update repo reference for publish, even if commented
* Use real configuration for publishing plugin-api
This was tested with the
`:plugins:publishMavenJavaPublicationToMavenLocal` task and checking the
local Maven repo to make sure it was using the correct paths
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Some auto-generated maven names cause collissions when used with the gradle
publish plugin, namely `core` and `util`. Rename two jars that have such
a colission so the colission doesn't occur.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Mark Sweep Pruner
* add `unload` method on Node interface, which is a noop everywhere but on `StoredNode`
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Display World State Sync Progress in Logs
- compute estimated world state completion
- display estimated world state completion in the logs
- `CompleteTaskStep` now have access to a `LongSupplier` to retireve the number of pending requests
- use a `RunnableCounter` to trigger displaying every 1000 requests completed
- only show a new log when the estimation changes
- added test to check the estimation computation
* report using raw ratio rather than percentage
* resolve PR discussion
* rename getTotal to get
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* use MockitoJUnitRunner to run TransactionsMessageProcessor tests
* implement metrics counter and logging
* (format) replace statement by lambda expression
* (doc) fix javadoc
* spotless apply
* remove modulo and reset step counter to zero
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Make MetricCategory an interface so it isn't locked into the pantheon specific categories.
* Split categories into StandardMetricCategory and PanteonMetricCategory to separate the common and Pantheon specific categories.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Don't copy collections if we don't need to. Change types higher up if needed.
* Don't use Guava's Object.equal, use Java's Objects.equals.
** add errorprone test to enforce the banning of Guava's Objects class.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Change all Stream<?> getX() and Stream<?> x() methods to Stream<?> streanX methods, such as `Stream<Peer> streamIdlePeers()`
* Update coding conventions to reflect this.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Number of metrics labels need to match up with constructor
* Number of labels must be consistant, so I split it into two metrics
* Also, naming best practices say that sum() and avg() of a metric
should be meaningful, separating into two metrics fixes that.
* fix style issues (finals, intellij warnings)
* Change NoOpMetrics to check label count.
* Cascading changes to support this in many support classes. Mostly places
we presumed all NoOpMetrics were equals.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Added ability to subscribe to dropped transactions from the transaction pending pool.
Implemented subscription webservice to support this.
* Added metrics to the pending transactions, tracking the number of local and remote transactions in the pool.
* Converted listener management in pending transactions to use the Subscribers util object.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Testing has shown that RocksDB and BigQueue metrics has a negative
impact on performance under load. Disable them by default.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
We are growing our list of metrics by the day, and some of them we don't
always need. So we should provide a way to trim them.
* If metrics are not enabled we supply a no-op metrics
* If only select metric categories are enabled we supply real metrics
for the selected metrics and no-op metrics for all the rest
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Instantiate individual metrics where they're used
* Cache prometheus metrics to allow "duplicate" creation of metrics
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Two changes:
* Stream chains now take up less vertical lines, only breaking on
stream operations.
* Long annotations that span multiple lines no longer have a dangling
parentesis and indent 4 spaces.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Metrics Service configuration incorrect
--metrics-push-enabled is not eabling metrics push and --metrics-enabled
is enabling push instead of polling.
* formatting
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Don't overload `--metrics-host` and `--metrics-port` for push mode, instead add '--metrics-push-enabled' , `--metrics-push-host` and `--metrics-push-port`, rename `--metrics-prometheus-job` to `--metrics-push-prometheus-job` and remove `--metrics-mode`
* Show an error if both metrics and metrics-push are enabled.
* Allow shutdown if we cannot communicate with the Push Gateway.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Sometimes metrics are hard to poll (docker containers with varying ip
addresses). Because of that the push gateway exists. This extends the
metrics system to support push or pull mode for metrics (but not both
at the same time).
Three new flags
`--metrics-mode=`<`push`|`pull`> - Whether we are in pull mode (the default) where
prometheus is expected to poll or push mode where pantheon pushes to
a push gateway.
`--metrics-push-interval=`<_integer_> the frequency, in seconds, between pushes to
the push gateway. Only relevant in push mode
`--metrics-prometheus-job=`<_string_> The name of the job to report in the push gateway
Also, `--metrics-host=` and `--metrics-port=` gain new meaning in push mode. Instead of the
server they are opening up it is the host and the port of the push gateway it should push to.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Trent Mohay
Lucas Saldanha
mark-terry
Joshua Fernandes
CJ Hare
Danno Ferrin
Edward
Ratan Rai Sur
mbaxter
Adrian Sutton
Abdelhamid Bakhta
Chris Mckay
Danno Ferrin
Rob Dawson
Adrian Sutton
Chris Mckay