* Create a BFT-specific pivot block handler
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Change visibility
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor sync-peer-count internal variable to match name, add experimental flag to enabled snap + BFT
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Merge with main
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix uppercase
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Address synchronization issue with trie pruning. Create BFT-specific account range handler. Add pipeline name and logs
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove debug log
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* fixing snapsync for empty state
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
* Don't queue up events we can't handle
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix timing window where a validator with an empty data dir sometimes falls back to full sync if peer status isn't received quickly enough
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove BFT-specific account request class. Not needed
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor some more 'fast' sync variables that are common to all pivot-based sync types
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* In FULL sync mode, disable bonsai-limit-trie-logs-enabled instead of failing to start
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Add javadoc comments, clarify overriding bonsai-limit-trie-logs-enabled
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Add BFT pivot block selector tests
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Fix failure error message
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove the unnamed Pipe constructor and update tests to set a pipe name
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Revert some info logs back to debug given the feedback on noise in the logs syncing with holesky
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Refactor fastSyncPivotDistance to syncPivotDistance
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Incomplete refactoring
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Update BFT event queueing tests
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Event queue test fixes
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
* Remove automatic setting of bonsai-limit-trie-logs-enabled to false if sync-mode = FULL (moving to another PR)
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
---------
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>
Co-authored-by: Karim Taam <karim.t2am@gmail.com>
* services to junit5
* removed some junit4 engine imports
* updated some plugins test since these extend from testutil KV storage
* one more form of EPL v2
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
---------
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
- Added missing javadocs so that javadoc doclint passes against JDK 17 (invoke by Besu gradle build).
- Exclude following packages from javadoc lint:
org.hyperledger.besu.privacy.contracts.generated
org.hyperledger.besu.tests.acceptance.*
- Temporarily exclude ethereum and evm submodule for doc lint checks.
- Run the javadoc task using GitHub actions (use Java 17) to report any javadoc errors during the PR builds
- Updating plugin-api build.gradle with new hash as javadoc comments caused it to change
Signed-off-by: Usman Saleem <usman@usmans.info>
Change the unit test execution to use the Junit5 JUnitPlatform. This
allows for a mix of junit 4 and junit 5 tests and for a gradual
migration to junit 5 instead of a big bang. One class depended on
junit 4 exceptions and was updated. Two tests depending on
native libraries fail gracefully on mac (and only mac).
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Bump SLF4J version
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace log4j2 API with SLF4j API
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace usage of LogManager#getFormatterLogger
This is for keeping compatibility with SLF4J. If neccesary, a specific formatter can be created for the RlpBlockImporter class
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Unset the default logging value for the retesteth
This is because it's not possible to resolve the root logger level into a Log4J2 field
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Prevent creation of Logger context outside SLF4J
org.hyperledger.besu.cli.BesuCommand#setAllLevels was taken from
https://github.com/apache/logging-log4j2/blob/rel%2F2.17.1/log4j-core/src/main/java/org/apache/logging/log4j/core/config/Configurator.java#L309
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add FATAL level deprecation message
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S2139
Exceptions should be either logged or rethrown but not both
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* [Sonar] Fix java:S3457
Printf-style format strings should be used correctly
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add changelog
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* lots of errorprone fixes
* some license updates
* some mockito updates
* upgrade the rocksdb version
* Prometheus left at 0.9.0 as 0.10.0+ introduces OpenMetrics
related changes that break unit tests.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Remove all but 4 log4j2.xml config files
* The main config for the besu CLI app
* The config for the evmTool CLI app
* The config for acceptance tests
* A config in testUtil
If any tests depend on a log4j file they should import testutil.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Add tracing support for internals and JSON-RPC
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove rocksdb tracing as it slows down execution too much
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add B3 headers extraction on JSON-RPC requests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove traces around trie tree as they slow down syncing significantly
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing to fast sync pipeline
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing for all pipelines
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Address code review
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add acceptance tests and break out the shaded dependency
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Fix tracer id
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Revert changes to trie
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Upgrade otel to latest, remove old tracing
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review comments
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Adding the Log4j "jul" (java.util.logging) adapter resulted in many
messages like this at startup:
`main INFO Registered Log4j as the java.util.logging.LogManager.`
These come from the Log4j status logger. We can get rid of those by
setting the status attribute on all configurations to a higher logging
level. WARN is the next higher level.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Removes as many Gradle 7.0 compatibility issues as possible
* `baseName` -> `archiveBaseName`
* `extension` -> `archiveExtension`
* `destinationDir` -> `destinationDirectory`
* `runtime` -> `runtimeOnly`
* Change some log4j-api and log4j-core dependencies
* Remove an unneeded and outdated plugin (`net.ltgt.apt`)
* tweak the plugin-api change detector's property annotations.
Warnings still exist with one external plugin used for license file
checking that we do not control the source code for.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Update dependencies to most current version
- except picocli which is a major version update
Alphabetize dependencies
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* adding in spdx-license-identifier & updated check for the same; removing license check from spotless
Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>
* Change CheckSpdxHeader to a task.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Ensure `plugin-api` module gets published at the correct maven path
* Move `plugins` to `plugin-api`
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Return plugin-api to the main repo
* Spotless
* Migrate all external plugin-api references to the project in this repo
* Add licence header
* Update repo reference for publish, even if commented
* Use real configuration for publishing plugin-api
This was tested with the
`:plugins:publishMavenJavaPublicationToMavenLocal` task and checking the
local Maven repo to make sure it was using the correct paths
Signed-off-by: Edward Evans <edward.evans@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Introduce a pipeline based full sync process. Currently toggled off but can be enabled via a --X option.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Allows monitoring to accurately calculate the number of items in a pipe even after it has aborted.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Number of metrics labels need to match up with constructor
* Number of labels must be consistant, so I split it into two metrics
* Also, naming best practices say that sum() and avg() of a metric
should be meaningful, separating into two metrics fixes that.
* fix style issues (finals, intellij warnings)
* Change NoOpMetrics to check label count.
* Cascading changes to support this in many support classes. Mostly places
we presumed all NoOpMetrics were equals.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Capture input and output metrics for each pipe rather than out input.
* Replace the batching processor with a BatchingReadPipe that wraps a normal pipe and creates batching on the fly.
Avoids needing an extra thread and synchronization overhead to create batches.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Matt Whitehead
Usman Saleem
garyschulte
Sally MacFarlane
matkt
Simon Dudley
Danno Ferrin
Diego López León
Antoine Toulme
Antoine Toulme
Pierre Grimaud
Joshua Fernandes
Edward
mark-terry
mbaxter
Adrian Sutton
Adrian Sutton
Danno Ferrin