Introduce a pipeline based full sync process. Currently toggled off but can be enabled via a --X option.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Implement max message size rather then cap with fixed number of transactions
Adding transactions to the RLP until the message size exceeds the limit and then send that.
* fix final variables
* Update AbstractRLPOutput.java
add javadoc
* pr discussion
put this factory method on LimitedTransactionsMessages rather than TransactionsMessage since it returns a LimitedTransactionsMessages.
* SpotlessApply
* fix PR discussion
- simplify design
- remove useless code
* Update LimitedTransactionsMessages.java
* fix PR discussion
- simplify logic
- add tests
* Update AbstractRLPOutput.java
* Update ethereum/eth/src/main/java/tech/pegasys/pantheon/ethereum/eth/messages/LimitedTransactionsMessages.java
Co-Authored-By: abdelhamidbakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
* Update Transaction.java
* fix PR discussion
* fix PR discussion
- add tests
* Update BlockDataGenerator.java
* Update LimitedTransactionsMessagesTest.java
fix PR unit test
* Update LimitedTransactionsMessagesTest.java
* Update LimitedTransactionsMessagesTest.java
Use LinkedHashSet to preserve order.
* Update LimitedTransactionsMessagesTest.java
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Increase total difficulty threshold required to switch sync targets.
Increase threshold for changing sync target based on height to 200.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
There is no need to keep entire blocks during import after they have
been imported. Keep just the hashes instead.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PRIV-41] Use metrics system for private state db
- Use PrivacyParametersBuilder to build PrivacyParameters
- refactor PrivacyParameters to expose default options
- refactor test builders to use PrivacyParameters.DEFAULT
- Use URI in Enclave
* Fix: enclave tests from bad merge
* Fix privacy acceptance tests after db configuration changes
* Switch to use nested class for PrivacyParametersBuilder
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PIE-1224] Different request limits for different request types
- create `EthLimits` with max size of response according to geth (https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/downloader.go)
- update `EthServer` to use those constants limits instead of the default one
- remove requestLimit from EthServer and EthProtocolManager constructor
fix PIE-1224
* eth wire protocol request limits
- add fields in `SynchronizerConfiguration` to configure per type request limit
- update `EthServer` constructor to add new fields
- update `EthProtocolManager` and subclasses to support new fields
- update unit tests accordingly
* Update SynchronizerConfiguration.java
* Update EthProtocolManagerTestUtil.java
* Update EthServerTest.java
* Update EthProtocolManagerTest.java
* Update EthServerTest.java
* fix after review discussion
- remove per type request limit configuration from `SynchronizerConfiguration`.
- add `EthServer` constructor without new fields that use default values.
- update unit tests accordingly
- update instanciation of different `PantheonController` accordingly
* Update EthServerTest.java
* fix review
- spotless
- fix clean up review comment
* spotlessApply
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Cancel the chain download via ChainDownloader.cancel instead of cancelling the future.
* Update SyncState when sync target changes in pipeline chain downloader.
* Use thenProcessAsyncOrdered
* Log chain download progress.
* Add a short delay after an error occurs before retrying.
* Replicate somewhat weird stop but don't be complete behaviour of EthTask chain downloader to get shutdown working properly.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds handling for options that need to be configurable for tuning and performance but in general are not long-term supported configuration options.
Options that are added via a mixin, are hidden, and that start with `--X` are considered 'Unstable Options.' These options do not show up via normal help facilities but are enumerated via a `XHelp` CLI invocation. These options are unstable, unsupported, and undocumented. They may be added, removed, or modified without prior warning or notice. Use them at your own risk.
For example, the `--Xrocksdb-max-open-files` flag can be set via CLI and can increase the number of max open files RocsDB uses. On a 8GB memory system the default is appropriate, but on a 128GB memory system a much larger number may improve performance. Normal operation scenarios will not need to modify this flag.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
- add `--fast-sync-min-peers` hidden cli option
- add `--fast-sync-max-wait-time` hidden cli option
- update `FastSyncActions` to handle no timeout behaviour (when timeout set to 0 or no timeout set then we just trigger asynchronously the `waitPeersTask`)
- update unit tests mock related to sync config builder
- add fast sync options in `everything_config.toml` and `partial_config.toml`
- re enable PantheonCommandTest.syncModeOptionMustBeUsed
- add new flags in `noOverrideDefaultValuesIfKeyIsNotPresentInConfigFile` and `overrideDefaultValuesIfKeyIsPresentInConfigFile` tests
fix PAN-2265
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Allows monitoring to accurately calculate the number of items in a pipe even after it has aborted.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Implementation currently gets as far as managing the sync target and downloading checkpoint headers.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Also increases the number of requests without progress before considering the download stalled.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Number of metrics labels need to match up with constructor
* Number of labels must be consistant, so I split it into two metrics
* Also, naming best practices say that sum() and avg() of a metric
should be meaningful, separating into two metrics fixes that.
* fix style issues (finals, intellij warnings)
* Change NoOpMetrics to check label count.
* Cascading changes to support this in many support classes. Mostly places
we presumed all NoOpMetrics were equals.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Added ability to subscribe to dropped transactions from the transaction pending pool.
Implemented subscription webservice to support this.
* Added metrics to the pending transactions, tracking the number of local and remote transactions in the pool.
* Converted listener management in pending transactions to use the Subscribers util object.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
The 25% of maximum peers is fairly arbitrary but is deliberately above zero to ensure we contribute something to the overall network even while catching up.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
When chain downloading, the SyncTargetManager continues to
attempt to download headers/blocks from the current syncTarget,
even if the system is "insync". This is due to the fact that
the "insync" check is only performed at initial peer selection,
rather than on each loop of the chain downloader.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
This only happens when there are multiple chain downloads in progress so the common ancestor is above the last header we imported, but safety-first...
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Synchronizer returns false if it is in sync
* expand RunnerTest to test eth_syncing behaviour
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Biggest change is that UnusedVariable and UnusedMethod went to WARN by
default. Since our build is a no warning build this means we either need
to turn them off or fix them. I mostly opted for the latter. Test code
was mostly fixed, unused loggers were deleted, and other shipped code
was mostly suppressed.
Two less noisy fixes to not use `SortedSet` and to use zero based
comparable results instead of -1, 0, and 1. Also a compiler nit in
errorprone was suppressed, per the description it won't affect us.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Capture input and output metrics for each pipe rather than out input.
* Replace the batching processor with a BatchingReadPipe that wraps a normal pipe and creates batching on the fly.
Avoids needing an extra thread and synchronization overhead to create batches.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Rob Dawson
Adrian Sutton
Abdelhamid Bakhta
mbaxter
S. Matthew English
Danno Ferrin
Antony Denyer
Trent Mohay
Sally MacFarlane
Lucas Saldanha