There is no need to keep entire blocks during import after they have
been imported. Keep just the hashes instead.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PRIV-41] Use metrics system for private state db
- Use PrivacyParametersBuilder to build PrivacyParameters
- refactor PrivacyParameters to expose default options
- refactor test builders to use PrivacyParameters.DEFAULT
- Use URI in Enclave
* Fix: enclave tests from bad merge
* Fix privacy acceptance tests after db configuration changes
* Switch to use nested class for PrivacyParametersBuilder
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PIE-1224] Different request limits for different request types
- create `EthLimits` with max size of response according to geth (https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/downloader.go)
- update `EthServer` to use those constants limits instead of the default one
- remove requestLimit from EthServer and EthProtocolManager constructor
fix PIE-1224
* eth wire protocol request limits
- add fields in `SynchronizerConfiguration` to configure per type request limit
- update `EthServer` constructor to add new fields
- update `EthProtocolManager` and subclasses to support new fields
- update unit tests accordingly
* Update SynchronizerConfiguration.java
* Update EthProtocolManagerTestUtil.java
* Update EthServerTest.java
* Update EthProtocolManagerTest.java
* Update EthServerTest.java
* fix after review discussion
- remove per type request limit configuration from `SynchronizerConfiguration`.
- add `EthServer` constructor without new fields that use default values.
- update unit tests accordingly
- update instanciation of different `PantheonController` accordingly
* Update EthServerTest.java
* fix review
- spotless
- fix clean up review comment
* spotlessApply
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Cancel the chain download via ChainDownloader.cancel instead of cancelling the future.
* Update SyncState when sync target changes in pipeline chain downloader.
* Use thenProcessAsyncOrdered
* Log chain download progress.
* Add a short delay after an error occurs before retrying.
* Replicate somewhat weird stop but don't be complete behaviour of EthTask chain downloader to get shutdown working properly.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds handling for options that need to be configurable for tuning and performance but in general are not long-term supported configuration options.
Options that are added via a mixin, are hidden, and that start with `--X` are considered 'Unstable Options.' These options do not show up via normal help facilities but are enumerated via a `XHelp` CLI invocation. These options are unstable, unsupported, and undocumented. They may be added, removed, or modified without prior warning or notice. Use them at your own risk.
For example, the `--Xrocksdb-max-open-files` flag can be set via CLI and can increase the number of max open files RocsDB uses. On a 8GB memory system the default is appropriate, but on a 128GB memory system a much larger number may improve performance. Normal operation scenarios will not need to modify this flag.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
- add `--fast-sync-min-peers` hidden cli option
- add `--fast-sync-max-wait-time` hidden cli option
- update `FastSyncActions` to handle no timeout behaviour (when timeout set to 0 or no timeout set then we just trigger asynchronously the `waitPeersTask`)
- update unit tests mock related to sync config builder
- add fast sync options in `everything_config.toml` and `partial_config.toml`
- re enable PantheonCommandTest.syncModeOptionMustBeUsed
- add new flags in `noOverrideDefaultValuesIfKeyIsNotPresentInConfigFile` and `overrideDefaultValuesIfKeyIsPresentInConfigFile` tests
fix PAN-2265
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Allows monitoring to accurately calculate the number of items in a pipe even after it has aborted.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Implementation currently gets as far as managing the sync target and downloading checkpoint headers.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Also increases the number of requests without progress before considering the download stalled.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Number of metrics labels need to match up with constructor
* Number of labels must be consistant, so I split it into two metrics
* Also, naming best practices say that sum() and avg() of a metric
should be meaningful, separating into two metrics fixes that.
* fix style issues (finals, intellij warnings)
* Change NoOpMetrics to check label count.
* Cascading changes to support this in many support classes. Mostly places
we presumed all NoOpMetrics were equals.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Added ability to subscribe to dropped transactions from the transaction pending pool.
Implemented subscription webservice to support this.
* Added metrics to the pending transactions, tracking the number of local and remote transactions in the pool.
* Converted listener management in pending transactions to use the Subscribers util object.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
The 25% of maximum peers is fairly arbitrary but is deliberately above zero to ensure we contribute something to the overall network even while catching up.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
When chain downloading, the SyncTargetManager continues to
attempt to download headers/blocks from the current syncTarget,
even if the system is "insync". This is due to the fact that
the "insync" check is only performed at initial peer selection,
rather than on each loop of the chain downloader.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
This only happens when there are multiple chain downloads in progress so the common ancestor is above the last header we imported, but safety-first...
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Synchronizer returns false if it is in sync
* expand RunnerTest to test eth_syncing behaviour
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Biggest change is that UnusedVariable and UnusedMethod went to WARN by
default. Since our build is a no warning build this means we either need
to turn them off or fix them. I mostly opted for the latter. Test code
was mostly fixed, unused loggers were deleted, and other shipped code
was mostly suppressed.
Two less noisy fixes to not use `SortedSet` and to use zero based
comparable results instead of -1, 0, and 1. Also a compiler nit in
errorprone was suppressed, per the description it won't affect us.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Capture input and output metrics for each pipe rather than out input.
* Replace the batching processor with a BatchingReadPipe that wraps a normal pipe and creates batching on the fly.
Avoids needing an extra thread and synchronization overhead to create batches.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Improve the import log line to include relevant metrics like number
of transactions, ommers, and gas usage.
Also, this fixes a bug in that we get task times when metrics are not
turned on.
Sample line:
2019-03-11 21:03:50.245+00:00 | EthScheduler-Workers-3 | INFO | BlockPropagationManager | Imported #36 / 20 tx / 0 om / 420,000 (4.0%) gas / (0xecb366469d88d9d75b20721581797c1eb1118557b93db9a460dc0e000b9ec7fe) in 0.143s.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Store the instant that a pending transaction is added to the txpool, and expose this out through the api.
* Clock.systemUTC only used once.
* TestClock class added with a factory method for a fixed clock.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Forces subclasses have to implement it so they specify minimum requirements for the peer to use.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Start P2P network synchronously so the ports are guaranteed to be known before we write the ports file
Include the P2P TCP port in ports file even when peer discovery is disabled.
Load information from the advertised peer rather than the discovery listening socket.
Fix admin_nodeInfo to include the ?discport param in the enode URI when the discovery port differs from the P2P port.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PAN-2341] Created mechanism to publish sync status
* Refactoring WebSockets syncing subscription to respond to SyncStatus events
* Removing rpc-ws-refresh-delay option
* Removing ws-refresh-delay option from ATs
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Danno Ferrin
Antony Denyer
Adrian Sutton
Abdelhamid Bakhta
Rob Dawson
Trent Mohay
mbaxter
Sally MacFarlane
Lucas Saldanha
S. Matthew English