* Change all Stream<?> getX() and Stream<?> x() methods to Stream<?> streanX methods, such as `Stream<Peer> streamIdlePeers()`
* Update coding conventions to reflect this.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Replaces the RocksDB based queue for pending world state download tasks with one that uses a simple file. Added tasks are appended to the file while the reader starts from the beginning of the file and reads forwards.
Periodically a new file is started to limit the disk space used. The reader deletes files it has completed reading.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Fix non-deterministic test caused by variable size of generated transactions.
Validate transactions against a size range rather than a fixed size
* Use assertj instead of own method.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Provides better control over the maximum number of concurrent requests to a peer.
Gives a fairer allocation of peers to requests being made.
Avoids a task failing because all peers are busy which then introduces a delay before the request is retried.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Store the header for the current chain head and total difficulty to avoid RocksDB lookups when requesting those common values.
Also uses that cache to avoid a database lookup when checking if a block has already been imported if the block's parent is the current chain head.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Implement max message size rather then cap with fixed number of transactions
Adding transactions to the RLP until the message size exceeds the limit and then send that.
* fix final variables
* Update AbstractRLPOutput.java
add javadoc
* pr discussion
put this factory method on LimitedTransactionsMessages rather than TransactionsMessage since it returns a LimitedTransactionsMessages.
* SpotlessApply
* fix PR discussion
- simplify design
- remove useless code
* Update LimitedTransactionsMessages.java
* fix PR discussion
- simplify logic
- add tests
* Update AbstractRLPOutput.java
* Update ethereum/eth/src/main/java/tech/pegasys/pantheon/ethereum/eth/messages/LimitedTransactionsMessages.java
Co-Authored-By: abdelhamidbakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
* Update Transaction.java
* fix PR discussion
* fix PR discussion
- add tests
* Update BlockDataGenerator.java
* Update LimitedTransactionsMessagesTest.java
fix PR unit test
* Update LimitedTransactionsMessagesTest.java
* Update LimitedTransactionsMessagesTest.java
Use LinkedHashSet to preserve order.
* Update LimitedTransactionsMessagesTest.java
* follow up PR
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Introduce a pipeline based full sync process. Currently toggled off but can be enabled via a --X option.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Implement max message size rather then cap with fixed number of transactions
Adding transactions to the RLP until the message size exceeds the limit and then send that.
* fix final variables
* Update AbstractRLPOutput.java
add javadoc
* pr discussion
put this factory method on LimitedTransactionsMessages rather than TransactionsMessage since it returns a LimitedTransactionsMessages.
* SpotlessApply
* fix PR discussion
- simplify design
- remove useless code
* Update LimitedTransactionsMessages.java
* fix PR discussion
- simplify logic
- add tests
* Update AbstractRLPOutput.java
* Update ethereum/eth/src/main/java/tech/pegasys/pantheon/ethereum/eth/messages/LimitedTransactionsMessages.java
Co-Authored-By: abdelhamidbakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
* Update Transaction.java
* fix PR discussion
* fix PR discussion
- add tests
* Update BlockDataGenerator.java
* Update LimitedTransactionsMessagesTest.java
fix PR unit test
* Update LimitedTransactionsMessagesTest.java
* Update LimitedTransactionsMessagesTest.java
Use LinkedHashSet to preserve order.
* Update LimitedTransactionsMessagesTest.java
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Increase total difficulty threshold required to switch sync targets.
Increase threshold for changing sync target based on height to 200.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Move RocksDBStats to it's own module. This also brings metrics to
metrics:core since none of our other module have nested modules but
they have peer modules.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
There is no need to keep entire blocks during import after they have
been imported. Keep just the hashes instead.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PRIV-41] Use metrics system for private state db
- Use PrivacyParametersBuilder to build PrivacyParameters
- refactor PrivacyParameters to expose default options
- refactor test builders to use PrivacyParameters.DEFAULT
- Use URI in Enclave
* Fix: enclave tests from bad merge
* Fix privacy acceptance tests after db configuration changes
* Switch to use nested class for PrivacyParametersBuilder
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PIE-1224] Different request limits for different request types
- create `EthLimits` with max size of response according to geth (https://github.com/ethereum/go-ethereum/blob/master/eth/downloader/downloader.go)
- update `EthServer` to use those constants limits instead of the default one
- remove requestLimit from EthServer and EthProtocolManager constructor
fix PIE-1224
* eth wire protocol request limits
- add fields in `SynchronizerConfiguration` to configure per type request limit
- update `EthServer` constructor to add new fields
- update `EthProtocolManager` and subclasses to support new fields
- update unit tests accordingly
* Update SynchronizerConfiguration.java
* Update EthProtocolManagerTestUtil.java
* Update EthServerTest.java
* Update EthProtocolManagerTest.java
* Update EthServerTest.java
* fix after review discussion
- remove per type request limit configuration from `SynchronizerConfiguration`.
- add `EthServer` constructor without new fields that use default values.
- update unit tests accordingly
- update instanciation of different `PantheonController` accordingly
* Update EthServerTest.java
* fix review
- spotless
- fix clean up review comment
* spotlessApply
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Cancel the chain download via ChainDownloader.cancel instead of cancelling the future.
* Update SyncState when sync target changes in pipeline chain downloader.
* Use thenProcessAsyncOrdered
* Log chain download progress.
* Add a short delay after an error occurs before retrying.
* Replicate somewhat weird stop but don't be complete behaviour of EthTask chain downloader to get shutdown working properly.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds handling for options that need to be configurable for tuning and performance but in general are not long-term supported configuration options.
Options that are added via a mixin, are hidden, and that start with `--X` are considered 'Unstable Options.' These options do not show up via normal help facilities but are enumerated via a `XHelp` CLI invocation. These options are unstable, unsupported, and undocumented. They may be added, removed, or modified without prior warning or notice. Use them at your own risk.
For example, the `--Xrocksdb-max-open-files` flag can be set via CLI and can increase the number of max open files RocsDB uses. On a 8GB memory system the default is appropriate, but on a 128GB memory system a much larger number may improve performance. Normal operation scenarios will not need to modify this flag.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
- add `--fast-sync-min-peers` hidden cli option
- add `--fast-sync-max-wait-time` hidden cli option
- update `FastSyncActions` to handle no timeout behaviour (when timeout set to 0 or no timeout set then we just trigger asynchronously the `waitPeersTask`)
- update unit tests mock related to sync config builder
- add fast sync options in `everything_config.toml` and `partial_config.toml`
- re enable PantheonCommandTest.syncModeOptionMustBeUsed
- add new flags in `noOverrideDefaultValuesIfKeyIsNotPresentInConfigFile` and `overrideDefaultValuesIfKeyIsPresentInConfigFile` tests
fix PAN-2265
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Trent Mohay
Danno Ferrin
Adrian Sutton
mbaxter
Abdelhamid Bakhta
S. Matthew English
Adrian Sutton
Rob Dawson
Danno Ferrin
Antony Denyer