* Added ability to subscribe to dropped transactions from the transaction pending pool.
Implemented subscription webservice to support this.
* Added metrics to the pending transactions, tracking the number of local and remote transactions in the pool.
* Converted listener management in pending transactions to use the Subscribers util object.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
The 25% of maximum peers is fairly arbitrary but is deliberately above zero to ensure we contribute something to the overall network even while catching up.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
When chain downloading, the SyncTargetManager continues to
attempt to download headers/blocks from the current syncTarget,
even if the system is "insync". This is due to the fact that
the "insync" check is only performed at initial peer selection,
rather than on each loop of the chain downloader.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
This only happens when there are multiple chain downloads in progress so the common ancestor is above the last header we imported, but safety-first...
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Synchronizer returns false if it is in sync
* expand RunnerTest to test eth_syncing behaviour
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Biggest change is that UnusedVariable and UnusedMethod went to WARN by
default. Since our build is a no warning build this means we either need
to turn them off or fix them. I mostly opted for the latter. Test code
was mostly fixed, unused loggers were deleted, and other shipped code
was mostly suppressed.
Two less noisy fixes to not use `SortedSet` and to use zero based
comparable results instead of -1, 0, and 1. Also a compiler nit in
errorprone was suppressed, per the description it won't affect us.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Capture input and output metrics for each pipe rather than out input.
* Replace the batching processor with a BatchingReadPipe that wraps a normal pipe and creates batching on the fly.
Avoids needing an extra thread and synchronization overhead to create batches.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Improve the import log line to include relevant metrics like number
of transactions, ommers, and gas usage.
Also, this fixes a bug in that we get task times when metrics are not
turned on.
Sample line:
2019-03-11 21:03:50.245+00:00 | EthScheduler-Workers-3 | INFO | BlockPropagationManager | Imported #36 / 20 tx / 0 om / 420,000 (4.0%) gas / (0xecb366469d88d9d75b20721581797c1eb1118557b93db9a460dc0e000b9ec7fe) in 0.143s.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Store the instant that a pending transaction is added to the txpool, and expose this out through the api.
* Clock.systemUTC only used once.
* TestClock class added with a factory method for a fixed clock.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Forces subclasses have to implement it so they specify minimum requirements for the peer to use.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Start P2P network synchronously so the ports are guaranteed to be known before we write the ports file
Include the P2P TCP port in ports file even when peer discovery is disabled.
Load information from the advertised peer rather than the discovery listening socket.
Fix admin_nodeInfo to include the ?discport param in the enode URI when the discovery port differs from the P2P port.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* [PAN-2341] Created mechanism to publish sync status
* Refactoring WebSockets syncing subscription to respond to SyncStatus events
* Removing rpc-ws-refresh-delay option
* Removing ws-refresh-delay option from ATs
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Use a single thread to persist received world state nodes to avoid RocksDB timeouts due to contention on the write lock.
* Fix RocksDbTaskQueue so it doesn't read stale data when resuming a transfer.
* Skip downloading empty trie nodes.
* Log unhandled errors from world state download requests
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Instantiate individual metrics where they're used
* Cache prometheus metrics to allow "duplicate" creation of metrics
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Adds a PeerValidator that, when the Dao fork milestone is in use, checks that the Dao block is present on each peer when they connect and disconnects them if they are on the wrong chain.
Also:
* Make GetHeadersFromPeer task stricter in validating response matches.
* Update BlockHeadersMessage to return a list of headers
* Add more controls to DeterministicEthScheduler test util
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* Do parallel extract signatures in the parallel block importer.
* remove the extraction from FullSyncBlockHandler
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
There's an issue with the world state downloader where the download process could stall. The sequence goes like:
1. Inside requestDataFromPeer thread A takes out the sendingRequests lock
2. Thread A checks shouldRequestNodeData which returns true
3. Thread A sends a request for data
4. Thread A checks shouldRequestNodeData which returns false so it exits the while loop
5. Thread B receives the response to the (only) outstanding request
6. Thread B enters shouldRequestNodeData but fails to get the sendingRequests lock so exits the method
7. Thread A releases the sendingRequests lock and exits the methods
There are now no threads checking if they should send new requests and no outstanding requests to trigger a check in the future so the download is stuck and will never make anymore progress.
The fix is to switch the order of taking out the sendingRequests lock and checking shouldRequestNodeData so we release the sendingRequests lock before we go back round the loop to check shouldRequestNodeData.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
* move to an offer() instead of a put() on the downloader so that when
the task is stopped the put will see that it is done and not wait
forever.
* remove peer based focus of the task. Let subtasks pick their peers.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
There is a failure mode of IBFT whereby a validator fails to import
a block, and also fails to receive the NewBlock message from its
peers. This means said validator is unable to participate in
subsequent rounds, and may cause the network to halt.
To overcome this issue, if an IBFT validator receives messages from
a future height, it will update the "BestEstimatedHeight" of the
corresponding EthPeer object, such that the Synchroniser will
(eventually) download the requisite blocks - thus allowing the
IBFT network to continue to operate.
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
--> adapt `BlockBroadcaster` to use `send` method on `EthPeer`
--> adapt `EthProtocolMamnager` to utilize `BlockBroadcaster` to disseminate newly mined block
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
Rob Dawson
Adrian Sutton
Trent Mohay
mbaxter
Sally MacFarlane
Danno Ferrin
Lucas Saldanha
S. Matthew English