345 Commits (25ae5155679800d25a9a38ad07c3d8f57abc7709)

Author	SHA1	Message	Date
zachdaniel	e3c4703a99	fixup! fix: poll on import completion and only new blocks	6 years ago
zachdaniel	c8fb42aba0	fix: poll on import completion and only new blocks	6 years ago
Luke Imhoff	57808d6b23	Separate usage of max consensus block number and block height ... The block height should be used when calculating confirmations while the max consensus block number when a block number is needed and you want to tell if the chain is empty or only the genesis block is available. The block height is 0 if there are no blocks or if there is only the genesis block.	6 years ago
Luke Imhoff	083e5d3278	Ensure min and max block number is for consensus blocks ... Adding `WHERE consensus = true` both fixes the bug that `min_block_number` and `max_block_number` were including non-consensus blocks, but it also makes the query faster because there is an index for consensus blocks. Times from Eth Mainnet production: ``` ether=> SELECT min(b0."number") FROM "blocks" AS b0; min ----- 0 (1 row) Time: 414.218 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0; QUERY PLAN ----------------------------------------------------------------------------------------------- Finalize Aggregate (cost=223454.95..223454.96 rows=1 width=8) -> Gather (cost=223454.73..223454.95 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=222454.73..222454.74 rows=1 width=8) -> Parallel Seq Scan on blocks b0 (cost=0.00..215051.99 rows=2961099 width=8) (5 rows) Time: 1.040 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------- Result (cost=0.45..0.46 rows=1 width=8) InitPlan 1 (returns $0) -> Limit (cost=0.43..0.45 rows=1 width=8) -> Index Only Scan using one_consensus_block_at_height on blocks b0 (cost=0.43..158050.90 rows=7045520 width=8) Index Cond: (number IS NOT NULL) (5 rows) Time: 3.350 ms ether=> SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; min ----- 0 (1 row) Time: 1.059 ms ```	6 years ago
Ayrat Badykov	5c1fca9ac9	fix sigil formatting	6 years ago
Ayrat Badykov	9f45c14ef7	fix CR issues	6 years ago
Ayrat Badykov	60a36a4739	return polling to realtime fetcher	6 years ago
Luke Imhoff	5524a49725	Don't fetch internal transactions for simple token transfers ... Resolves #1303 Don't fetch internal transactions for transactions that have token transfers from the logs if the status is `:ok` and the transaction did not create a contract.	6 years ago
Luke Imhoff	1c915bdb3a	conflict_target and on_conflict for Block Rewards importer ... Fixes #1297	6 years ago
Luke Imhoff	a571a4edb3	Don't fetch or import internal transactions for value transfers ... Fixes #1287 For simple value transfers, the UI already doesn't show the redundant single internal transaction, but we can speed up the indexing by determine the internal transactions aren't needed when the transacion is a successful simple value transfer as determined by status being `:ok` and the input being empty (`0x`).	6 years ago
Luke Imhoff	6165f6757f	Move uniquifiation of address current token balances out of runner ... For all other runners, it is the responsibility of the caller to ensure unique entries. This ensures that the least amount of data as possible is sent to `Explorer.Chain.Import.all`, which speeds up the whole import process.	6 years ago
Gustavo Santos Ferreira	0b3e7f987d	genserver to find blocks without rewards and send them to the importer module	6 years ago
Gustavo Santos Ferreira	3a6751c3cc	importer module to import batches of block rewards	6 years ago
Luke Imhoff	e22fca4ada	Don't use step variable twice ... Accidentally used `step` variable twice when the actual error is like `{:import, {:error, import_step, ..., ...}` which would only allow `{:import, {:error, :import, ..., ...}`.	6 years ago
Luke Imhoff	3592b66340	Chunk pending transactions ... Fixes #1253 Ensure pending transaction import without timeout by chunking in case the node has a lot of pending transactions.	6 years ago
Luke Imhoff	8e1bfc0589	Convert DBConnection.ConnectionError tcp recv: closed to {:error, :timeout} ... Resolves #1251	6 years ago
Luke Imhoff	3bfc342941	Missed error pattern change in #1233	6 years ago
Luke Imhoff	cbe3308c3d	fetch_and_import_range_from_sequence log exceptions instead of raise ... Fixes #1234 Raising the exception will kill all Tasks, the sequence and the fetcher because of the linking from `Task.async_stream`. This leads to when 1 Task times out in a DB transaction, they all are affected, leading to slowdowns when the sequence has to be rebuilt when the fetcher restarts.	6 years ago
Luke Imhoff	adc03fd08b	Label all steps in Block.Fetcher implementations ... Fixes #1232 `{:error, :timeout}` should have been caught, but additionally, it should never have been allowed to propagate out unassigned to a step as it is not possible to track the timeout to a specific fetch.	6 years ago
William Sanches	a754a72390	Add block_reward import	6 years ago
Luke Imhoff	6a0f625da7	Port fix for InvalidConsensus to Uncataloged ... They have a similar style where Uncataloged could also receive :queue_unavailable.	6 years ago
Luke Imhoff	c978b6168b	Don't log when queue is unavailable ... Fixes #1223 For large chains like ETH Mainnet, it is fairly common for the catchup fetcher to take a long time to calculate the missing block numbers and start the named Sequence used by `push_front(block)`. Since this is normal and not exceptional, `InvalidConsensus.Worker` shouldn't log this and retry silently.	6 years ago
Luke Imhoff	e4902fc479	Fix picking wrong clause for fetch_token_balances_from_blockchain ... Fixes #1215 The short-circuiting `[]` was not being used.	6 years ago
Luke Imhoff	5a0f02019b	Don't log errors when uncles aren't found ... Fixes #1221 It happens too often to be worth treating as an error as we should only log things that are exceptional and need human attention as errors.	6 years ago
Luke Imhoff	14c17cb2ee	Retry retried_entries for uncles ... `Indexer.Block.Uncle.Fetcher.retry/1` when there are errors did not actually retry the calculated `retried_errors` and instead only logged them. This didn't cause an unused variable warning because `retried_errors` is used in the Logger metadata. The `Task` for the `run/2` callback did not error because `Logger.error` returns `:ok`. To fix this problem, `{:retry, retried_errors}` need to be returned after the `Logger.error` call.	6 years ago
Luke Imhoff	30e0ac4498	Log timeout in PendingTransaction fetcher instead of crashing ... Fixes #1216	6 years ago
Luke Imhoff	65197f5913	Add shrunk to metadata ... Allow separating shrunk vs unshrunk to monitor how often shrinking occurs.	6 years ago
Luke Imhoff	2569d2595e	Move missing_block_range_count and missing_block_count out of prose ... Being in metadata will allow it to be graphed by tools.	6 years ago
Luke Imhoff	1af8f89001	Move step out of prose and into metadata	6 years ago
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Sebastian Abondano	c762719ba0	Ensures consensus is valid for parent blocks ... Why: * We noticed some blocks with invalid consensus values on Goerli. * Issue link: https://github.com/poanetwork/blockscout/issues/1181 This change addresses the need by: * Editing the realtime fetcher to check, after insertion to DB, a block's parent consensus value and trigger a refetch for that block number if consensus is `FALSE`. * Creating `Indexer.Block.Realtime.ConsensusEnsurer` module, where logic to ensure a given block has consensus lives. * Adding `list_block_number_with_invalid_consensus/0` to `Explorer.Chain` to get block numbers with invalid consensus from DB. * Adding `Indexer.Block.InvalidConsensus.Worker` to find blocks with invalid consensus and queue them for refetching.	6 years ago
Sebastian Abondano	532a0a97bb	Fix realtime indexer's 'no skips prevention' bug ... Why: * For the realtime indexer's logic that fills in skipped block numbers to account for reorgs. See issue for details and examples. * Issue link: https://github.com/poanetwork/blockscout/issues/1189 This change addresses the need by: * Editing the realtime fetcher to keep track of the max block number seen. * Editing realtime fetcher's `start_fetch_and_import` function to only fill in for skipped block numbers when the block number received from the `newHeads` subscription is not a reorg. A reorg in this case is defined as a block number that is less than or equal to the max block number the realtime fetcher has seen. * Editing realtime fetcher's `fetch_and_import_block` function to wait 5 seconds if the block number given for it to fetch is a reorg. This is useful to improve the chance that the latest fetched block for a given number gets consensus. E.g. when goerli's `newHeads` sends two identical block numbers at virtually the same time: `100, 101, 101, 102, 103`, we want to increase the chance taht the second attempt at fetching 101 gets consensus.	6 years ago
Sebastian Abondano	0204d08fba	Fix realtime indexer's 'no skips prevention' bug ... Why: * For the realtime indexer's logic that fills in skipped block numbers to account for reorgs. See issue for details and examples. * Issue link: https://github.com/poanetwork/blockscout/issues/1189 This change addresses the need by: * Editing the realtime fetcher to keep track of the max block number seen. * Editing realtime fetcher's `start_fetch_and_import` function to only fill in for skipped block numbers when the block number received from the `newHeads` subscription is not a reorg. A reorg in this case is defined as a block number that is less than or equal to the max block number the realtime fetcher has seen. * Editing realtime fetcher's `fetch_and_import_block` function to wait 5 seconds if the block number given for it to fetch is a reorg. This is useful to improve the chance that the latest fetched block for a given number gets consensus. E.g. when goerli's `newHeads` sends two identical block numbers at virtually the same time: `100, 101, 101, 102, 103`, we want to increase the chance taht the second attempt at fetching 101 gets consensus.	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
Luke Imhoff	3499155cdb	Partial retries for fetch_blocks_by_range ... Means that both Block.Fetchers, realtime and catchup, now handle fetch errors. The Block.Catchup.Fetcher will retry those blocks that failed to fetch while importing the rest that did fetch instead of refetching the whole batch.	6 years ago
Luke Imhoff	d43a8c6133	Partial retry of batches of blocks by hash	6 years ago
Luke Imhoff	ad1119eb4c	Fetched Beneficiaries partial batch retries	6 years ago
Luke Imhoff	3efa2d9996	Partial retries for CoinBalance.Fetcher	6 years ago
svenski123	3b45474e36	Update address extraction testcases	6 years ago
svenski123	29199eb54e	Add nonce field to Address object and addresses table. ... This caches the highest nonce value seen for sent transactions for a given address. The top addresses query no longer needs to scan the transactions table for nonce values. The database migration script adds the nonce field to the addresses table and sets the nonce value according to the contents of the transactions table.	6 years ago
William Sanches	421262a2da	Fix credo and format issues	6 years ago
William Sanches	2df87c2cd1	Refactor metadata_updater to get interval on start	6 years ago
William Sanches	5b14292aa9	Fix reducer function on metadata updater ... It was concatenating the result in the reverse order.	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Felipe Renan	d627e700d9	Increase the timeout to 15s in Indexer.TokenBalances	6 years ago
Felipe Renan	321a93da88	Standardize Realtime to a list of token balances	6 years ago