926 Commits (6f5dc3b5d365b0b0e3c2ce1ade3bed67ea9331aa)

Author	SHA1	Message	Date
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Sebastian Abondano	c762719ba0	Ensures consensus is valid for parent blocks ... Why: * We noticed some blocks with invalid consensus values on Goerli. * Issue link: https://github.com/poanetwork/blockscout/issues/1181 This change addresses the need by: * Editing the realtime fetcher to check, after insertion to DB, a block's parent consensus value and trigger a refetch for that block number if consensus is `FALSE`. * Creating `Indexer.Block.Realtime.ConsensusEnsurer` module, where logic to ensure a given block has consensus lives. * Adding `list_block_number_with_invalid_consensus/0` to `Explorer.Chain` to get block numbers with invalid consensus from DB. * Adding `Indexer.Block.InvalidConsensus.Worker` to find blocks with invalid consensus and queue them for refetching.	6 years ago
Sebastian Abondano	532a0a97bb	Fix realtime indexer's 'no skips prevention' bug ... Why: * For the realtime indexer's logic that fills in skipped block numbers to account for reorgs. See issue for details and examples. * Issue link: https://github.com/poanetwork/blockscout/issues/1189 This change addresses the need by: * Editing the realtime fetcher to keep track of the max block number seen. * Editing realtime fetcher's `start_fetch_and_import` function to only fill in for skipped block numbers when the block number received from the `newHeads` subscription is not a reorg. A reorg in this case is defined as a block number that is less than or equal to the max block number the realtime fetcher has seen. * Editing realtime fetcher's `fetch_and_import_block` function to wait 5 seconds if the block number given for it to fetch is a reorg. This is useful to improve the chance that the latest fetched block for a given number gets consensus. E.g. when goerli's `newHeads` sends two identical block numbers at virtually the same time: `100, 101, 101, 102, 103`, we want to increase the chance taht the second attempt at fetching 101 gets consensus.	6 years ago
Sebastian Abondano	0204d08fba	Fix realtime indexer's 'no skips prevention' bug ... Why: * For the realtime indexer's logic that fills in skipped block numbers to account for reorgs. See issue for details and examples. * Issue link: https://github.com/poanetwork/blockscout/issues/1189 This change addresses the need by: * Editing the realtime fetcher to keep track of the max block number seen. * Editing realtime fetcher's `start_fetch_and_import` function to only fill in for skipped block numbers when the block number received from the `newHeads` subscription is not a reorg. A reorg in this case is defined as a block number that is less than or equal to the max block number the realtime fetcher has seen. * Editing realtime fetcher's `fetch_and_import_block` function to wait 5 seconds if the block number given for it to fetch is a reorg. This is useful to improve the chance that the latest fetched block for a given number gets consensus. E.g. when goerli's `newHeads` sends two identical block numbers at virtually the same time: `100, 101, 101, 102, 103`, we want to increase the chance taht the second attempt at fetching 101 gets consensus.	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
Luke Imhoff	3499155cdb	Partial retries for fetch_blocks_by_range ... Means that both Block.Fetchers, realtime and catchup, now handle fetch errors. The Block.Catchup.Fetcher will retry those blocks that failed to fetch while importing the rest that did fetch instead of refetching the whole batch.	6 years ago
Luke Imhoff	d43a8c6133	Partial retry of batches of blocks by hash	6 years ago
Luke Imhoff	ad1119eb4c	Fetched Beneficiaries partial batch retries	6 years ago
Luke Imhoff	3efa2d9996	Partial retries for CoinBalance.Fetcher	6 years ago
svenski123	3b45474e36	Update address extraction testcases	6 years ago
svenski123	29199eb54e	Add nonce field to Address object and addresses table. ... This caches the highest nonce value seen for sent transactions for a given address. The top addresses query no longer needs to scan the transactions table for nonce values. The database migration script adds the nonce field to the addresses table and sets the nonce value according to the contents of the transactions table.	6 years ago
William Sanches	421262a2da	Fix credo and format issues	6 years ago
William Sanches	2df87c2cd1	Refactor metadata_updater to get interval on start	6 years ago
William Sanches	5b14292aa9	Fix reducer function on metadata updater ... It was concatenating the result in the reverse order.	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Felipe Renan	d627e700d9	Increase the timeout to 15s in Indexer.TokenBalances	6 years ago
Felipe Renan	321a93da88	Standardize Realtime to a list of token balances	6 years ago
Felipe Renan	e26a0aab93	Improve TokenBalance.Fetcher docs	6 years ago
Felipe Renan	c97d7c1572	Handle empty list in Indexer.TokenBalances	6 years ago
Felipe Renan	e2827ba907	Log token balances that raised errors	6 years ago
Felipe Renan	a79fa5b3e4	Add retry feature to the TokenBalance.Fetcher ... As some token balances always will raise errors interacting with the Smart Contract, we have to ignore them. Otherwise, we will accumulate a lot of toke balances in the indexer memory that never will be fetched.	6 years ago
Felipe Renan	10821e6b0b	Rename FunctionsReader to MetadaRetriever ... Also: * Renaming `fetch_functions_from_contract` to `fetch_functions_with_retries` * Using pattern matching instead of guard in `fetch_functions_with_retries`	6 years ago
Felipe Renan	477de645fb	Refactor Indexer.Token.Fetcher ... This refactor aims to remove the responsability of reading functions from the Smart Contract from Token.Fetcher. Now, we have the module Explorer.Token.FunctionsReader that is responsible for it.	6 years ago
Luke Imhoff	e4b7a4ee5f	Geth internal transactions ... Get internal transactions using debug_traceTransaction and a custom tracer. The tracer was based on the callTracer that ships with Geth, but rewritten to break it into more functions with more explicit purpose and to patch some holes in callTracer: * All calls have gas and gasUsed, which patches the hole in callTracer for handling `DELEGATECALL`. * All calls have value, which patches the hole in callTracer for handling `DELEGATECALL` and `STATICALL`. Since the tracer needed to be resubmitted in whole anyway, it was further customized to match the format needed for internal transactions: * Instead of returning the `op` as `type`, the `type` and `callType` used for internal transactions is returned. * Instead of returning a call tree, the tree is walked in pre-order, displaying the parent call before its nested calls to match the flat order returned from parity's trace_transaction.	6 years ago
Luke Imhoff	9cce7ab661	Adopt EIP-6 ... Geth has EIP-6, while Parity does not, but the Ethereum Yellow Paper includes EIP-6, so it is better to use EIP-6 when looking up community docs.	6 years ago
Gustavo Santos Ferreira	c50d77d233	upsert addresses along with their token balances	6 years ago
Felipe Renan	ed183c988d	Import the current token balances	6 years ago
Felipe Renan	af6a735894	Ensure that token balances that gave errors will be scheduled	6 years ago
Alex Garibay	e23ebab069	Add transform functions for block indexing	6 years ago
Alex Garibay	c0ffdd2f0c	Add function for recovering signer address for Clique	6 years ago
Andrew Cravenho	5892c8357b	enqueue fix for push_front ... Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Luke Imhoff	6134d5ad07	credo 0.9.2 => 0.10.2	6 years ago
Gustavo Santos Ferreira	cf1be63b60	add new field to address extractor	6 years ago
Felipe Renan	5d1441d515	Do not store token_hash as address_hash in token balances ... We don't need this extraction because it will try to fetch the balance from the own token in its contract and it doesn't make sense for us. We will stop to store a lot of unnecessary data removing this extraction.	6 years ago
Luke Imhoff	41ea1f55e7	Don't use :replace_all for transactions	6 years ago
Felipe Renan	5ccfb258d6	Send unfetched token balances to be fetched by TokenBalance.Fetcher	6 years ago
Felipe Renan	0fbd09653c	Increase batch_size in TokenBalance.Fetcher	6 years ago
Luke Imhoff	91042f4af3	Shrinkable BufferedTasks ... `Indexer.BufferedTask` now uses an `Indexer.BoundQueue` instead of a `:queue`. All `Indexer.BufferedTask`s register with the memory monitor passed in to `BufferedTask.init/1` and will shrink their bound queue when asked. If they run out of work and have shrunk `c:init/2` will be called again on the callback module. `Indexer.Logger.process/1` will generate iodata of the pid and registered name of a process if available.	6 years ago
Luke Imhoff	0976a572ff	Implement Enumerable for BoundQueue	6 years ago
Luke Imhoff	9f09ee0226	Remove init_chunk_size ... It didn't really do much besides decided when to send GenServer.calls from the init_task.	6 years ago
Luke Imhoff	67ca25725d	Don't batch until run ... Instead of queuing up batches, store entries in queue and only form a batch when run is called. This ensures that the batch size is maximized and previously partial adjacent batches will go out as a full batch.	6 years ago
Luke Imhoff	7abc419617	Don't store pid in BufferedTask state	6 years ago
Luke Imhoff	b436d8bf5d	Don't track retry counts ... All implemented of BufferedTask ignore the number and not tracking it saves memory and simplifies BufferedTask itself.	6 years ago
Luke Imhoff	1fce095a35	Shrink Catchup Sequence if its consumes too much memory	6 years ago
Luke Imhoff	cae25195e5	Indexer.Memory.Monitor ... `Indexer.Memory.Monitor` checks if the BEAM memory usage exceeds a set limit (defaults to 1 GiB) and if it does, it asks the process with the most memory that is registered as shrinkable to shrink.	6 years ago
Amanda Sposito	b86335cf34	Add 'transaction_index' in the fetchers to save at 'internal_transaction' ... Add 'transaction_index' in the fetchers to save at 'internal_transaction'	6 years ago
Alex Garibay	0d0aa457cd	Remove null bytes from token name and symbol	6 years ago
Sebastian Abondano	2b8ceb4e32	Update block reward contract beneficiary balances ... Why: * "Parity’s consensus engine allows using a smart contract for block reward calculation." - see https://wiki.parity.io/Block-Reward-Contract for details. This means that we need to use `trace_block` (https://wiki.parity.io/JSONRPC-trace-module#trace_block) to fetch the block reward contract beneficiaries and update their balances. Currently this doesn't happen so there are addresses (block contract reward beneficiaries) with inaccurate balances and missing rows in the `address_coin_balances` table. * Issue link: https://github.com/poanetwork/blockscout/issues/767 This change addresses the need by: * Adding `EthereumJSONRPC.fetch_beneficiaries/2` to help us fetch the block reward contract beneficiaries of a given block range. This only works with Parity. * Editing Indexer's dev and prod config by adding `trace_block` config to `:method_to_url`. * Adding `block_reward_contract_beneficiaries` to `Indexer.AddressExtraction` for it to know how to get addresses in this scenario. * Editing `Indexer.Block.Fetcher.fetch_and_import_range/2` to get block reward contract beneficiaries, create their addresses if they don't yet exist, and update their coin balances.	6 years ago
Felipe Renan	bed1a730dd	Consider mint transfers in the logs to index the address	6 years ago