274 Commits (f7ec66aafcc7354dc8ffd2040638c2c7d9c7f868)

Author	SHA1	Message	Date
Gustavo Santos Ferreira	3a6751c3cc	importer module to import batches of block rewards	6 years ago
Luke Imhoff	81179f9459	Use :timer functions to make timeout values clearer	6 years ago
Luke Imhoff	254d6b46a1	Increase assert_receive timeout	6 years ago
Sebastian Abondano	c762719ba0	Ensures consensus is valid for parent blocks ... Why: * We noticed some blocks with invalid consensus values on Goerli. * Issue link: https://github.com/poanetwork/blockscout/issues/1181 This change addresses the need by: * Editing the realtime fetcher to check, after insertion to DB, a block's parent consensus value and trigger a refetch for that block number if consensus is `FALSE`. * Creating `Indexer.Block.Realtime.ConsensusEnsurer` module, where logic to ensure a given block has consensus lives. * Adding `list_block_number_with_invalid_consensus/0` to `Explorer.Chain` to get block numbers with invalid consensus from DB. * Adding `Indexer.Block.InvalidConsensus.Worker` to find blocks with invalid consensus and queue them for refetching.	6 years ago
Luke Imhoff	3499155cdb	Partial retries for fetch_blocks_by_range ... Means that both Block.Fetchers, realtime and catchup, now handle fetch errors. The Block.Catchup.Fetcher will retry those blocks that failed to fetch while importing the rest that did fetch instead of refetching the whole batch.	6 years ago
Luke Imhoff	d43a8c6133	Partial retry of batches of blocks by hash	6 years ago
Luke Imhoff	ad1119eb4c	Fetched Beneficiaries partial batch retries	6 years ago
Luke Imhoff	3efa2d9996	Partial retries for CoinBalance.Fetcher	6 years ago
svenski123	3b45474e36	Update address extraction testcases	6 years ago
Andrew Cravenho	bd769c6724	Fix core tests	6 years ago
Amanda	3c146f92e4	Add test to MetadataUpdater's GenServer ... Co-Authored-By: w-sanches <1727723+w-sanches@users.noreply.github.com>	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Felipe Renan	d627e700d9	Increase the timeout to 15s in Indexer.TokenBalances	6 years ago
Felipe Renan	e2827ba907	Log token balances that raised errors	6 years ago
Felipe Renan	a79fa5b3e4	Add retry feature to the TokenBalance.Fetcher ... As some token balances always will raise errors interacting with the Smart Contract, we have to ignore them. Otherwise, we will accumulate a lot of toke balances in the indexer memory that never will be fetched.	6 years ago
Felipe Renan	477de645fb	Refactor Indexer.Token.Fetcher ... This refactor aims to remove the responsability of reading functions from the Smart Contract from Token.Fetcher. Now, we have the module Explorer.Token.FunctionsReader that is responsible for it.	6 years ago
Luke Imhoff	9cce7ab661	Adopt EIP-6 ... Geth has EIP-6, while Parity does not, but the Ethereum Yellow Paper includes EIP-6, so it is better to use EIP-6 when looking up community docs.	6 years ago
Gustavo Santos Ferreira	c50d77d233	upsert addresses along with their token balances	6 years ago
Felipe Renan	af6a735894	Ensure that token balances that gave errors will be scheduled	6 years ago
Alex Garibay	e23ebab069	Add transform functions for block indexing	6 years ago
Alex Garibay	c0ffdd2f0c	Add function for recovering signer address for Clique	6 years ago
Gustavo Santos Ferreira	965bbb5e75	replace decimals from int to decimal	6 years ago
Andrew Cravenho	5892c8357b	enqueue fix for push_front ... Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Felipe Renan	5d1441d515	Do not store token_hash as address_hash in token balances ... We don't need this extraction because it will try to fetch the balance from the own token in its contract and it doesn't make sense for us. We will stop to store a lot of unnecessary data removing this extraction.	6 years ago
Luke Imhoff	96b97f0dc1	Switch logs to composite primary key ... * Drop id. * Make (transaaction_hash, index) the primary key. This ports the change from `internal_transactions`.	6 years ago
Felipe Renan	5ccfb258d6	Send unfetched token balances to be fetched by TokenBalance.Fetcher	6 years ago
Luke Imhoff	91042f4af3	Shrinkable BufferedTasks ... `Indexer.BufferedTask` now uses an `Indexer.BoundQueue` instead of a `:queue`. All `Indexer.BufferedTask`s register with the memory monitor passed in to `BufferedTask.init/1` and will shrink their bound queue when asked. If they run out of work and have shrunk `c:init/2` will be called again on the callback module. `Indexer.Logger.process/1` will generate iodata of the pid and registered name of a process if available.	6 years ago
Luke Imhoff	9f09ee0226	Remove init_chunk_size ... It didn't really do much besides decided when to send GenServer.calls from the init_task.	6 years ago
Luke Imhoff	67ca25725d	Don't batch until run ... Instead of queuing up batches, store entries in queue and only form a batch when run is called. This ensures that the batch size is maximized and previously partial adjacent batches will go out as a full batch.	6 years ago
Luke Imhoff	b436d8bf5d	Don't track retry counts ... All implemented of BufferedTask ignore the number and not tracking it saves memory and simplifies BufferedTask itself.	6 years ago
Luke Imhoff	1fce095a35	Shrink Catchup Sequence if its consumes too much memory	6 years ago
Amanda Sposito	b86335cf34	Add 'transaction_index' in the fetchers to save at 'internal_transaction' ... Add 'transaction_index' in the fetchers to save at 'internal_transaction'	6 years ago
Sebastian Abondano	2b8ceb4e32	Update block reward contract beneficiary balances ... Why: * "Parity’s consensus engine allows using a smart contract for block reward calculation." - see https://wiki.parity.io/Block-Reward-Contract for details. This means that we need to use `trace_block` (https://wiki.parity.io/JSONRPC-trace-module#trace_block) to fetch the block reward contract beneficiaries and update their balances. Currently this doesn't happen so there are addresses (block contract reward beneficiaries) with inaccurate balances and missing rows in the `address_coin_balances` table. * Issue link: https://github.com/poanetwork/blockscout/issues/767 This change addresses the need by: * Adding `EthereumJSONRPC.fetch_beneficiaries/2` to help us fetch the block reward contract beneficiaries of a given block range. This only works with Parity. * Editing Indexer's dev and prod config by adding `trace_block` config to `:method_to_url`. * Adding `block_reward_contract_beneficiaries` to `Indexer.AddressExtraction` for it to know how to get addresses in this scenario. * Editing `Indexer.Block.Fetcher.fetch_and_import_range/2` to get block reward contract beneficiaries, create their addresses if they don't yet exist, and update their coin balances.	6 years ago
Felipe Renan	bed1a730dd	Consider mint transfers in the logs to index the address	6 years ago
Luke Imhoff	fcf5e6ef99	InternalTransaction.Fetcher buffer tuples instead maps ... ```elixir test "memory" do InternalTransaction.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(InternalTransaction.Fetcher) Stream.repeatedly(fn -> %{block_number: block_number(), hash: transaction_hash()} end) |> Enum.take(100) |> InternalTransaction.Fetcher.async_fetch() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.82 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	18e06d14f7	Indexer.CoinBalance.Fetcher tuple instead of map in buffers ... ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 372,444 | 230,484 | 61.88 | | 1,000 | After | 142,924 | 88,636 | 62.02 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	f04ea541d0	Indexer.CoinBalance.Fetcher bytes and integer instead of string in ... buffers ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | String (bytes) | Bytes and Integer (bytes) | Bytes / String (%) | |----------|--------------------|----------------|---------------------------|--------------------| | 100 | Before | 55,244 | 34,548 | 62.54 | | 100 | After | 21,764 | 13,860 | 63.68 | | 1,000 | Before | 601,972 | 372,444 | 61.87 | | 1,000 | After | 230,596 | 142,924 | 61.98 | | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.83 | | 100,000 | Before | 16,586,644 | 11,519,724 | 69.45 | | 100,000 | After | 16,583,636 | 11,516,716 | 69.45 |	6 years ago
Luke Imhoff	8d542f07bd	Buffer tuples instead of maps ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 1_000 |> build_list(:token_balance) |> Enum.map(& %{&1 | address_hash: &1.address.hash, token_contract_address_hash: address_hash()}) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 233,500 | 145,828 | 62.45 | | 1,000 | After | 230,492 | 142,820 | 61.96 | | 50,000 | Before | 8,001,028 | 6,668,188 | 83.34 | | 50,000 | After | 7,998,020 | 6,665,180 | 83.35 |	6 years ago
Luke Imhoff	59c7c38921	Extract Explorer.Chain.Import.Address.TokenBalances ... Standardized on :address_token_balances instead of :token_balances for options.	6 years ago
Luke Imhoff	03557f4200	Extract Explorer.Chain.Address.CoinBalances ... Standardize on `:address_coin_balances` instead of `:balances` for option names.	6 years ago
Luke Imhoff	2e3627d536	Don't run imports for empty params	6 years ago
Luke Imhoff	bf7e8f11c5	Unsupport -> Unsupported	6 years ago
Luke Imhoff	1ba2e01bfa	lastest -> latest	6 years ago
Alex Garibay	d7e38a223e	Missed token transfer cataloger (#807) ... * Rename token transfer params parser * Rename token transfer parsing function * Requeue blocks with incomplete token transfer indexing on start up * Add additional test for worker * Add missed token transfers to the front of the catchup queue * Address code review concerns * Remove unused alias * Add process names for uncataloged token transfer processes * Improved handling of when sequence isn't avaialbe to queue * Move Uncataloged.Supervisor name out of init and into start_link call Name is not a valid option to init as it is a GenServer.option, which means it is passed to the 3rd argument to Supervisor.start_link.	6 years ago
Felipe Renan	cc89b1f6db	Ignore burn address token balance for Tokens ERC-721 ... As we always get errors trying to fetch token balances when the address is the burn addres for Tokens ERC-721, we are gonna ignore them when we are processing the token balances from the token transfer.	6 years ago
Luke Imhoff	b8d9edf97d	Indexer.Block.Catchup.BoundIntervalSupervisor is proper supervisor ... Fixes #592 Ensure that Indexer.Block.Catchup.BoundIntervalSupervisor acts as a proper Supervisor by ensure it supports all Supervisor and :supervisor functions.	6 years ago
Luke Imhoff	8e8b4b5508	Indexer.Block.Uncle.Fetcher ... * Use `Explorer.Chain.stream_unfetched_uncle_hashes` to initialize queue. * Have `Indexer.Block.Catchup.Fetcher` and `Indexer.Block.Realtime.Fetcher` queue uncles to fetch by calling `Indexer.Block.Uncle.Fetcher.async_fetch_blocks`.	6 years ago
Luke Imhoff	4b140c87df	Block in the consensus chain marked consensus true	6 years ago
Luke Imhoff	96d5aea311	Block.SecondDegreeRelation ... The database table block_second_degree_relations contains links between the `uncle_hash` of the uncle and the `nephew_hash` of the block that said `uncle_hash` was an uncle. It also has a `uncle_fetched_at` that is `null` until the `uncle_hash` is confirmed as fetched in the database. This does not contain such a fetcher.	6 years ago
Luke Imhoff	48b302f935	Return uncles from EtherumJSONRPC.Block.elixir_to_params	6 years ago