137 Commits (3a97967b5f4860613a4ba63dab7b0c7d3a318eec)

Author	SHA1	Message	Date
Felipe Renan	cc89b1f6db	Ignore burn address token balance for Tokens ERC-721 ... As we always get errors trying to fetch token balances when the address is the burn addres for Tokens ERC-721, we are gonna ignore them when we are processing the token balances from the token transfer.	6 years ago
Luke Imhoff	b8d9edf97d	Indexer.Block.Catchup.BoundIntervalSupervisor is proper supervisor ... Fixes #592 Ensure that Indexer.Block.Catchup.BoundIntervalSupervisor acts as a proper Supervisor by ensure it supports all Supervisor and :supervisor functions.	6 years ago
Luke Imhoff	2f7be3fb7d	OTP application descriptions ... The Application selector in Wobserver shows the descriptions, so it is useful to be set.	6 years ago
Sebastian Abondano	fdd0c84721	Realtime fetcher retries for unknown block number ... Why: * We want to add a retry mechanism, instead of waiting half a second for every block the realtime fetcher attempts to fetch and index. We had this half a second wait in place in an attempt to avoid "Unknown block number" errors. The half a second wait was effective at reducing these errors but it also meant we were awaiting unnecessarily for most fetches. Retrying after half a second, when Parity says it doesn't yet know of a given block number, makes more sense. * Issue link: n/a This change addresses the need by: * Editing `Indexer.Block.Realtime.Fetcher.fetch_and_import_block/3` to retry failed fetch/import attempts because of "Unknown block number" errors. It waits half a second before retrying and retries a maximum of three times.	6 years ago
Luke Imhoff	8e8b4b5508	Indexer.Block.Uncle.Fetcher ... * Use `Explorer.Chain.stream_unfetched_uncle_hashes` to initialize queue. * Have `Indexer.Block.Catchup.Fetcher` and `Indexer.Block.Realtime.Fetcher` queue uncles to fetch by calling `Indexer.Block.Uncle.Fetcher.async_fetch_blocks`.	6 years ago
Felipe Renan	e74bfb9d3b	Fix token balance log error ... We inverted the value of address_hash with the contract_address_hash. This commit fixes that.	6 years ago
Luke Imhoff	4b140c87df	Block in the consensus chain marked consensus true	6 years ago
Luke Imhoff	96d5aea311	Block.SecondDegreeRelation ... The database table block_second_degree_relations contains links between the `uncle_hash` of the uncle and the `nephew_hash` of the block that said `uncle_hash` was an uncle. It also has a `uncle_fetched_at` that is `null` until the `uncle_hash` is confirmed as fetched in the database. This does not contain such a fetcher.	6 years ago
Luke Imhoff	48b302f935	Return uncles from EtherumJSONRPC.Block.elixir_to_params	6 years ago
Stamates	ff5d4d0163	Wireup live update of pending transactions	6 years ago
Sebastian Abondano	8e3d80a6db	Realtime indexer waits half a second before fetch ... Why: * For Sokol, on staging, we noticed many errors like this one: ``` 00:45:05.627 application=indexer [error] realtime indexer failed to validate for block 4668478: [%{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0x532e069e200be3e372da9680872826bb08caef91"}, message: "Unknown block number"}, %{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0x95426f2bc716022fcf1def006dbc4bb81f5b5164"}, message: "Unknown block number"}, %{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0xe797a1da01eb0f951e0e400f9343de9d17a06bac"}, message: "Unknown block number"}]. Block will be retried by catchup indexer. ``` * Issue link: https://github.com/poanetwork/blockscout/issues/808 This change addresses the need by: * Editing `Indexer.Block.Realtime.Fetcher.fetch_and_import_block/2` for it to wait half a second before starting to fetch and import. The rationale here is that we're hoping that half a second will be enough for Parity to sync and not return an "Unknown block number" error.	6 years ago
Sebastian Abondano	14b916f9e8	Realtime.Fetcher async fetching and no skips ... Why: * For the realtime fetcher to keep up on ETH we want it to fetch blocks asynchronously. We noticed that the realtime fetcher was falling behind on ETH mainnet. Currently the realtime fetcher fetches and imports one block at a time. This commit changes that, so that whenever we're notified of a new block number, through websockets, we start a new process to fetch and import each block asynchronously. * Also, whenever the websocket subscription, through which we're notified of new block numbers, skips block numbers, we want to make sure the skipped blocks are also fetched and imported. * Issue link: n/a This change addresses the need by: * Adding a new `Task.Supervisor` called `Indexer.Block.Realtime.TaskSupervisor` under the `Indexer.Block.Realtime.Supervisor` supervisor. This supervisor was created to supervise fetching and importing processes started by the realtime fetcher every time a new block number is received through the websocket subscription. * Editing `Indexer.Block.Realtime.Fetcher` to start a new process, under the new supervisor mentioned above, to fetch the latest block per the websocket `newHeads` subscription. It also determines if any block numbers have been skipped and starts individual processes to fetch and import the skipped blocks also.	6 years ago
Igor Florian	52e861fa73	Include and Log token_balances error attribute	6 years ago
Igor Florian	99012b2c4c	Add a new log file for token_balances fetching errors	6 years ago
Lucas Narciso	d8b22f6988	Handle large strings when fetching tokens	6 years ago
Amanda Sposito	32ad1214b6	Increase timeout at the `Chain.import` at the 'Indexer.InternalTransaction.Fetcher'	6 years ago
Amanda Sposito	3244113be4	Increase timeout at the `Repo.transaction` level at the 'Indexer.TokenBalance.Fetcher'	6 years ago
Amanda Sposito	eea81817f7	Increase 'Indexer.TokenBalance.Fetcher' timeout	6 years ago
Amanda Sposito	b4a66ad7f0	Add treatment to TokenBalance.Fetcher when there is an error with the import data ... * The TokenBalanceFetcher was not considering empty lists and errors when trying to import the data into the database.	6 years ago
Alex Garibay	56299e308b	Add support for ERC-721 token transfer logs with addresses in `data`	6 years ago
Luke Imhoff	1a8bb6cbed	Restructure Indexer.BlockFetcher to match other Fetchers ... Move the "Fetcher" part below the "time" part of "Realtime" or "Catchup", so that the "Fetcher" and "Supervisor" are siblings like in the fetchers that don't have a "time" part.	6 years ago
Luke Imhoff	5b7872f030	Don't use closure for BufferedTask Task.Supervisor.async_nolink call ... Fixes #676 For some reason, which I haven't tracked down yet, a closure inside of `BufferedTask.spawn_next_batch/1` continually grows in size. When this closure is passed to `Task.Supervisor.async_nolink`, the `Task.Supervisor`, for #676, this was `Indexer.TaskSupervisor` and is now `Indexer.InternalTransaction.TaskSupervisor`, it also grows in memory size. I think this is related to the way that functions and closures are serialized when sent between processes, but I haven't figured out how to verify that. The change also cleared the MsgQ for `Indexer.TransactionFetcher`, probably because its not blocked by the call to `Indexer.InternalTransaction.Fetcher`, which was blocked due to something to do with the memory usage of the closures. The `Indexer.InternalTransaction.TaskSupervisor` consistently shows multiple tasks under it in `:observer`, which it did not previously either.	6 years ago
Luke Imhoff	6aaa300de0	Setup excoveralls for indexer in isolation ... `mix coveralls.html` only works from the root project for indexer and not the `app/indexer`, which made it difficult to generate the indexer-only report.	6 years ago
Luke Imhoff	279c6987b1	Sort Indexer project keys	6 years ago
Luke Imhoff	09051ceb8a	Separate usage of Task.Supervisor ... Have a separate subtree for each fetcher with a supervisor supervising the fetcher and a Task.Supervisor that is only used by the fetcher, so that looking at the Task.Supervisor in that subtree will show which fetcher is causing any memory, message, or blocking problems.	6 years ago
Luke Imhoff	9b3ed74507	Allow all genserver options to be passed to BufferedTasks ... Previously, only name could be passed, but now, by separating callback init options from genserver options, we can pass any options that GenServer supports, including :debug options, which is helpful for tracing performance issues.	6 years ago
Felipe Renan	6148658f61	Store tokens with invalid strings in Token indexer. ... It's a temp fix for https://github.com/poanetwork/blockscout/issues/669. For now, we are going to store those tokens that have names and/or symbols invalids considering: - the first 6 characters from the contract address hash for the name. - nil for the symbol.	6 years ago
Stamates	18380efe0a	fix broken tests ... Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Felipe Renan	e86c5c79f3	Handle timeout when fetching token balances from blockchain	6 years ago
Igor Florian	7077a41576	Change Address fetched_balance's columns to fetched_coin_balace	6 years ago
Igor Florian	acf14d6cf3	Rename Balance references to Address.CoinBalance	6 years ago
Igor Florian	c640bf2de1	Rename AddressBalanceFetcherCase to CoinBalanceCase	6 years ago
Igor Florian	8e0f8b2e92	Rename BalanceFetcher to CoinBalanceFetcher	6 years ago
Igor Florian	2a7c2feac2	Rename Balances module to CoinBalances	6 years ago
Luke Imhoff	9a8cdc4031	Use Path.absname so that log paths are stable during reload ... Fixes #628 The Phoenix.CodeReloader causes the file loggers to restart and reload their config, but when they reload, the current working directory has shifted from the project root to the OTP app root, so it produces log files under each OTP app (`apps/APP_NAME/logs`). We don't want this, so compute the absolute path in the config files, so that the loaded `Application` environment has an absolute path in it that can't shift when the working directory changes.	6 years ago
Felipe Renan	d7bfcd1e87	Change Address.TokenBalance value to be decimal ... It was being created as WEI, but the correct type is decimal such as we are doing with Token and Token Transfers.	6 years ago
Felipe Renan	ad50a21c1e	Fix dialyzer	6 years ago
Felipe Renan	01af13e325	Include TokenBalanceFetcher to Indexer.Supervisor test support	6 years ago
Igor Florian	53e7866016	Fetch TokenBalances value in the Catchup Indexer	6 years ago
Igor Florian	b56f8d4137	Fetch TokenBalances values in the Realtime Indexer	6 years ago
Igor Florian	c2ff17756c	Get TokensBalance in the Blockchain ... We're now able to get the balances from a Address given it's address_hash, block_number and token_contract_address_hash.	6 years ago
Luke Imhoff	c484bc2d25	Separate Indexer.Address.TokenBalances from Indexer.Balances ... TokenBalances need to extract 3 fields, while Balances only need 2, so they shouldn't be mixed together in the same module since the expected params in the set differs.	6 years ago
Luke Imhoff	f32d6d43dd	Use curl to signal parallel build is done to coveralls.io ... Using `curl` in a job instead of the `notify` `webhook` allows us to interpolate the COVERALLS_REPO_TOKEN instead of disclosing it in the config.yml. Use KronicDeth/excoveralls@circle-workflows because upstream hex.pm excoveralls does not properly support parallel jobs run in separate Workflow jobs.	6 years ago
Luke Imhoff	fdd1090c45	Re-enable Ecto loggers ... Allow Ecto loggers to log to the application-specific log **file**, `log/<Mix.env>/ecto.log` but raise `:console`'s level to `:info`, so that Ecto output does not flood `iex` and make it unusable.	6 years ago
Luke Imhoff	c032ea2291	Improved logging ... Have separate loggers for errors and per-application. * `:console` will continue to have all applications and levels. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` * `:error` is only for errors across all environments. It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/error.log`. * `:block_scout_web` is only when `:application` metadata is `:block_scout_web`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/block_scout_web.log`. * `:ethereum_jsonrpc` is only when `:application` metadata is `:ethereum_jsonrpc`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/ethereum_jsonrpc.log`. * `:explorer` is only when `:application` metadata is `:exploer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/explorer.log`. * `:indexer` is only when `:application` metadata is `:indexer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/indexer.log`.	6 years ago
Alex Garibay	cade255e15	Add support for named addresses (#590)	6 years ago
Luke Imhoff	b7c0d188e3	Use handle_continue so that Realtime.init does not block on RPC call	6 years ago
Luke Imhoff	64585c4938	Insert all BufferedTaskTest assert_receive timeouts to 200ms	6 years ago
Luke Imhoff	cddd810a58	Use :websocket_client for EthereumJSONRPC.WebSocket ... :websocket_client can properly use :ssl_verify_hostname.verify_fun/3 for :verify_fun and allow for the handshake to both be secure and pass for infura.	6 years ago
Luke Imhoff	c8c53405fc	Subscribe to newHeads for realtime indexer ... Separate Catchup and Realtime indexer's fetching, so that Catchup continues to use the old polling, `EthereumJSONRPC.json_rpc` while Realtime subscribes with `EthereumJSONRPC.subscribe`. Because Realtime gets notified of new blocks through the subscription, it no longer has a Sequence, so Sequence has been removed from `%BlockFetcher{}`. There is also no block concurrency or block batch size for Realtime, so that was removed from `%BlockFetcher{}` too and moved to `%BlockFetcher.Catchup{}`. Because Realtime requires a websocket to be connected, there is now an `Indexer.BlockFetcher.Realtime.Supervisor` that supervises the `Indexer.BlockFetcher.Realtime.WebSocket` and `Indexer.BlockFetcher.Realtime`. `Indexer.BlockFetcher.Realtime` because it is waiting on notifications from is subscription becomes a `GenServer` insteado of a `Task`, so it can be supervised normally. BlockFetcher.Supervisor becomes a normal `Supervisor` of `Catchup.Supervisor` and `Realtime.Supervisor`. As Realtime is using `EthereumJSONRPC.subscribe`, `subscribe_named_arguments` has been added as a sibling of `json_rpc_named_arguments` in the `:indexer` `config`s.	6 years ago