819 Commits (61577bceb459c3fd37962456968f2e3924e2f5a8)

Author	SHA1	Message	Date
Luke Imhoff	fcf5e6ef99	InternalTransaction.Fetcher buffer tuples instead maps ... ```elixir test "memory" do InternalTransaction.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(InternalTransaction.Fetcher) Stream.repeatedly(fn -> %{block_number: block_number(), hash: transaction_hash()} end) |> Enum.take(100) |> InternalTransaction.Fetcher.async_fetch() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.82 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	18e06d14f7	Indexer.CoinBalance.Fetcher tuple instead of map in buffers ... ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 372,444 | 230,484 | 61.88 | | 1,000 | After | 142,924 | 88,636 | 62.02 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	f04ea541d0	Indexer.CoinBalance.Fetcher bytes and integer instead of string in ... buffers ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | String (bytes) | Bytes and Integer (bytes) | Bytes / String (%) | |----------|--------------------|----------------|---------------------------|--------------------| | 100 | Before | 55,244 | 34,548 | 62.54 | | 100 | After | 21,764 | 13,860 | 63.68 | | 1,000 | Before | 601,972 | 372,444 | 61.87 | | 1,000 | After | 230,596 | 142,924 | 61.98 | | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.83 | | 100,000 | Before | 16,586,644 | 11,519,724 | 69.45 | | 100,000 | After | 16,583,636 | 11,516,716 | 69.45 |	6 years ago
Luke Imhoff	8d542f07bd	Buffer tuples instead of maps ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 1_000 |> build_list(:token_balance) |> Enum.map(& %{&1 | address_hash: &1.address.hash, token_contract_address_hash: address_hash()}) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 233,500 | 145,828 | 62.45 | | 1,000 | After | 230,492 | 142,820 | 61.96 | | 50,000 | Before | 8,001,028 | 6,668,188 | 83.34 | | 50,000 | After | 7,998,020 | 6,665,180 | 83.35 |	6 years ago
Luke Imhoff	b313af6d7f	Buffer hash bytes instead of hash strings ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Hash Strings (bytes) | Hash Bytes (bytes) | Bytes / String (%) | |----------------|--------------------|----------------------|--------------------|--------------------| | 1,000 | Before | 233,500 | 233,500 | 100.00 | | 1,000 | After | 233,500 | 230,492 | 98.71 | | 10,000 | Before | 2,549,476 | 1,577,180 | 61.86 | | 10,000 | After | 2,546,468 | 1,574,172 | 61.82 |	6 years ago
Luke Imhoff	48b1de4617	Buffer formatted params instead of %TokenBalance{} ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | `%TokenBalance{}`s (bytes) | `%{token_contract_address_hash: Hash.to_string(token_contract_address_hash), address_hash: Hash.to_string(address_hash), block_number: block_number}` (bytes) | Map / Struct (%) | |----------------|--------------------|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| | 1,000 | Before | 2,549,476 | 233,500 | 9.16 | | 1,000 | After | 2,546,468 | 233,500 | 9.17 | | 10,000 | Before | 19,903,172 | 2,549,476 | 12.81 | | 10,000 | After | 19,900,260 | 2,546,468 | 12.80 |	6 years ago
Luke Imhoff	753319de4a	Store ranges as tuples in Sequence :queue ... Found a memory savings for `Indexer.Sequence`: store Ranges as tuples in the `:queue` as the tuples take up less space than than the `%Range{}` struct, but only after garbage-collection. The incoming and outgoing format is still Ranges, so this is a non-breaking implementation detail only. ```elixir test "memory usage" do {:ok, pid} = Sequence.start_link(ranges: [max_block_number..0], step: -10) Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | max_block_number | Garbage Collection | Ranges (bytes) | Tuples (bytes) | Tuples / Ranges (%) | |------------------|--------------------|----------------|----------------|---------------------| | 5,000,000 | Before | 41,050,964 | 22,445,660 | 54.68 | | 5,000,000 | After | 34,386,756 | 23,879,996 | 69.45 | | 10,000,000 | Before | 80,889,772 | 47,928,116 | 59.24 | | 10,000,000 | After | 71,303,316 | 49,516,492 | 69.44 |	6 years ago
Sebastian Abondano	f58bf1586f	Log InternalTransaction.Fetcher errors as error ... Why: * Currently errors when importing internal transactions are being logged with 'debug' level. Making them 'error' level would make sure we notice them quicker and that they're prioritized accordingly. * Issue link: n/a	6 years ago
Amanda Sposito	0ed9020a96	Alter 'Realtime::Fetcher' and 'Catchup::Fetcher' to specify the broadcast type. ... The Token Counter needs to listen to both :realtime and :catchup events so it can keep up to date with the counting, but the 'Live Update' in the front-end doesn't need to hear the catchup events.	6 years ago
Luke Imhoff	1918040eeb	Explorer.Chain.Import.Runner behaviour	6 years ago
Luke Imhoff	59c7c38921	Extract Explorer.Chain.Import.Address.TokenBalances ... Standardized on :address_token_balances instead of :token_balances for options.	6 years ago
Luke Imhoff	991a2fe672	Extract Explorer.Chain.Import.Transactions	6 years ago
Luke Imhoff	22cb89058a	Extract Explorer.Chain.Import.Tokens	6 years ago
Luke Imhoff	14aa12e9bd	Extract Explorer.Chain.Import.TokenTransfers	6 years ago
Luke Imhoff	753fbd8bd5	Remove obsolete receipts	6 years ago
Luke Imhoff	5ffb54129f	Extract Explorer.Chain.Import.Logs	6 years ago
Luke Imhoff	4842462667	Extract Explorer.Chain.Import.Block.SecondDegreeRelations	6 years ago
Luke Imhoff	56f0365239	Extract Explorer.Chain.Import.Blocks	6 years ago
Luke Imhoff	03557f4200	Extract Explorer.Chain.Address.CoinBalances ... Standardize on `:address_coin_balances` instead of `:balances` for option names.	6 years ago
Luke Imhoff	147a7e14fb	Don't do async imports with empty lists	6 years ago
Luke Imhoff	2e3627d536	Don't run imports for empty params	6 years ago
Luke Imhoff	9264a92db3	remaning -> remaining	6 years ago
Luke Imhoff	c60ae9467f	upto -> up to	6 years ago
Luke Imhoff	088589b11a	transaactions -> transactions	6 years ago
Luke Imhoff	65cbce71b4	registred -> registered	6 years ago
Luke Imhoff	f87235c79a	strucutured -> structured	6 years ago
Alex Garibay	d7e38a223e	Missed token transfer cataloger (#807) ... * Rename token transfer params parser * Rename token transfer parsing function * Requeue blocks with incomplete token transfer indexing on start up * Add additional test for worker * Add missed token transfers to the front of the catchup queue * Address code review concerns * Remove unused alias * Add process names for uncataloged token transfer processes * Improved handling of when sequence isn't avaialbe to queue * Move Uncataloged.Supervisor name out of init and into start_link call Name is not a valid option to init as it is a GenServer.option, which means it is passed to the 3rd argument to Supervisor.start_link.	6 years ago
Felipe Renan	cc89b1f6db	Ignore burn address token balance for Tokens ERC-721 ... As we always get errors trying to fetch token balances when the address is the burn addres for Tokens ERC-721, we are gonna ignore them when we are processing the token balances from the token transfer.	6 years ago
Luke Imhoff	b8d9edf97d	Indexer.Block.Catchup.BoundIntervalSupervisor is proper supervisor ... Fixes #592 Ensure that Indexer.Block.Catchup.BoundIntervalSupervisor acts as a proper Supervisor by ensure it supports all Supervisor and :supervisor functions.	6 years ago
Sebastian Abondano	fdd0c84721	Realtime fetcher retries for unknown block number ... Why: * We want to add a retry mechanism, instead of waiting half a second for every block the realtime fetcher attempts to fetch and index. We had this half a second wait in place in an attempt to avoid "Unknown block number" errors. The half a second wait was effective at reducing these errors but it also meant we were awaiting unnecessarily for most fetches. Retrying after half a second, when Parity says it doesn't yet know of a given block number, makes more sense. * Issue link: n/a This change addresses the need by: * Editing `Indexer.Block.Realtime.Fetcher.fetch_and_import_block/3` to retry failed fetch/import attempts because of "Unknown block number" errors. It waits half a second before retrying and retries a maximum of three times.	6 years ago
Luke Imhoff	8e8b4b5508	Indexer.Block.Uncle.Fetcher ... * Use `Explorer.Chain.stream_unfetched_uncle_hashes` to initialize queue. * Have `Indexer.Block.Catchup.Fetcher` and `Indexer.Block.Realtime.Fetcher` queue uncles to fetch by calling `Indexer.Block.Uncle.Fetcher.async_fetch_blocks`.	6 years ago
Felipe Renan	e74bfb9d3b	Fix token balance log error ... We inverted the value of address_hash with the contract_address_hash. This commit fixes that.	6 years ago
Luke Imhoff	4b140c87df	Block in the consensus chain marked consensus true	6 years ago
Luke Imhoff	96d5aea311	Block.SecondDegreeRelation ... The database table block_second_degree_relations contains links between the `uncle_hash` of the uncle and the `nephew_hash` of the block that said `uncle_hash` was an uncle. It also has a `uncle_fetched_at` that is `null` until the `uncle_hash` is confirmed as fetched in the database. This does not contain such a fetcher.	6 years ago
Stamates	ff5d4d0163	Wireup live update of pending transactions	6 years ago
Sebastian Abondano	8e3d80a6db	Realtime indexer waits half a second before fetch ... Why: * For Sokol, on staging, we noticed many errors like this one: ``` 00:45:05.627 application=indexer [error] realtime indexer failed to validate for block 4668478: [%{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0x532e069e200be3e372da9680872826bb08caef91"}, message: "Unknown block number"}, %{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0x95426f2bc716022fcf1def006dbc4bb81f5b5164"}, message: "Unknown block number"}, %{code: -32602, data: %{"blockNumber" => "0x473C3E", "hash" => "0xe797a1da01eb0f951e0e400f9343de9d17a06bac"}, message: "Unknown block number"}]. Block will be retried by catchup indexer. ``` * Issue link: https://github.com/poanetwork/blockscout/issues/808 This change addresses the need by: * Editing `Indexer.Block.Realtime.Fetcher.fetch_and_import_block/2` for it to wait half a second before starting to fetch and import. The rationale here is that we're hoping that half a second will be enough for Parity to sync and not return an "Unknown block number" error.	6 years ago
Sebastian Abondano	14b916f9e8	Realtime.Fetcher async fetching and no skips ... Why: * For the realtime fetcher to keep up on ETH we want it to fetch blocks asynchronously. We noticed that the realtime fetcher was falling behind on ETH mainnet. Currently the realtime fetcher fetches and imports one block at a time. This commit changes that, so that whenever we're notified of a new block number, through websockets, we start a new process to fetch and import each block asynchronously. * Also, whenever the websocket subscription, through which we're notified of new block numbers, skips block numbers, we want to make sure the skipped blocks are also fetched and imported. * Issue link: n/a This change addresses the need by: * Adding a new `Task.Supervisor` called `Indexer.Block.Realtime.TaskSupervisor` under the `Indexer.Block.Realtime.Supervisor` supervisor. This supervisor was created to supervise fetching and importing processes started by the realtime fetcher every time a new block number is received through the websocket subscription. * Editing `Indexer.Block.Realtime.Fetcher` to start a new process, under the new supervisor mentioned above, to fetch the latest block per the websocket `newHeads` subscription. It also determines if any block numbers have been skipped and starts individual processes to fetch and import the skipped blocks also.	6 years ago
Igor Florian	52e861fa73	Include and Log token_balances error attribute	6 years ago
Lucas Narciso	d8b22f6988	Handle large strings when fetching tokens	6 years ago
Amanda Sposito	32ad1214b6	Increase timeout at the `Chain.import` at the 'Indexer.InternalTransaction.Fetcher'	6 years ago
Amanda Sposito	3244113be4	Increase timeout at the `Repo.transaction` level at the 'Indexer.TokenBalance.Fetcher'	6 years ago
Amanda Sposito	eea81817f7	Increase 'Indexer.TokenBalance.Fetcher' timeout	6 years ago
Amanda Sposito	b4a66ad7f0	Add treatment to TokenBalance.Fetcher when there is an error with the import data ... * The TokenBalanceFetcher was not considering empty lists and errors when trying to import the data into the database.	6 years ago
Alex Garibay	56299e308b	Add support for ERC-721 token transfer logs with addresses in `data`	6 years ago
Luke Imhoff	1a8bb6cbed	Restructure Indexer.BlockFetcher to match other Fetchers ... Move the "Fetcher" part below the "time" part of "Realtime" or "Catchup", so that the "Fetcher" and "Supervisor" are siblings like in the fetchers that don't have a "time" part.	6 years ago
Luke Imhoff	5b7872f030	Don't use closure for BufferedTask Task.Supervisor.async_nolink call ... Fixes #676 For some reason, which I haven't tracked down yet, a closure inside of `BufferedTask.spawn_next_batch/1` continually grows in size. When this closure is passed to `Task.Supervisor.async_nolink`, the `Task.Supervisor`, for #676, this was `Indexer.TaskSupervisor` and is now `Indexer.InternalTransaction.TaskSupervisor`, it also grows in memory size. I think this is related to the way that functions and closures are serialized when sent between processes, but I haven't figured out how to verify that. The change also cleared the MsgQ for `Indexer.TransactionFetcher`, probably because its not blocked by the call to `Indexer.InternalTransaction.Fetcher`, which was blocked due to something to do with the memory usage of the closures. The `Indexer.InternalTransaction.TaskSupervisor` consistently shows multiple tasks under it in `:observer`, which it did not previously either.	6 years ago
Luke Imhoff	09051ceb8a	Separate usage of Task.Supervisor ... Have a separate subtree for each fetcher with a supervisor supervising the fetcher and a Task.Supervisor that is only used by the fetcher, so that looking at the Task.Supervisor in that subtree will show which fetcher is causing any memory, message, or blocking problems.	6 years ago
Luke Imhoff	9b3ed74507	Allow all genserver options to be passed to BufferedTasks ... Previously, only name could be passed, but now, by separating callback init options from genserver options, we can pass any options that GenServer supports, including :debug options, which is helpful for tracing performance issues.	6 years ago
Felipe Renan	6148658f61	Store tokens with invalid strings in Token indexer. ... It's a temp fix for https://github.com/poanetwork/blockscout/issues/669. For now, we are going to store those tokens that have names and/or symbols invalids considering: - the first 6 characters from the contract address hash for the name. - nil for the symbol.	6 years ago
Felipe Renan	e86c5c79f3	Handle timeout when fetching token balances from blockchain	6 years ago