418 Commits (c156865c2c199709e27566977b4c8cadfa6c2032)

Author	SHA1	Message	Date
svenski123	29199eb54e	Add nonce field to Address object and addresses table. ... This caches the highest nonce value seen for sent transactions for a given address. The top addresses query no longer needs to scan the transactions table for nonce values. The database migration script adds the nonce field to the addresses table and sets the nonce value according to the contents of the transactions table.	6 years ago
William Sanches	421262a2da	Fix credo and format issues	6 years ago
William Sanches	2df87c2cd1	Refactor metadata_updater to get interval on start	6 years ago
William Sanches	5b14292aa9	Fix reducer function on metadata updater ... It was concatenating the result in the reverse order.	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Felipe Renan	d627e700d9	Increase the timeout to 15s in Indexer.TokenBalances	6 years ago
Felipe Renan	321a93da88	Standardize Realtime to a list of token balances	6 years ago
Felipe Renan	e26a0aab93	Improve TokenBalance.Fetcher docs	6 years ago
Felipe Renan	c97d7c1572	Handle empty list in Indexer.TokenBalances	6 years ago
Felipe Renan	e2827ba907	Log token balances that raised errors	6 years ago
Felipe Renan	a79fa5b3e4	Add retry feature to the TokenBalance.Fetcher ... As some token balances always will raise errors interacting with the Smart Contract, we have to ignore them. Otherwise, we will accumulate a lot of toke balances in the indexer memory that never will be fetched.	6 years ago
Felipe Renan	10821e6b0b	Rename FunctionsReader to MetadaRetriever ... Also: * Renaming `fetch_functions_from_contract` to `fetch_functions_with_retries` * Using pattern matching instead of guard in `fetch_functions_with_retries`	6 years ago
Felipe Renan	477de645fb	Refactor Indexer.Token.Fetcher ... This refactor aims to remove the responsability of reading functions from the Smart Contract from Token.Fetcher. Now, we have the module Explorer.Token.FunctionsReader that is responsible for it.	6 years ago
Luke Imhoff	e4b7a4ee5f	Geth internal transactions ... Get internal transactions using debug_traceTransaction and a custom tracer. The tracer was based on the callTracer that ships with Geth, but rewritten to break it into more functions with more explicit purpose and to patch some holes in callTracer: * All calls have gas and gasUsed, which patches the hole in callTracer for handling `DELEGATECALL`. * All calls have value, which patches the hole in callTracer for handling `DELEGATECALL` and `STATICALL`. Since the tracer needed to be resubmitted in whole anyway, it was further customized to match the format needed for internal transactions: * Instead of returning the `op` as `type`, the `type` and `callType` used for internal transactions is returned. * Instead of returning a call tree, the tree is walked in pre-order, displaying the parent call before its nested calls to match the flat order returned from parity's trace_transaction.	6 years ago
Luke Imhoff	9cce7ab661	Adopt EIP-6 ... Geth has EIP-6, while Parity does not, but the Ethereum Yellow Paper includes EIP-6, so it is better to use EIP-6 when looking up community docs.	6 years ago
Gustavo Santos Ferreira	c50d77d233	upsert addresses along with their token balances	6 years ago
Felipe Renan	ed183c988d	Import the current token balances	6 years ago
Felipe Renan	af6a735894	Ensure that token balances that gave errors will be scheduled	6 years ago
Alex Garibay	e23ebab069	Add transform functions for block indexing	6 years ago
Alex Garibay	c0ffdd2f0c	Add function for recovering signer address for Clique	6 years ago
Andrew Cravenho	5892c8357b	enqueue fix for push_front ... Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Luke Imhoff	6134d5ad07	credo 0.9.2 => 0.10.2	6 years ago
Gustavo Santos Ferreira	cf1be63b60	add new field to address extractor	6 years ago
Felipe Renan	5d1441d515	Do not store token_hash as address_hash in token balances ... We don't need this extraction because it will try to fetch the balance from the own token in its contract and it doesn't make sense for us. We will stop to store a lot of unnecessary data removing this extraction.	6 years ago
Luke Imhoff	41ea1f55e7	Don't use :replace_all for transactions	6 years ago
Felipe Renan	5ccfb258d6	Send unfetched token balances to be fetched by TokenBalance.Fetcher	6 years ago
Felipe Renan	0fbd09653c	Increase batch_size in TokenBalance.Fetcher	6 years ago
Luke Imhoff	91042f4af3	Shrinkable BufferedTasks ... `Indexer.BufferedTask` now uses an `Indexer.BoundQueue` instead of a `:queue`. All `Indexer.BufferedTask`s register with the memory monitor passed in to `BufferedTask.init/1` and will shrink their bound queue when asked. If they run out of work and have shrunk `c:init/2` will be called again on the callback module. `Indexer.Logger.process/1` will generate iodata of the pid and registered name of a process if available.	6 years ago
Luke Imhoff	0976a572ff	Implement Enumerable for BoundQueue	6 years ago
Luke Imhoff	9f09ee0226	Remove init_chunk_size ... It didn't really do much besides decided when to send GenServer.calls from the init_task.	6 years ago
Luke Imhoff	67ca25725d	Don't batch until run ... Instead of queuing up batches, store entries in queue and only form a batch when run is called. This ensures that the batch size is maximized and previously partial adjacent batches will go out as a full batch.	6 years ago
Luke Imhoff	7abc419617	Don't store pid in BufferedTask state	6 years ago
Luke Imhoff	b436d8bf5d	Don't track retry counts ... All implemented of BufferedTask ignore the number and not tracking it saves memory and simplifies BufferedTask itself.	6 years ago
Luke Imhoff	1fce095a35	Shrink Catchup Sequence if its consumes too much memory	6 years ago
Luke Imhoff	cae25195e5	Indexer.Memory.Monitor ... `Indexer.Memory.Monitor` checks if the BEAM memory usage exceeds a set limit (defaults to 1 GiB) and if it does, it asks the process with the most memory that is registered as shrinkable to shrink.	6 years ago
Amanda Sposito	b86335cf34	Add 'transaction_index' in the fetchers to save at 'internal_transaction' ... Add 'transaction_index' in the fetchers to save at 'internal_transaction'	6 years ago
Alex Garibay	0d0aa457cd	Remove null bytes from token name and symbol	6 years ago
Sebastian Abondano	2b8ceb4e32	Update block reward contract beneficiary balances ... Why: * "Parity’s consensus engine allows using a smart contract for block reward calculation." - see https://wiki.parity.io/Block-Reward-Contract for details. This means that we need to use `trace_block` (https://wiki.parity.io/JSONRPC-trace-module#trace_block) to fetch the block reward contract beneficiaries and update their balances. Currently this doesn't happen so there are addresses (block contract reward beneficiaries) with inaccurate balances and missing rows in the `address_coin_balances` table. * Issue link: https://github.com/poanetwork/blockscout/issues/767 This change addresses the need by: * Adding `EthereumJSONRPC.fetch_beneficiaries/2` to help us fetch the block reward contract beneficiaries of a given block range. This only works with Parity. * Editing Indexer's dev and prod config by adding `trace_block` config to `:method_to_url`. * Adding `block_reward_contract_beneficiaries` to `Indexer.AddressExtraction` for it to know how to get addresses in this scenario. * Editing `Indexer.Block.Fetcher.fetch_and_import_range/2` to get block reward contract beneficiaries, create their addresses if they don't yet exist, and update their coin balances.	6 years ago
Felipe Renan	bed1a730dd	Consider mint transfers in the logs to index the address	6 years ago
Luke Imhoff	b1f4696569	Reconnect, rerequest, and resubscribe when websocket disconnects ... * Use struct, `%EthereumJSONRPC.WebSocket.WebSocketClient.Options{}` to hold `web_socket_options`. * `%EthereumJSONRPC.WebSocket.Registration{}` * Record `t:EthereumJSONRPC.request/0` in `%EthereumJSONRPC.WebSocket.Registration{}`, so that the request can be re-requested if the websocket disconnects and reconnects. * Drop server-side `subscription_id` from `%EthereumJSONRPC.WebSocket.Registration{}` as it can retrieved from `request` now. * `EthereumJSONRPC.WebSocket.WebSocketClient` * Track whether the web socket is connected or not, so that requests aren't sent when in the disconnected state. This prevents double-sends as `handle_call` can be called before `onconnect`, which caused that first subscribe to be sent twice. * Instead of the subscriber knowing the server-side subscription id, an opaque `t:reference/0` is used, so that the reference is stable in the subscriber's `%Subscription{}`, but the subscription ID can change on reconnect, which means replacing `subscription_id_to_subscription` in `%WebSocketClient{}` with 3 maps 1. `subscription_id_to_subscription_reference` - used translate server-side, per-connection subscription to the stable reference in `subscription_reference_to_subscription` 2. `subscription_reference_to_subscription_id` - used to convert stable reference to subscription id to unsubscribe. 3. `subscription_reference_to_subscription` - used to get full Subscription, including the subscriber_pid for when subscription messages are delivered or the subscription needs to be resubscribed using the `web_socket_options`. * `ondisconnect`, `:reconnect` immediately. * When disconnecting, tell unsubscribers they were successful. * Hold any json_rpc or subscribe requests to retry on reconnect. * Hold any subscriptions to resubscribe on reconnect. * `onconnect` * re-request the saved json_rpc and subscribe requests. * issue new requests to re-establish the subscriptions that had already succeeded in their subscribe requests. * `%EthereumJSONRPC.Subscription{}` * Replace `id`, which did match the server created ID, with a `t:reference/0`, which allows for indirection and for the server created ID to change on reconnect while the subscriber does not need to care about the reconnections. * `EthereumJSONRPC.WebSocket` * `options` is switched from a Keyword.t to a struct, `t`, so that it is easier to pattern match for the reconnect logic in `WebSocketClient`. * `web_socket_options` is no longer a generic `term` because the code did require it to be a map with a `:web_socket` key for the `t:GenServer.server/0`.	6 years ago
Luke Imhoff	8bb151da71	Log and retry import errors in uncle fetcher	6 years ago
Luke Imhoff	fcf5e6ef99	InternalTransaction.Fetcher buffer tuples instead maps ... ```elixir test "memory" do InternalTransaction.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(InternalTransaction.Fetcher) Stream.repeatedly(fn -> %{block_number: block_number(), hash: transaction_hash()} end) |> Enum.take(100) |> InternalTransaction.Fetcher.async_fetch() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.82 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	18e06d14f7	Indexer.CoinBalance.Fetcher tuple instead of map in buffers ... ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 372,444 | 230,484 | 61.88 | | 1,000 | After | 142,924 | 88,636 | 62.02 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	f04ea541d0	Indexer.CoinBalance.Fetcher bytes and integer instead of string in ... buffers ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | String (bytes) | Bytes and Integer (bytes) | Bytes / String (%) | |----------|--------------------|----------------|---------------------------|--------------------| | 100 | Before | 55,244 | 34,548 | 62.54 | | 100 | After | 21,764 | 13,860 | 63.68 | | 1,000 | Before | 601,972 | 372,444 | 61.87 | | 1,000 | After | 230,596 | 142,924 | 61.98 | | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.83 | | 100,000 | Before | 16,586,644 | 11,519,724 | 69.45 | | 100,000 | After | 16,583,636 | 11,516,716 | 69.45 |	6 years ago
Luke Imhoff	8d542f07bd	Buffer tuples instead of maps ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 1_000 |> build_list(:token_balance) |> Enum.map(& %{&1 | address_hash: &1.address.hash, token_contract_address_hash: address_hash()}) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 233,500 | 145,828 | 62.45 | | 1,000 | After | 230,492 | 142,820 | 61.96 | | 50,000 | Before | 8,001,028 | 6,668,188 | 83.34 | | 50,000 | After | 7,998,020 | 6,665,180 | 83.35 |	6 years ago
Luke Imhoff	b313af6d7f	Buffer hash bytes instead of hash strings ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Hash Strings (bytes) | Hash Bytes (bytes) | Bytes / String (%) | |----------------|--------------------|----------------------|--------------------|--------------------| | 1,000 | Before | 233,500 | 233,500 | 100.00 | | 1,000 | After | 233,500 | 230,492 | 98.71 | | 10,000 | Before | 2,549,476 | 1,577,180 | 61.86 | | 10,000 | After | 2,546,468 | 1,574,172 | 61.82 |	6 years ago
Luke Imhoff	48b1de4617	Buffer formatted params instead of %TokenBalance{} ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | `%TokenBalance{}`s (bytes) | `%{token_contract_address_hash: Hash.to_string(token_contract_address_hash), address_hash: Hash.to_string(address_hash), block_number: block_number}` (bytes) | Map / Struct (%) | |----------------|--------------------|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| | 1,000 | Before | 2,549,476 | 233,500 | 9.16 | | 1,000 | After | 2,546,468 | 233,500 | 9.17 | | 10,000 | Before | 19,903,172 | 2,549,476 | 12.81 | | 10,000 | After | 19,900,260 | 2,546,468 | 12.80 |	6 years ago
Luke Imhoff	753319de4a	Store ranges as tuples in Sequence :queue ... Found a memory savings for `Indexer.Sequence`: store Ranges as tuples in the `:queue` as the tuples take up less space than than the `%Range{}` struct, but only after garbage-collection. The incoming and outgoing format is still Ranges, so this is a non-breaking implementation detail only. ```elixir test "memory usage" do {:ok, pid} = Sequence.start_link(ranges: [max_block_number..0], step: -10) Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | max_block_number | Garbage Collection | Ranges (bytes) | Tuples (bytes) | Tuples / Ranges (%) | |------------------|--------------------|----------------|----------------|---------------------| | 5,000,000 | Before | 41,050,964 | 22,445,660 | 54.68 | | 5,000,000 | After | 34,386,756 | 23,879,996 | 69.45 | | 10,000,000 | Before | 80,889,772 | 47,928,116 | 59.24 | | 10,000,000 | After | 71,303,316 | 49,516,492 | 69.44 |	6 years ago
Sebastian Abondano	f58bf1586f	Log InternalTransaction.Fetcher errors as error ... Why: * Currently errors when importing internal transactions are being logged with 'debug' level. Making them 'error' level would make sure we notice them quicker and that they're prioritized accordingly. * Issue link: n/a	6 years ago
Amanda Sposito	0ed9020a96	Alter 'Realtime::Fetcher' and 'Catchup::Fetcher' to specify the broadcast type. ... The Token Counter needs to listen to both :realtime and :catchup events so it can keep up to date with the counting, but the 'Live Update' in the front-end doesn't need to hear the catchup events.	6 years ago