487 Commits (9156f2d2e9f1d4b1d12b207f2f2a1978667ab1ee)

Author	SHA1	Message	Date
Felipe Renan	5d1441d515	Do not store token_hash as address_hash in token balances ... We don't need this extraction because it will try to fetch the balance from the own token in its contract and it doesn't make sense for us. We will stop to store a lot of unnecessary data removing this extraction.	6 years ago
Luke Imhoff	41ea1f55e7	Don't use :replace_all for transactions	6 years ago
Luke Imhoff	96b97f0dc1	Switch logs to composite primary key ... * Drop id. * Make (transaaction_hash, index) the primary key. This ports the change from `internal_transactions`.	6 years ago
Felipe Renan	5ccfb258d6	Send unfetched token balances to be fetched by TokenBalance.Fetcher	6 years ago
Felipe Renan	0fbd09653c	Increase batch_size in TokenBalance.Fetcher	6 years ago
Luke Imhoff	91042f4af3	Shrinkable BufferedTasks ... `Indexer.BufferedTask` now uses an `Indexer.BoundQueue` instead of a `:queue`. All `Indexer.BufferedTask`s register with the memory monitor passed in to `BufferedTask.init/1` and will shrink their bound queue when asked. If they run out of work and have shrunk `c:init/2` will be called again on the callback module. `Indexer.Logger.process/1` will generate iodata of the pid and registered name of a process if available.	6 years ago
Luke Imhoff	0976a572ff	Implement Enumerable for BoundQueue	6 years ago
Luke Imhoff	9f09ee0226	Remove init_chunk_size ... It didn't really do much besides decided when to send GenServer.calls from the init_task.	6 years ago
Luke Imhoff	67ca25725d	Don't batch until run ... Instead of queuing up batches, store entries in queue and only form a batch when run is called. This ensures that the batch size is maximized and previously partial adjacent batches will go out as a full batch.	6 years ago
Luke Imhoff	7abc419617	Don't store pid in BufferedTask state	6 years ago
Luke Imhoff	b436d8bf5d	Don't track retry counts ... All implemented of BufferedTask ignore the number and not tracking it saves memory and simplifies BufferedTask itself.	6 years ago
Luke Imhoff	1fce095a35	Shrink Catchup Sequence if its consumes too much memory	6 years ago
Luke Imhoff	cae25195e5	Indexer.Memory.Monitor ... `Indexer.Memory.Monitor` checks if the BEAM memory usage exceeds a set limit (defaults to 1 GiB) and if it does, it asks the process with the most memory that is registered as shrinkable to shrink.	6 years ago
Amanda Sposito	b86335cf34	Add 'transaction_index' in the fetchers to save at 'internal_transaction' ... Add 'transaction_index' in the fetchers to save at 'internal_transaction'	6 years ago
Alex Garibay	0d0aa457cd	Remove null bytes from token name and symbol	6 years ago
Sebastian Abondano	1cbaafd894	Adds log rotation ... Why: * To limit the total size of the logs retained while still allowing analysis of recent events. * Issue link: n/a This change addresses the need by: * Editing the logger backends to do file rotation every 50mb and keep the last 19 log files per logger backend. 50mb * 20 = ~1gb	6 years ago
Sebastian Abondano	2b8ceb4e32	Update block reward contract beneficiary balances ... Why: * "Parity’s consensus engine allows using a smart contract for block reward calculation." - see https://wiki.parity.io/Block-Reward-Contract for details. This means that we need to use `trace_block` (https://wiki.parity.io/JSONRPC-trace-module#trace_block) to fetch the block reward contract beneficiaries and update their balances. Currently this doesn't happen so there are addresses (block contract reward beneficiaries) with inaccurate balances and missing rows in the `address_coin_balances` table. * Issue link: https://github.com/poanetwork/blockscout/issues/767 This change addresses the need by: * Adding `EthereumJSONRPC.fetch_beneficiaries/2` to help us fetch the block reward contract beneficiaries of a given block range. This only works with Parity. * Editing Indexer's dev and prod config by adding `trace_block` config to `:method_to_url`. * Adding `block_reward_contract_beneficiaries` to `Indexer.AddressExtraction` for it to know how to get addresses in this scenario. * Editing `Indexer.Block.Fetcher.fetch_and_import_range/2` to get block reward contract beneficiaries, create their addresses if they don't yet exist, and update their coin balances.	6 years ago
Felipe Renan	bed1a730dd	Consider mint transfers in the logs to index the address	6 years ago
Luke Imhoff	b1f4696569	Reconnect, rerequest, and resubscribe when websocket disconnects ... * Use struct, `%EthereumJSONRPC.WebSocket.WebSocketClient.Options{}` to hold `web_socket_options`. * `%EthereumJSONRPC.WebSocket.Registration{}` * Record `t:EthereumJSONRPC.request/0` in `%EthereumJSONRPC.WebSocket.Registration{}`, so that the request can be re-requested if the websocket disconnects and reconnects. * Drop server-side `subscription_id` from `%EthereumJSONRPC.WebSocket.Registration{}` as it can retrieved from `request` now. * `EthereumJSONRPC.WebSocket.WebSocketClient` * Track whether the web socket is connected or not, so that requests aren't sent when in the disconnected state. This prevents double-sends as `handle_call` can be called before `onconnect`, which caused that first subscribe to be sent twice. * Instead of the subscriber knowing the server-side subscription id, an opaque `t:reference/0` is used, so that the reference is stable in the subscriber's `%Subscription{}`, but the subscription ID can change on reconnect, which means replacing `subscription_id_to_subscription` in `%WebSocketClient{}` with 3 maps 1. `subscription_id_to_subscription_reference` - used translate server-side, per-connection subscription to the stable reference in `subscription_reference_to_subscription` 2. `subscription_reference_to_subscription_id` - used to convert stable reference to subscription id to unsubscribe. 3. `subscription_reference_to_subscription` - used to get full Subscription, including the subscriber_pid for when subscription messages are delivered or the subscription needs to be resubscribed using the `web_socket_options`. * `ondisconnect`, `:reconnect` immediately. * When disconnecting, tell unsubscribers they were successful. * Hold any json_rpc or subscribe requests to retry on reconnect. * Hold any subscriptions to resubscribe on reconnect. * `onconnect` * re-request the saved json_rpc and subscribe requests. * issue new requests to re-establish the subscriptions that had already succeeded in their subscribe requests. * `%EthereumJSONRPC.Subscription{}` * Replace `id`, which did match the server created ID, with a `t:reference/0`, which allows for indirection and for the server created ID to change on reconnect while the subscriber does not need to care about the reconnections. * `EthereumJSONRPC.WebSocket` * `options` is switched from a Keyword.t to a struct, `t`, so that it is easier to pattern match for the reconnect logic in `WebSocketClient`. * `web_socket_options` is no longer a generic `term` because the code did require it to be a map with a `:web_socket` key for the `t:GenServer.server/0`.	6 years ago
Konstantin Zolotarev	0604f7a234	Fixes for pr comments	6 years ago
Konstantin Zolotarev	f89a27adf4	Fixes for ganache integration	6 years ago
Luke Imhoff	8bb151da71	Log and retry import errors in uncle fetcher	6 years ago
Luke Imhoff	fcf5e6ef99	InternalTransaction.Fetcher buffer tuples instead maps ... ```elixir test "memory" do InternalTransaction.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(InternalTransaction.Fetcher) Stream.repeatedly(fn -> %{block_number: block_number(), hash: transaction_hash()} end) |> Enum.take(100) |> InternalTransaction.Fetcher.async_fetch() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.82 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	18e06d14f7	Indexer.CoinBalance.Fetcher tuple instead of map in buffers ... ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 372,444 | 230,484 | 61.88 | | 1,000 | After | 142,924 | 88,636 | 62.02 | | 100,000 | Before | 11,519,724 | 9,600,436 | 83.34 | | 100,000 | After | 11,516,716 | 9,597,428 | 83.33 |	6 years ago
Luke Imhoff	f04ea541d0	Indexer.CoinBalance.Fetcher bytes and integer instead of string in ... buffers ```elixir test "memory" do CoinBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(CoinBalance.Fetcher) Stream.repeatedly(fn -> %{address_hash: address_hash(), block_number: block_number()} end) |> Enum.take(100) |> CoinBalance.Fetcher.async_fetch_balances() Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> elem(1) |> IO.inspect(label: "AFTER GC") end ``` | Balances | Garbage Collection | String (bytes) | Bytes and Integer (bytes) | Bytes / String (%) | |----------|--------------------|----------------|---------------------------|--------------------| | 100 | Before | 55,244 | 34,548 | 62.54 | | 100 | After | 21,764 | 13,860 | 63.68 | | 1,000 | Before | 601,972 | 372,444 | 61.87 | | 1,000 | After | 230,596 | 142,924 | 61.98 | | 10,000 | Before | 4,119,668 | 2,546,460 | 61.81 | | 10,000 | After | 1,574,172 | 973,260 | 61.83 | | 100,000 | Before | 16,586,644 | 11,519,724 | 69.45 | | 100,000 | After | 16,583,636 | 11,516,716 | 69.45 |	6 years ago
Luke Imhoff	8d542f07bd	Buffer tuples instead of maps ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 1_000 |> build_list(:token_balance) |> Enum.map(& %{&1 | address_hash: &1.address.hash, token_contract_address_hash: address_hash()}) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Map (bytes) | Tuple (bytes) | Tuple / Map (%) | |----------------|--------------------|-------------|---------------|-----------------| | 1,000 | Before | 233,500 | 145,828 | 62.45 | | 1,000 | After | 230,492 | 142,820 | 61.96 | | 50,000 | Before | 8,001,028 | 6,668,188 | 83.34 | | 50,000 | After | 7,998,020 | 6,665,180 | 83.35 |	6 years ago
Luke Imhoff	b313af6d7f	Buffer hash bytes instead of hash strings ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | Hash Strings (bytes) | Hash Bytes (bytes) | Bytes / String (%) | |----------------|--------------------|----------------------|--------------------|--------------------| | 1,000 | Before | 233,500 | 233,500 | 100.00 | | 1,000 | After | 233,500 | 230,492 | 98.71 | | 10,000 | Before | 2,549,476 | 1,577,180 | 61.86 | | 10,000 | After | 2,546,468 | 1,574,172 | 61.82 |	6 years ago
Luke Imhoff	48b1de4617	Buffer formatted params instead of %TokenBalance{} ... ```elixir test "memory" do TokenBalance.Supervisor.Case.start_supervised!(json_rpc_named_arguments: []) pid = Process.whereis(TokenBalance.Fetcher) 10_000 |> insert_list(:token_balance) |> TokenBalance.Fetcher.async_fetch() Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | Token Balances | Garbage Collection | `%TokenBalance{}`s (bytes) | `%{token_contract_address_hash: Hash.to_string(token_contract_address_hash), address_hash: Hash.to_string(address_hash), block_number: block_number}` (bytes) | Map / Struct (%) | |----------------|--------------------|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| | 1,000 | Before | 2,549,476 | 233,500 | 9.16 | | 1,000 | After | 2,546,468 | 233,500 | 9.17 | | 10,000 | Before | 19,903,172 | 2,549,476 | 12.81 | | 10,000 | After | 19,900,260 | 2,546,468 | 12.80 |	6 years ago
Luke Imhoff	753319de4a	Store ranges as tuples in Sequence :queue ... Found a memory savings for `Indexer.Sequence`: store Ranges as tuples in the `:queue` as the tuples take up less space than than the `%Range{}` struct, but only after garbage-collection. The incoming and outgoing format is still Ranges, so this is a non-breaking implementation detail only. ```elixir test "memory usage" do {:ok, pid} = Sequence.start_link(ranges: [max_block_number..0], step: -10) Process.info(pid, :memory) |> IO.inspect(label: "BEFORE GC") :erlang.garbage_collect(pid) Process.info(pid, :memory) |> IO.inspect(label: "AFTER GC") end ``` | max_block_number | Garbage Collection | Ranges (bytes) | Tuples (bytes) | Tuples / Ranges (%) | |------------------|--------------------|----------------|----------------|---------------------| | 5,000,000 | Before | 41,050,964 | 22,445,660 | 54.68 | | 5,000,000 | After | 34,386,756 | 23,879,996 | 69.45 | | 10,000,000 | Before | 80,889,772 | 47,928,116 | 59.24 | | 10,000,000 | After | 71,303,316 | 49,516,492 | 69.44 |	6 years ago
Sebastian Abondano	f58bf1586f	Log InternalTransaction.Fetcher errors as error ... Why: * Currently errors when importing internal transactions are being logged with 'debug' level. Making them 'error' level would make sure we notice them quicker and that they're prioritized accordingly. * Issue link: n/a	6 years ago
Amanda Sposito	0ed9020a96	Alter 'Realtime::Fetcher' and 'Catchup::Fetcher' to specify the broadcast type. ... The Token Counter needs to listen to both :realtime and :catchup events so it can keep up to date with the counting, but the 'Live Update' in the front-end doesn't need to hear the catchup events.	6 years ago
Luke Imhoff	1918040eeb	Explorer.Chain.Import.Runner behaviour	6 years ago
Luke Imhoff	59c7c38921	Extract Explorer.Chain.Import.Address.TokenBalances ... Standardized on :address_token_balances instead of :token_balances for options.	6 years ago
Luke Imhoff	991a2fe672	Extract Explorer.Chain.Import.Transactions	6 years ago
Luke Imhoff	22cb89058a	Extract Explorer.Chain.Import.Tokens	6 years ago
Luke Imhoff	14aa12e9bd	Extract Explorer.Chain.Import.TokenTransfers	6 years ago
Luke Imhoff	753fbd8bd5	Remove obsolete receipts	6 years ago
Luke Imhoff	5ffb54129f	Extract Explorer.Chain.Import.Logs	6 years ago
Luke Imhoff	4842462667	Extract Explorer.Chain.Import.Block.SecondDegreeRelations	6 years ago
Luke Imhoff	56f0365239	Extract Explorer.Chain.Import.Blocks	6 years ago
Luke Imhoff	03557f4200	Extract Explorer.Chain.Address.CoinBalances ... Standardize on `:address_coin_balances` instead of `:balances` for option names.	6 years ago
Nam Chu Hoai	d17917ce30	Allow unsecure WebSocket connections ... Some setup like ours will have the WebSocket connection in a private network to a local node and thus do not require a secure connection. This change allows to specify ws:// instead of wss:// endpoints as well as reading from the ENV var for the geth config of the indexer Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Luke Imhoff	147a7e14fb	Don't do async imports with empty lists	6 years ago
Luke Imhoff	2e3627d536	Don't run imports for empty params	6 years ago
Andrew Cravenho	9892d1bd17	Add env variables to the production parity files	6 years ago
Luke Imhoff	bf7e8f11c5	Unsupport -> Unsupported	6 years ago
Luke Imhoff	9264a92db3	remaning -> remaining	6 years ago
Luke Imhoff	c60ae9467f	upto -> up to	6 years ago
Luke Imhoff	088589b11a	transaactions -> transactions	6 years ago
Luke Imhoff	65cbce71b4	registred -> registered	6 years ago