Merge branch 'master' into split-js-logic-into-multiple-files

pull/2944/head
Max Alekseenko 5 years ago committed by GitHub
commit 822d444922
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 6
      CHANGELOG.md
  2. 3
      apps/block_scout_web/test/block_scout_web/controllers/api/rpc/address_controller_test.exs
  3. 3
      apps/block_scout_web/test/block_scout_web/controllers/transaction_internal_transaction_controller_test.exs
  4. 5
      apps/ethereum_jsonrpc/lib/ethereum_jsonrpc/block.ex
  5. 139
      apps/explorer/lib/explorer/chain.ex
  6. 28
      apps/explorer/lib/explorer/chain/block/reward.ex
  7. 12
      apps/explorer/lib/explorer/chain/internal_transaction.ex
  8. 105
      apps/explorer/lib/explorer/etherscan.ex
  9. 61
      apps/explorer/lib/explorer/etherscan/logs.ex
  10. 35
      apps/explorer/priv/repo/migrations/20191018140054_add_pending_internal_txs_operation.exs
  11. 25
      apps/explorer/priv/repo/migrations/20191203112646_internal_transactions_add_to_address_hash_index.exs
  12. 2
      apps/explorer/test/explorer/chain/import/runner/blocks_test.exs
  13. 59
      apps/explorer/test/explorer/chain_test.exs
  14. 16
      apps/explorer/test/explorer/etherscan_test.exs

@ -1,24 +1,28 @@
## Current ## Current
### Features ### Features
- [#2835](https://github.com/poanetwork/blockscout/pull/2835), [#2871](https://github.com/poanetwork/blockscout/pull/2871), [#2872](https://github.com/poanetwork/blockscout/pull/2872), [#2886](https://github.com/poanetwork/blockscout/pull/2886), [#2925](https://github.com/poanetwork/blockscout/pull/2925) - Add "block_hash" to logs, token_transfers and internal transactions and "pending blocks operations" approach - [#2835](https://github.com/poanetwork/blockscout/pull/2835), [#2871](https://github.com/poanetwork/blockscout/pull/2871), [#2872](https://github.com/poanetwork/blockscout/pull/2872), [#2886](https://github.com/poanetwork/blockscout/pull/2886), [#2925](https://github.com/poanetwork/blockscout/pull/2925), [#2936](https://github.com/poanetwork/blockscout/pull/2936) - Add "block_hash" to logs, token_transfers and internal transactions and "pending blocks operations" approach
- [#2926](https://github.com/poanetwork/blockscout/pull/2926) - API endpoint: sum balances except burnt address - [#2926](https://github.com/poanetwork/blockscout/pull/2926) - API endpoint: sum balances except burnt address
- [#2918](https://github.com/poanetwork/blockscout/pull/2918) - Add tokenID for tokentx API action explicitly - [#2918](https://github.com/poanetwork/blockscout/pull/2918) - Add tokenID for tokentx API action explicitly
### Fixes ### Fixes
- [#2944](https://github.com/poanetwork/blockscout/pull/2944) - Split js logic into multiple files - [#2944](https://github.com/poanetwork/blockscout/pull/2944) - Split js logic into multiple files
- [#2934](https://github.com/poanetwork/blockscout/pull/2934) - RSK release 1.2.0 breaking changes support
- [#2933](https://github.com/poanetwork/blockscout/pull/2933) - Get rid of deadlock in the query to address_current_token_balance table - [#2933](https://github.com/poanetwork/blockscout/pull/2933) - Get rid of deadlock in the query to address_current_token_balance table
- [#2932](https://github.com/poanetwork/blockscout/pull/2932) - fix duplicate websocket connection - [#2932](https://github.com/poanetwork/blockscout/pull/2932) - fix duplicate websocket connection
- [#2928](https://github.com/poanetwork/blockscout/pull/2928) - Speedup pending block ops int txs to fetch query - [#2928](https://github.com/poanetwork/blockscout/pull/2928) - Speedup pending block ops int txs to fetch query
- [#2924](https://github.com/poanetwork/blockscout/pull/2924) - Speedup address to logs query - [#2924](https://github.com/poanetwork/blockscout/pull/2924) - Speedup address to logs query
- [#2915](https://github.com/poanetwork/blockscout/pull/2915) - Speedup of blocks_without_reward_query - [#2915](https://github.com/poanetwork/blockscout/pull/2915) - Speedup of blocks_without_reward_query
- [#2914](https://github.com/poanetwork/blockscout/pull/2914) - Reduce execution time of stream_unfetched_token_instances query - [#2914](https://github.com/poanetwork/blockscout/pull/2914) - Reduce execution time of stream_unfetched_token_instances query
- [#2908](https://github.com/poanetwork/blockscout/pull/2908) - Fix performance of address page
- [#2906](https://github.com/poanetwork/blockscout/pull/2906) - fix address sum cache - [#2906](https://github.com/poanetwork/blockscout/pull/2906) - fix address sum cache
- [#2902](https://github.com/poanetwork/blockscout/pull/2902) - Offset in blocks retrieval for average block time - [#2902](https://github.com/poanetwork/blockscout/pull/2902) - Offset in blocks retrieval for average block time
- [#2900](https://github.com/poanetwork/blockscout/pull/2900) - check fetched instance metadata in multiple places - [#2900](https://github.com/poanetwork/blockscout/pull/2900) - check fetched instance metadata in multiple places
- [#2899](https://github.com/poanetwork/blockscout/pull/2899) - fix empty buffered task - [#2899](https://github.com/poanetwork/blockscout/pull/2899) - fix empty buffered task
- [#2887](https://github.com/poanetwork/blockscout/pull/2887) - increase chart loading speed - [#2887](https://github.com/poanetwork/blockscout/pull/2887) - increase chart loading speed
- [2910](https://github.com/poanetwork/blockscout/pull/2910) - Reorganize queries and indexes for internal_transactions table
### Chore ### Chore
- [#2896](https://github.com/poanetwork/blockscout/pull/2896) - Disable Parity websockets tests - [#2896](https://github.com/poanetwork/blockscout/pull/2896) - Disable Parity websockets tests

@ -1626,6 +1626,7 @@ defmodule BlockScoutWeb.API.RPC.AddressControllerTest do
transaction: transaction, transaction: transaction,
index: 0, index: 0,
from_address: address, from_address: address,
block_number: block.number,
block_hash: transaction.block_hash, block_hash: transaction.block_hash,
block_index: 0 block_index: 0
) )
@ -1681,6 +1682,7 @@ defmodule BlockScoutWeb.API.RPC.AddressControllerTest do
index: 0, index: 0,
type: :reward, type: :reward,
error: "some error", error: "some error",
block_number: transaction.block_number,
block_hash: transaction.block_hash, block_hash: transaction.block_hash,
block_index: 0 block_index: 0
] ]
@ -1718,6 +1720,7 @@ defmodule BlockScoutWeb.API.RPC.AddressControllerTest do
from_address: address, from_address: address,
transaction: transaction, transaction: transaction,
index: index, index: index,
block_number: transaction.block_number,
block_hash: transaction.block_hash, block_hash: transaction.block_hash,
block_index: index block_index: index
} }

@ -68,7 +68,8 @@ defmodule BlockScoutWeb.TransactionInternalTransactionControllerTest do
assert json_response(conn, 200) assert json_response(conn, 200)
assert Enum.count(items) == 2 # excluding of internal transactions with type=call and index=0
assert Enum.count(items) == 1
end end
test "includes USD exchange rate value for address in assigns", %{conn: conn} do test "includes USD exchange rate value for address in assigns", %{conn: conn} do

@ -437,7 +437,8 @@ defmodule EthereumJSONRPC.Block do
end end
defp entry_to_elixir({key, quantity}) defp entry_to_elixir({key, quantity})
when key in ~w(difficulty gasLimit gasUsed minimumGasPrice number size totalDifficulty) and not is_nil(quantity) do when key in ~w(difficulty gasLimit gasUsed minimumGasPrice number size totalDifficulty paidFees) and
not is_nil(quantity) do
{key, quantity_to_integer(quantity)} {key, quantity_to_integer(quantity)}
end end
@ -451,7 +452,7 @@ defmodule EthereumJSONRPC.Block do
# hash format # hash format
defp entry_to_elixir({key, _} = entry) defp entry_to_elixir({key, _} = entry)
when key in ~w(author extraData hash logsBloom miner mixHash nonce parentHash receiptsRoot sealFields sha3Uncles when key in ~w(author extraData hash logsBloom miner mixHash nonce parentHash receiptsRoot sealFields sha3Uncles
signature stateRoot step transactionsRoot uncles), signature stateRoot step transactionsRoot uncles bitcoinMergedMiningCoinbaseTransaction bitcoinMergedMiningHeader bitcoinMergedMiningMerkleProof hashForMergedMining),
do: entry do: entry
defp entry_to_elixir({"timestamp" = key, timestamp}) do defp entry_to_elixir({"timestamp" = key, timestamp}) do

@ -15,6 +15,7 @@ defmodule Explorer.Chain do
preload: 2, preload: 2,
select: 2, select: 2,
subquery: 1, subquery: 1,
union: 2,
union_all: 2, union_all: 2,
where: 2, where: 2,
where: 3 where: 3
@ -194,9 +195,66 @@ defmodule Explorer.Chain do
direction = Keyword.get(options, :direction) direction = Keyword.get(options, :direction)
paging_options = Keyword.get(options, :paging_options, @default_paging_options) paging_options = Keyword.get(options, :paging_options, @default_paging_options)
InternalTransaction if direction == nil do
|> InternalTransaction.where_nonpending_block() query_to_address_hash_wrapped =
|> InternalTransaction.where_address_fields_match(hash, direction) InternalTransaction
|> InternalTransaction.where_address_fields_match(hash, :to_address_hash)
|> common_where_limit_order(paging_options)
|> wrapped_union_subquery()
query_from_address_hash_wrapped =
InternalTransaction
|> InternalTransaction.where_address_fields_match(hash, :from_address_hash)
|> common_where_limit_order(paging_options)
|> wrapped_union_subquery()
query_created_contract_address_hash_wrapped =
InternalTransaction
|> InternalTransaction.where_address_fields_match(hash, :created_contract_address_hash)
|> common_where_limit_order(paging_options)
|> wrapped_union_subquery()
full_query =
query_to_address_hash_wrapped
|> union(^query_from_address_hash_wrapped)
|> union(^query_created_contract_address_hash_wrapped)
full_wrapped_query =
from(
q in subquery(full_query),
select: q
)
full_wrapped_query
|> order_by(
[q],
desc: q.block_number,
desc: q.transaction_index,
desc: q.index
)
|> preload(transaction: :block)
|> join_associations(necessity_by_association)
|> Repo.all()
else
InternalTransaction
|> InternalTransaction.where_nonpending_block()
|> InternalTransaction.where_address_fields_match(hash, direction)
|> common_where_limit_order(paging_options)
|> preload(transaction: :block)
|> join_associations(necessity_by_association)
|> Repo.all()
end
end
def wrapped_union_subquery(query) do
from(
q in subquery(query),
select: q
)
end
defp common_where_limit_order(query, paging_options) do
query
|> InternalTransaction.where_is_different_from_parent_transaction() |> InternalTransaction.where_is_different_from_parent_transaction()
|> InternalTransaction.where_block_number_is_not_null() |> InternalTransaction.where_block_number_is_not_null()
|> page_internal_transaction(paging_options) |> page_internal_transaction(paging_options)
@ -207,9 +265,6 @@ defmodule Explorer.Chain do
desc: it.transaction_index, desc: it.transaction_index,
desc: it.index desc: it.index
) )
|> preload(transaction: :block)
|> join_associations(necessity_by_association)
|> Repo.all()
end end
@doc """ @doc """
@ -263,10 +318,10 @@ defmodule Explorer.Chain do
paging_options = Keyword.get(options, :paging_options, @default_paging_options) paging_options = Keyword.get(options, :paging_options, @default_paging_options)
if Application.get_env(:block_scout_web, BlockScoutWeb.Chain)[:has_emission_funds] do if Application.get_env(:block_scout_web, BlockScoutWeb.Chain)[:has_emission_funds] do
blocks_range = address_to_transactions_tasks_range_of_blocks(address_hash, options)
rewards_task = rewards_task =
Task.async(fn -> Task.async(fn -> Reward.fetch_emission_rewards_tuples(address_hash, paging_options, blocks_range) end)
Reward.fetch_emission_rewards_tuples(address_hash, paging_options)
end)
[rewards_task | address_to_transactions_tasks(address_hash, options)] [rewards_task | address_to_transactions_tasks(address_hash, options)]
|> wait_for_address_transactions() |> wait_for_address_transactions()
@ -305,21 +360,72 @@ defmodule Explorer.Chain do
|> Enum.take(paging_options.page_size) |> Enum.take(paging_options.page_size)
end end
defp address_to_transactions_tasks_query(options) do
options
|> Keyword.get(:paging_options, @default_paging_options)
|> fetch_transactions()
end
defp address_to_transactions_tasks(address_hash, options) do defp address_to_transactions_tasks(address_hash, options) do
paging_options = Keyword.get(options, :paging_options, @default_paging_options)
direction = Keyword.get(options, :direction) direction = Keyword.get(options, :direction)
necessity_by_association = Keyword.get(options, :necessity_by_association, %{}) necessity_by_association = Keyword.get(options, :necessity_by_association, %{})
base_query = options
paging_options |> address_to_transactions_tasks_query()
|> fetch_transactions() |> join_associations(necessity_by_association)
|> join_associations(necessity_by_association)
base_query
|> Transaction.matching_address_queries_list(direction, address_hash) |> Transaction.matching_address_queries_list(direction, address_hash)
|> Enum.map(fn query -> Task.async(fn -> Repo.all(query) end) end) |> Enum.map(fn query -> Task.async(fn -> Repo.all(query) end) end)
end end
defp address_to_transactions_tasks_range_of_blocks(address_hash, options) do
direction = Keyword.get(options, :direction)
extremums_list =
options
|> address_to_transactions_tasks_query()
|> Transaction.matching_address_queries_list(direction, address_hash)
|> Enum.map(fn query ->
max_query =
from(
q in subquery(query),
select: %{min_block_number: min(q.block_number), max_block_number: max(q.block_number)}
)
max_query
|> Repo.one!()
end)
extremums_list
|> Enum.reduce(%{min_block_number: nil, max_block_number: 0}, fn %{
min_block_number: min_number,
max_block_number: max_number
},
extremums_result ->
current_min_number = Map.get(extremums_result, :min_block_number)
current_max_number = Map.get(extremums_result, :max_block_number)
extremums_result =
if is_number(current_min_number) do
if is_number(min_number) and min_number > 0 and min_number < current_min_number do
extremums_result
|> Map.put(:min_block_number, min_number)
else
extremums_result
end
else
extremums_result
|> Map.put(:min_block_number, min_number)
end
if is_number(max_number) and max_number > 0 and max_number > current_max_number do
extremums_result
|> Map.put(:max_block_number, max_number)
else
extremums_result
end
end)
end
defp wait_for_address_transactions(tasks) do defp wait_for_address_transactions(tasks) do
tasks tasks
|> Task.yield_many(:timer.seconds(20)) |> Task.yield_many(:timer.seconds(20))
@ -2398,6 +2504,7 @@ defmodule Explorer.Chain do
|> for_parent_transaction(hash) |> for_parent_transaction(hash)
|> join_associations(necessity_by_association) |> join_associations(necessity_by_association)
|> where_transaction_has_multiple_internal_transactions() |> where_transaction_has_multiple_internal_transactions()
|> InternalTransaction.where_is_different_from_parent_transaction()
|> InternalTransaction.where_nonpending_block() |> InternalTransaction.where_nonpending_block()
|> page_internal_transaction(paging_options) |> page_internal_transaction(paging_options)
|> limit(^paging_options.page_size) |> limit(^paging_options.page_size)

@ -68,8 +68,10 @@ defmodule Explorer.Chain.Block.Reward do
Returns a list of tuples representing rewards by the EmissionFunds on POA chains. Returns a list of tuples representing rewards by the EmissionFunds on POA chains.
The tuples have the format {EmissionFunds, Validator} The tuples have the format {EmissionFunds, Validator}
""" """
@spec fetch_emission_rewards_tuples(Hash.Address.t(), PagingOptions.t()) :: [{t(), t()}] def fetch_emission_rewards_tuples(address_hash, paging_options, %{
def fetch_emission_rewards_tuples(address_hash, paging_options) do min_block_number: min_block_number,
max_block_number: max_block_number
}) do
address_rewards = address_rewards =
__MODULE__ __MODULE__
|> join_associations() |> join_associations()
@ -77,6 +79,7 @@ defmodule Explorer.Chain.Block.Reward do
|> limit(^paging_options.page_size) |> limit(^paging_options.page_size)
|> order_by([_, block], desc: block.number) |> order_by([_, block], desc: block.number)
|> where([reward], reward.address_hash == ^address_hash) |> where([reward], reward.address_hash == ^address_hash)
|> address_rewards_blocks_ranges_clause(min_block_number, max_block_number, paging_options)
|> Repo.all() |> Repo.all()
case List.first(address_rewards) do case List.first(address_rewards) do
@ -117,4 +120,25 @@ defmodule Explorer.Chain.Block.Reward do
|> join(:inner, [reward], block in assoc(reward, :block)) |> join(:inner, [reward], block in assoc(reward, :block))
|> preload(:block) |> preload(:block)
end end
defp address_rewards_blocks_ranges_clause(query, min_block_number, max_block_number, paging_options) do
if is_number(min_block_number) and max_block_number > 0 and min_block_number > 0 do
cond do
paging_options.page_number == 1 ->
query
|> where([_, block], block.number >= ^min_block_number)
min_block_number == max_block_number ->
query
|> where([_, block], block.number == ^min_block_number)
true ->
query
|> where([_, block], block.number >= ^min_block_number)
|> where([_, block], block.number <= ^max_block_number)
end
else
query
end
end
end end

@ -555,6 +555,18 @@ defmodule Explorer.Chain.InternalTransaction do
) )
end end
def where_address_fields_match(query, address_hash, :to_address_hash) do
where(query, [it], it.to_address_hash == ^address_hash)
end
def where_address_fields_match(query, address_hash, :from_address_hash) do
where(query, [it], it.from_address_hash == ^address_hash)
end
def where_address_fields_match(query, address_hash, :created_contract_address_hash) do
where(query, [it], it.created_contract_address_hash == ^address_hash)
end
def where_is_different_from_parent_transaction(query) do def where_is_different_from_parent_transaction(query) do
where( where(
query, query,

@ -3,7 +3,7 @@ defmodule Explorer.Etherscan do
The etherscan context. The etherscan context.
""" """
import Ecto.Query, only: [from: 2, where: 3, or_where: 3] import Ecto.Query, only: [from: 2, where: 3, or_where: 3, union: 2]
alias Explorer.Etherscan.Logs alias Explorer.Etherscan.Logs
alias Explorer.{Chain, Repo} alias Explorer.{Chain, Repo}
@ -97,6 +97,7 @@ defmodule Explorer.Etherscan do
query query
|> Chain.where_transaction_has_multiple_internal_transactions() |> Chain.where_transaction_has_multiple_internal_transactions()
|> InternalTransaction.where_is_different_from_parent_transaction()
|> InternalTransaction.where_nonpending_block() |> InternalTransaction.where_nonpending_block()
|> Repo.all() |> Repo.all()
end end
@ -121,28 +122,92 @@ defmodule Explorer.Etherscan do
) do ) do
options = Map.merge(@default_options, raw_options) options = Map.merge(@default_options, raw_options)
query = direction =
case options do
%{filter_by: "to"} -> :to
%{filter_by: "from"} -> :from
_ -> nil
end
consensus_blocks =
from( from(
it in InternalTransaction, b in Block,
inner_join: t in assoc(it, :transaction), where: b.consensus == true
inner_join: b in assoc(t, :block),
order_by: [{^options.order_by_direction, t.block_number}],
limit: ^options.page_size,
offset: ^offset(options),
select:
merge(map(it, ^@internal_transaction_fields), %{
block_timestamp: b.timestamp,
block_number: b.number
})
) )
query if direction == nil do
|> Chain.where_transaction_has_multiple_internal_transactions() query =
|> where_address_match(address_hash, options) from(
|> where_start_block_match(options) it in InternalTransaction,
|> where_end_block_match(options) inner_join: b in subquery(consensus_blocks),
|> InternalTransaction.where_nonpending_block() on: it.block_number == b.number,
|> Repo.all() order_by: [
{^options.order_by_direction, it.block_number},
{:desc, it.transaction_index},
{:desc, it.index}
],
limit: ^options.page_size,
offset: ^offset(options),
select:
merge(map(it, ^@internal_transaction_fields), %{
block_timestamp: b.timestamp,
block_number: b.number
})
)
query_to_address_hash_wrapped =
query
|> InternalTransaction.where_address_fields_match(address_hash, :to_address_hash)
|> InternalTransaction.where_is_different_from_parent_transaction()
|> where_start_block_match(options)
|> where_end_block_match(options)
|> Chain.wrapped_union_subquery()
query_from_address_hash_wrapped =
query
|> InternalTransaction.where_address_fields_match(address_hash, :from_address_hash)
|> InternalTransaction.where_is_different_from_parent_transaction()
|> where_start_block_match(options)
|> where_end_block_match(options)
|> Chain.wrapped_union_subquery()
query_created_contract_address_hash_wrapped =
query
|> InternalTransaction.where_address_fields_match(address_hash, :created_contract_address_hash)
|> InternalTransaction.where_is_different_from_parent_transaction()
|> where_start_block_match(options)
|> where_end_block_match(options)
|> Chain.wrapped_union_subquery()
query_to_address_hash_wrapped
|> union(^query_from_address_hash_wrapped)
|> union(^query_created_contract_address_hash_wrapped)
|> Repo.all()
else
query =
from(
it in InternalTransaction,
inner_join: t in assoc(it, :transaction),
inner_join: b in assoc(t, :block),
order_by: [{^options.order_by_direction, t.block_number}],
limit: ^options.page_size,
offset: ^offset(options),
select:
merge(map(it, ^@internal_transaction_fields), %{
block_timestamp: b.timestamp,
block_number: b.number
})
)
query
|> Chain.where_transaction_has_multiple_internal_transactions()
|> InternalTransaction.where_address_fields_match(address_hash, direction)
|> InternalTransaction.where_is_different_from_parent_transaction()
|> where_start_block_match(options)
|> where_end_block_match(options)
|> InternalTransaction.where_nonpending_block()
|> Repo.all()
end
end end
@doc """ @doc """

@ -5,10 +5,10 @@ defmodule Explorer.Etherscan.Logs do
""" """
import Ecto.Query, only: [from: 2, where: 3, subquery: 1, order_by: 3] import Ecto.Query, only: [from: 2, where: 3, subquery: 1, order_by: 3, union: 2]
alias Explorer.{Chain, Repo}
alias Explorer.Chain.{Block, InternalTransaction, Log, Transaction} alias Explorer.Chain.{Block, InternalTransaction, Log, Transaction}
alias Explorer.Repo
@base_filter %{ @base_filter %{
from_block: nil, from_block: nil,
@ -78,25 +78,25 @@ defmodule Explorer.Etherscan.Logs do
logs_query = where_topic_match(Log, prepared_filter) logs_query = where_topic_match(Log, prepared_filter)
query_to_address_hash_wrapped =
logs_query
|> internal_transaction_query(:to_address_hash, prepared_filter, address_hash)
|> Chain.wrapped_union_subquery()
query_from_address_hash_wrapped =
logs_query
|> internal_transaction_query(:from_address_hash, prepared_filter, address_hash)
|> Chain.wrapped_union_subquery()
query_created_contract_address_hash_wrapped =
logs_query
|> internal_transaction_query(:created_contract_address_hash, prepared_filter, address_hash)
|> Chain.wrapped_union_subquery()
internal_transaction_log_query = internal_transaction_log_query =
from(internal_transaction in InternalTransaction.where_nonpending_block(), query_to_address_hash_wrapped
join: transaction in assoc(internal_transaction, :transaction), |> union(^query_from_address_hash_wrapped)
join: log in ^logs_query, |> union(^query_created_contract_address_hash_wrapped)
on: log.transaction_hash == internal_transaction.transaction_hash,
where: internal_transaction.block_number >= ^prepared_filter.from_block,
where: internal_transaction.block_number <= ^prepared_filter.to_block,
where:
internal_transaction.to_address_hash == ^address_hash or
internal_transaction.from_address_hash == ^address_hash or
internal_transaction.created_contract_address_hash == ^address_hash,
select:
merge(map(log, ^@log_fields), %{
gas_price: transaction.gas_price,
gas_used: transaction.gas_used,
transaction_index: transaction.index,
block_number: transaction.block_number
})
)
all_transaction_logs_query = all_transaction_logs_query =
from(transaction in Transaction, from(transaction in Transaction,
@ -256,4 +256,25 @@ defmodule Explorer.Etherscan.Logs do
data.transaction_index >= ^transaction_index data.transaction_index >= ^transaction_index
) )
end end
defp internal_transaction_query(logs_query, direction, prepared_filter, address_hash) do
query =
from(internal_transaction in InternalTransaction.where_nonpending_block(),
join: transaction in assoc(internal_transaction, :transaction),
join: log in ^logs_query,
on: log.transaction_hash == internal_transaction.transaction_hash,
where: internal_transaction.block_number >= ^prepared_filter.from_block,
where: internal_transaction.block_number <= ^prepared_filter.to_block,
select:
merge(map(log, ^@log_fields), %{
gas_price: transaction.gas_price,
gas_used: transaction.gas_used,
transaction_index: transaction.index,
block_number: transaction.block_number
})
)
query
|> InternalTransaction.where_address_fields_match(address_hash, direction)
end
end end

@ -60,11 +60,37 @@ defmodule Explorer.Repo.Migrations.AddPendingInternalTxsOperation do
DECLARE DECLARE
duplicates_count INTEGER := 0; duplicates_count INTEGER := 0;
blocks_scanned INTEGER := 0; blocks_scanned INTEGER := 0;
int_txs_count INTEGER := 0;
temprow RECORD; temprow RECORD;
BEGIN BEGIN
FOR temprow IN SELECT COUNT(*) INTO int_txs_count FROM internal_transactions;
SELECT number, hash FROM blocks IF int_txs_count < 10000000 THEN
LOOP
FOR temprow IN
SELECT block_hash FROM internal_transactions
GROUP BY block_hash, block_index HAVING COUNT(*) > 1
LOOP
duplicates_count := duplicates_count + 1;
RAISE NOTICE '% duplicates, blocks scanned %, block #%, block hash is %', duplicates_count, blocks_scanned, temprow.number , temprow.hash;
IF NOT EXISTS (
SELECT 1 FROM pending_block_operations
WHERE block_hash = temprow.block_hash
) THEN
INSERT INTO pending_block_operations
(block_hash, inserted_at, updated_at, fetch_internal_transactions)
SELECT b.hash, now(), now(), TRUE FROM blocks b
WHERE b.hash = temprow.block_hash;
END IF;
DELETE FROM internal_transactions
WHERE block_hash = temprow.block_hash;
RAISE NOTICE 'DELETED';
END LOOP;
ELSE
FOR temprow IN SELECT number, hash FROM blocks LOOP
blocks_scanned := blocks_scanned + 1; blocks_scanned := blocks_scanned + 1;
IF EXISTS ( IF EXISTS (
SELECT 1 FROM transactions WHERE block_hash = temprow.hash SELECT 1 FROM transactions WHERE block_hash = temprow.hash
@ -93,7 +119,8 @@ defmodule Explorer.Repo.Migrations.AddPendingInternalTxsOperation do
RAISE NOTICE 'DELETED'; RAISE NOTICE 'DELETED';
END IF; END IF;
END IF; END IF;
END LOOP; END LOOP;
END IF;
RAISE NOTICE 'SCRIPT FINISHED'; RAISE NOTICE 'SCRIPT FINISHED';
END $$; END $$;
""") """)

@ -0,0 +1,25 @@
defmodule Explorer.Repo.Migrations.InternalTransactionsAddToAddressHashIndex do
use Ecto.Migration
def change do
execute(
"CREATE INDEX IF NOT EXISTS internal_transactions_from_address_hash_partial_index on public.internal_transactions(from_address_hash, block_number DESC, transaction_index DESC, index DESC) WHERE (((type = 'call') AND (index > 0)) OR (type != 'call'));"
)
execute(
"CREATE INDEX IF NOT EXISTS internal_transactions_to_address_hash_partial_index on public.internal_transactions(to_address_hash, block_number DESC, transaction_index DESC, index DESC) WHERE (((type = 'call') AND (index > 0)) OR (type != 'call'));"
)
execute(
"CREATE INDEX IF NOT EXISTS internal_transactions_created_contract_address_hash_partial_index on public.internal_transactions(created_contract_address_hash, block_number DESC, transaction_index DESC, index DESC) WHERE (((type = 'call') AND (index > 0)) OR (type != 'call'));"
)
drop_if_exists(
index(
:internal_transactions,
[:to_address_hash, :from_address_hash, :created_contract_address_hash, :type, :index],
name: "internal_transactions_to_address_hash_from_address_hash_created"
)
)
end
end

@ -312,7 +312,7 @@ defmodule Explorer.Chain.Import.Runner.BlocksTest do
end end
test "removes duplicate blocks (by hash) before inserting", test "removes duplicate blocks (by hash) before inserting",
%{consensus_block: %{number: block_number, hash: block_hash, miner_hash: miner_hash}, options: options} do %{consensus_block: %{number: _, hash: block_hash, miner_hash: miner_hash}, options: options} do
new_block = params_for(:block, miner_hash: miner_hash, consensus: true) new_block = params_for(:block, miner_hash: miner_hash, consensus: true)
%Ecto.Changeset{valid?: true, changes: block_changes} = Block.changeset(%Block{}, new_block) %Ecto.Changeset{valid?: true, changes: block_changes} = Block.changeset(%Block{}, new_block)

@ -615,7 +615,7 @@ defmodule Explorer.ChainTest do
:transaction :transaction
|> insert(from_address: block.miner) |> insert(from_address: block.miner)
|> with_block() |> with_block(block)
|> Repo.preload(:token_transfers) |> Repo.preload(:token_transfers)
assert [_, {_, _}] = Chain.address_to_transactions_with_rewards(block.miner.hash, direction: :from) assert [_, {_, _}] = Chain.address_to_transactions_with_rewards(block.miner.hash, direction: :from)
@ -623,6 +623,35 @@ defmodule Explorer.ChainTest do
Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: false) Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: false)
end end
test "with transactions if rewards are not in the range of blocks" do
Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: true)
block = insert(:block)
insert(
:reward,
address_hash: block.miner_hash,
block_hash: block.hash,
address_type: :validator
)
insert(
:reward,
address_hash: block.miner_hash,
block_hash: block.hash,
address_type: :emission_funds
)
:transaction
|> insert(from_address: block.miner)
|> with_block()
|> Repo.preload(:token_transfers)
assert [_] = Chain.address_to_transactions_with_rewards(block.miner.hash, direction: :from)
Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: false)
end
test "with emissions rewards, but feature disabled" do test "with emissions rewards, but feature disabled" do
Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: false) Application.put_env(:block_scout_web, BlockScoutWeb.Chain, has_emission_funds: false)
@ -2363,7 +2392,8 @@ defmodule Explorer.ChainTest do
results = [internal_transaction | _] = Chain.transaction_to_internal_transactions(transaction.hash) results = [internal_transaction | _] = Chain.transaction_to_internal_transactions(transaction.hash)
assert 2 == length(results) # excluding of internal transactions with type=call and index=0
assert 1 == length(results)
assert Enum.all?( assert Enum.all?(
results, results,
@ -2508,7 +2538,7 @@ defmodule Explorer.ChainTest do
|> insert() |> insert()
|> with_block() |> with_block()
%InternalTransaction{transaction_hash: first_transaction_hash, index: first_index} = %InternalTransaction{transaction_hash: _, index: _} =
insert(:internal_transaction, insert(:internal_transaction,
transaction: transaction, transaction: transaction,
index: 0, index: 0,
@ -2533,7 +2563,8 @@ defmodule Explorer.ChainTest do
|> Chain.transaction_to_internal_transactions() |> Chain.transaction_to_internal_transactions()
|> Enum.map(&{&1.transaction_hash, &1.index}) |> Enum.map(&{&1.transaction_hash, &1.index})
assert [{first_transaction_hash, first_index}, {second_transaction_hash, second_index}] == result # excluding of internal transactions with type=call and index=0
assert [{second_transaction_hash, second_index}] == result
end end
test "pages by index" do test "pages by index" do
@ -2542,7 +2573,7 @@ defmodule Explorer.ChainTest do
|> insert() |> insert()
|> with_block() |> with_block()
%InternalTransaction{transaction_hash: first_transaction_hash, index: first_index} = %InternalTransaction{transaction_hash: _, index: _} =
insert(:internal_transaction, insert(:internal_transaction,
transaction: transaction, transaction: transaction,
index: 0, index: 0,
@ -2562,19 +2593,29 @@ defmodule Explorer.ChainTest do
transaction_index: transaction.index transaction_index: transaction.index
) )
assert [{first_transaction_hash, first_index}, {second_transaction_hash, second_index}] == %InternalTransaction{transaction_hash: third_transaction_hash, index: third_index} =
insert(:internal_transaction,
transaction: transaction,
index: 2,
block_number: transaction.block_number,
block_hash: transaction.block_hash,
block_index: 2,
transaction_index: transaction.index
)
assert [{second_transaction_hash, second_index}, {third_transaction_hash, third_index}] ==
transaction.hash transaction.hash
|> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {-1}, page_size: 2}) |> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {-1}, page_size: 2})
|> Enum.map(&{&1.transaction_hash, &1.index}) |> Enum.map(&{&1.transaction_hash, &1.index})
assert [{first_transaction_hash, first_index}] == assert [{second_transaction_hash, second_index}] ==
transaction.hash transaction.hash
|> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {-1}, page_size: 1}) |> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {-1}, page_size: 1})
|> Enum.map(&{&1.transaction_hash, &1.index}) |> Enum.map(&{&1.transaction_hash, &1.index})
assert [{second_transaction_hash, second_index}] == assert [{third_transaction_hash, third_index}] ==
transaction.hash transaction.hash
|> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {0}, page_size: 2}) |> Chain.transaction_to_internal_transactions(paging_options: %PagingOptions{key: {1}, page_size: 2})
|> Enum.map(&{&1.transaction_hash, &1.index}) |> Enum.map(&{&1.transaction_hash, &1.index})
end end
end end

@ -541,7 +541,8 @@ defmodule Explorer.EtherscanTest do
found_internal_transactions = Etherscan.list_internal_transactions(transaction.hash) found_internal_transactions = Etherscan.list_internal_transactions(transaction.hash)
assert length(found_internal_transactions) == 3 # excluding of internal transactions with type=call and index=0
assert length(found_internal_transactions) == 2
end end
test "only returns internal transactions that belong to the transaction" do test "only returns internal transactions that belong to the transaction" do
@ -585,7 +586,8 @@ defmodule Explorer.EtherscanTest do
internal_transactions1 = Etherscan.list_internal_transactions(transaction1.hash) internal_transactions1 = Etherscan.list_internal_transactions(transaction1.hash)
assert length(internal_transactions1) == 2 # excluding of internal transactions with type=call and index=0
assert length(internal_transactions1) == 1
internal_transactions2 = Etherscan.list_internal_transactions(transaction2.hash) internal_transactions2 = Etherscan.list_internal_transactions(transaction2.hash)
@ -675,7 +677,7 @@ defmodule Explorer.EtherscanTest do
|> insert() |> insert()
|> with_block() |> with_block()
for index <- 0..2 do for index <- 0..3 do
internal_transaction_details = %{ internal_transaction_details = %{
transaction: transaction, transaction: transaction,
index: index, index: index,
@ -745,7 +747,8 @@ defmodule Explorer.EtherscanTest do
internal_transactions1 = Etherscan.list_internal_transactions(address1.hash) internal_transactions1 = Etherscan.list_internal_transactions(address1.hash)
assert length(internal_transactions1) == 3 # excluding of internal transactions with type=call and index=0
assert length(internal_transactions1) == 2
internal_transactions2 = Etherscan.list_internal_transactions(address2.hash) internal_transactions2 = Etherscan.list_internal_transactions(address2.hash)
@ -760,7 +763,7 @@ defmodule Explorer.EtherscanTest do
|> insert() |> insert()
|> with_block() |> with_block()
for index <- 0..2 do for index <- 0..3 do
internal_transaction_details = %{ internal_transaction_details = %{
transaction: transaction, transaction: transaction,
index: index, index: index,
@ -825,7 +828,8 @@ defmodule Explorer.EtherscanTest do
expected_block_numbers = [second_block.number, third_block.number] expected_block_numbers = [second_block.number, third_block.number]
assert length(found_internal_transactions) == 4 # excluding of internal transactions with type=call and index=0
assert length(found_internal_transactions) == 2
for internal_transaction <- found_internal_transactions do for internal_transaction <- found_internal_transactions do
assert internal_transaction.block_number in expected_block_numbers assert internal_transaction.block_number in expected_block_numbers

Loading…
Cancel
Save