44 Commits (0aa4d2506e6a55a489b06a90055bc64d8d181c3c)

Author	SHA1	Message	Date
Viktor Baranov	29aa23e213	An ability to separate Ecto Repo DB endpoint for the API endpoints	3 years ago
Victor Baranov	cc348ef824	Price for bridged tokens	4 years ago
Paul Tsupikoff	6a5916764d	Consolidate POSDAO contract reading (#2371) ... Previously, staking contract reading was arbitrarily spread across Explorer.Staking.ContractState, Explorer.Staking.PoolsReader and Indexer.Fetcher.StakingPools. Using async fetcher infrastructure for staking contracts is inadequate, as blocks may arrive out of order, while we want fetching to be triggered by newer block arrival. Also, contract calls were not batched enough, and it was hard to follow their sequence across involved modules. Now, on each incoming block, which is newer than the last seen, we fully update not only global contract state, but all validators, pools and delegators. All requests are intelligently compiled into four batches. These are defined together to give better overview of performed actions. Chain.import infrastructure is still used for inserting data into DB to leverage smart batching of INSERT queries. Likelihood calculation is added: it zips results of getPoolsToBeElected and getPoolsLikelihood calls to make a map of addresses. Only POS_STAKING_CONTRACT env variable is required now. Other contract addresses are fetched from it during launch. ABIs are concatenated together during launch as well, as we don't support batch requests against distinct ABIs yet. This should be addressed in further PRs. Up-to-date ABIs copied from build artifacts of posdao-contracts repo. File names now correspond to contract names, README directs to their origin. * Fetch inactive delegators, mark pools and delegators deleted (#2205) * Fetch min stakes and token contract address from StakingAuRa (#2313) * Fetch block reward ratio using validatorRewardPercent getter (#2424) * Add missing `banned_until` field in ON CONFLICT clause for staking pools.	4 years ago
Victor Baranov	cce0ff7aca	Add CHANGELOG entry	5 years ago
Victor Baranov	f9e853de97	Eliminate test secret too	5 years ago
Victor Baranov	5539ad2a77	Revert refactor to generalized cache solution	5 years ago
Victor Baranov	1dda3e50c7	Estimated address count on the main page	5 years ago
Ayrat Badykov	ad68c42fe0	add simple sender	5 years ago
Victor Baranov	455d6d647f	Addresses counter	5 years ago
saneery	ea0d47cfa4	Add mock to send events directly to listener	5 years ago
slightlycyborg	6af33827d8	fixed formatting	5 years ago
slightlycyborg	49dde3e239	Enbl historians in config.exs, disbl in test.exs; add to Application.ex	5 years ago
pasqu4le	da775b07b7	Group Explorer caches ... Problem: a bunch of different caches are under the Explorer.Chain namespaces, cluttering the codebase Solution: create a subdirectory and collect them all under the Explorer.Chain.Cache namespace	5 years ago
maxgrapps	5892fa6de6	content issues, test runned	6 years ago
Ayrat Badykov	390bb24d64	fix tests	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
fvictorio	616658adbf	Show token price in USD	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Luke Imhoff	f67241cfc0	Use COUNT(*) for count_token_transfers_from_token_hash ... By using `COUNT(*)` instead of `COUNT(column)`, Postgres optimizes the count to use the row count recorded in the index. The ETS counter appeared necessary because in 95de1ebcb0, `COUNT(id)` was being used (95de1ebcb0/apps/explorer/lib/explorer/chain/token_transfer.ex (L138)), which seems reasonable in Ecto as you need some column to count, but in Postgres, it is more efficient to count everything with `*` as it implies "count rows". `COUNT(column)` requires Postgres check if any of the values are `NULL` and it does not optimize the query even if the column is `NOT NULL` in most cases like when the `id` was used. Testing on eth60-test database, `COUNT(*)` took 62ms while `COUNT(column)` (in this case `amount` since the table no longer has `id) took 16s demonstrating the vast speed improvement of `*` counting. ``` sql> SELECT COUNT(*) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:02] 1 row retrieved starting from 1 in 70 ms (execution: 62 ms, fetching: 8 ms) sql> SELECT COUNT(amount) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:43] 1 row retrieved starting from 1 in 16 s 862 ms (execution: 16 s 855 ms, fetching: 7 ms) ```	6 years ago
Luke Imhoff	81179f9459	Use :timer functions to make timeout values clearer	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
Amanda Sposito	7eef405a36	Add BlockValidationCounter config to disable it in test env.	6 years ago
Amanda Sposito	845b399281	Add a cache to count the addresses with balance > 0.	6 years ago
Locke	ad606e4df4	Conform to code review. ... Only enable Explorer.ExchangeRates when it's needed. Also changed the required store based on the app needed. IE: BlockScoutWeb tests didn't care about the store but Explorer tests did.	6 years ago
Lokraan	582345cfb7	Create tests for new dependency injection model.	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
fvictorio	73f70745dd	Fix loading of secrets file for test	6 years ago
Gustavo Santos Ferreira	92cf06ef3e	module to calculate market cap	6 years ago
Lucas Narciso	ea3da7fd07	Consolidate Token Holders count info ... The number of token holders is now available in the token page again. It is cached by the TokenHoldersCounter GenServer that periodically updateds an ETS with all pairs of tokens and token holders.	6 years ago
Luke Imhoff	9a8cdc4031	Use Path.absname so that log paths are stable during reload ... Fixes #628 The Phoenix.CodeReloader causes the file loggers to restart and reload their config, but when they reload, the current working directory has shifted from the project root to the OTP app root, so it produces log files under each OTP app (`apps/APP_NAME/logs`). We don't want this, so compute the absolute path in the config files, so that the loaded `Application` environment has an absolute path in it that can't shift when the working directory changes.	6 years ago
Luke Imhoff	c032ea2291	Improved logging ... Have separate loggers for errors and per-application. * `:console` will continue to have all applications and levels. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` * `:error` is only for errors across all environments. It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/error.log`. * `:block_scout_web` is only when `:application` metadata is `:block_scout_web`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/block_scout_web.log`. * `:ethereum_jsonrpc` is only when `:application` metadata is `:ethereum_jsonrpc`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/ethereum_jsonrpc.log`. * `:explorer` is only when `:application` metadata is `:exploer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/explorer.log`. * `:indexer` is only when `:application` metadata is `:indexer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/indexer.log`.	6 years ago
Stamates	faa53be558	Eliminate unused statistics context and gen_server and move statistics function into Chain context.	6 years ago
Alex Garibay	8b16f81b73	Catalog token transfers and tokens during indexing (#484) ... * Fetch token information asynchronously during indexing * Include token transfers imports in main importer * Extract token transfers when importing blocks while indexing * Move ecto to explorer deps * Update References to the Indexer namespace * Add token insertions with main importer * Import token address/type during block indexing	6 years ago
Luke Imhoff	e71781cde8	Increase pool_timeout for tests ... test/indexer/block_fetcher_test.exs:50 was flaky and sometimes fail due to errors like ``` [error] Realtime index stream exited with reason ({:timeout, {GenServer, :call, [#PID<0.596.0>, {:checkout, #Reference<0.851308976.2388656130.16563>, true, 120000}, 5000]}}) ```	6 years ago
Tim Mecklem	89c1416b2c	Allow test environment configuration to be overridden similar to dev	6 years ago
Amanda Sposito	dae9ff2f10	Add support to Pluggable Transport and HTTP transport modules	6 years ago
Alex Garibay	c39154e6eb	Add user accounts (#176) ... * Add user accounts * Delete user_contacts on user delete * Add additional database enforcements * Add missing typespecs	7 years ago
Alex Garibay	aa8dfff5fc	Use configuration to conditionally start processes (#178) ... * Use configuration to conditionally start processes * Ignore application.ex for coverage	7 years ago
Luke Imhoff	5c187420db	Remove ethereumex	7 years ago
Chris McCord	117871a834	WIP ... * Fix bug when restarting index from lst block * Include tests for block sequencer * Check for new blocks periodically after indexing Co-authored-by: Alex Garibay <alex@alexgaribay.com> Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	7 years ago
Luke Imhoff	d6b00db768	Split explorer_web from explorer OTP app ... Co-authored-by: Alex Garibay <alex.garibay@dockyard.com>	7 years ago
Luke Imhoff	526eb526ac	Umbrellify ... Move old root down to apps/explorer. Make root umbrella. Co-authored-by: Alex Garibay <alex.garibay@dockyard.com>	7 years ago
CJ Bryan and Desmond Bowe	18e0d9afde	Cache Balances for Addresses	7 years ago
CJ Bryan and Desmond Bowe	5f937d84ed	FORMAT ALL TEH THINGS	7 years ago
CJ Bryan, Derek Barnes, Desmond Bowe and Matt Olenick	2ba3cc821e	Display Internal Transactions on the Transaction detail page.	7 years ago
CJ Bryan and Matt Olenick	261f15f10d	Double the test database timeout.	7 years ago
Doc Ritezel	5409faf7e3	Upgrade ethereumex	7 years ago
Doc Ritezel	4f326808cd	Update Jasmine test page build structure	7 years ago
CJ Bryan and Matt Olenick	f5dd228551	Use a larger ownership timeout in test mode.	7 years ago