213 Commits (1caf21cf52ccd1116c7b46c4e992d483b2f54f1d)

Author	SHA1	Message	Date
Victor Baranov	cc348ef824	Price for bridged tokens	4 years ago
Paul Tsupikoff	6a5916764d	Consolidate POSDAO contract reading (#2371) ... Previously, staking contract reading was arbitrarily spread across Explorer.Staking.ContractState, Explorer.Staking.PoolsReader and Indexer.Fetcher.StakingPools. Using async fetcher infrastructure for staking contracts is inadequate, as blocks may arrive out of order, while we want fetching to be triggered by newer block arrival. Also, contract calls were not batched enough, and it was hard to follow their sequence across involved modules. Now, on each incoming block, which is newer than the last seen, we fully update not only global contract state, but all validators, pools and delegators. All requests are intelligently compiled into four batches. These are defined together to give better overview of performed actions. Chain.import infrastructure is still used for inserting data into DB to leverage smart batching of INSERT queries. Likelihood calculation is added: it zips results of getPoolsToBeElected and getPoolsLikelihood calls to make a map of addresses. Only POS_STAKING_CONTRACT env variable is required now. Other contract addresses are fetched from it during launch. ABIs are concatenated together during launch as well, as we don't support batch requests against distinct ABIs yet. This should be addressed in further PRs. Up-to-date ABIs copied from build artifacts of posdao-contracts repo. File names now correspond to contract names, README directs to their origin. * Fetch inactive delegators, mark pools and delegators deleted (#2205) * Fetch min stakes and token contract address from StakingAuRa (#2313) * Fetch block reward ratio using validatorRewardPercent getter (#2424) * Add missing `banned_until` field in ON CONFLICT clause for staking pools.	4 years ago
Victor Baranov	8e4b354664	Disable flaky tests	4 years ago
Victor Baranov	27964a5e3d	Add unit tests	5 years ago
Victor Baranov	c6693596d5	Finalize coin balance history chart update	5 years ago
Victor Baranov	d227729db6	Get rid of all first traces	5 years ago
Victor Baranov	f8cf0d194b	Revert realtime fetcher small skips feature	5 years ago
Ayrat Badykov	767d8c9999	add separate process for pending ops removal	5 years ago
Ayrat Badykov	605a4a27f2	fix tests	5 years ago
Victor Baranov	38f2e0f7b5	Offset for average block time	5 years ago
pasqu4le	6a933d514f	internal_txs	5 years ago
Ayrat Badykov	d37fb1d579	fix most tests	5 years ago
Ayrat Badykov	f385a1b44d	fix tests	5 years ago
pasqu4le	e8c2fa554e	Allow Realtime Fetcher to wait for small skips ... Problem: Sometimes the realtime fetcher's `newHeads` websocket subscription skips a small amount of block numbers, just to be followed by those very same numbers that were skipped, without repeating the number that caused the skipping. This may be cause by out-of-order events or be a legitimate reorg, but in any case this causes close-in-time refetching which in turn leads to async insertions problems. Solution: allow the Realtime fetcher to keep track of a small (settable) skip and wait for the block number inside the skip to arrive before fetching the whole window. Obviously if it gets notified of the blocks following the skip (or it is forced into polling) the whole window gets fetched and inserted.	5 years ago
Ayrat Badykov	de497d8d94	fix token updating	5 years ago
Ayrat Badykov	351771907f	add TokenUpdater.Supervisor case	5 years ago
Ayrat Badykov	ad1f1c1826	Revert "Filter pending logs"	5 years ago
pasqu4le	144f39db5b	Remove duplicate params in `Indexer.Fetcher.TokenBalance` ... Problem: `Explorer.Chain.Import.Runner.Address.TokenBalances` fails with the `cardinality_violation` error because it tries to insert more than once token balances (for the same {token_contract_address_hash, address_hash, block_number} tuple) in a single database transaction. Solution: filter out duplicate parameters in `Indexer.Fetcher.TokenBalance` so that no duplicates are sent to be imported.	6 years ago
saneery	50794d8a43	fix test	6 years ago
pasqu4le	2aec1f4c9f	Removes duplicate entries from Indexer.Fetcher.UncleBlock	6 years ago
saneery	741d0005bd	mix format	6 years ago
saneery	b4376e027c	filter pending logs	6 years ago
Ayrat Badykov	55854d9b3b	add test	6 years ago
Ayrat Badykov	a9bbf8acd4	fix retries	6 years ago
saneery	8d2ecb9f61	update pools fetcher	6 years ago
maxgrapps	5892fa6de6	content issues, test runned	6 years ago
saneery	147f0c4347	edit staking pools fetcher test	6 years ago
Ayrat Badykov	e49412f389	remove temporary worker modules	6 years ago
saneery	eb4154f14e	batching pool data requests	6 years ago
goodsoft	8e6ee4ceb1	Use eth_getUncleByBlockHashAndIndex for uncle block fetching ... Parity returns `null` for many uncle blocks, if they are fetched using `eth_getBlockByHash`. This is most probably caused by the fact that Parity only keeps full block data for recent non-consensus blocks. This causes uncle_block fetcher to loop endlessly trying to fetch "missing" blocks, hogging both app server and Parity resources. Instead, we use `eth_getUncleByBlockHashAndIndex` method, which works for all uncle blocks. As we didn't previously store index of an uncle block within a nephew block, a new field and a temporary fixup fetcher is added to get the index from nephew blocks.	6 years ago
saneery	b158d71219	add staking pools fetcher test	6 years ago
Paul Tsupikoff	f02e2d8841	Remove internal tx and token balance fetching from realtime fetcher (#1724) ... Internal transactions and token balances are the slowest to fetch and import. We'd rather have almost instant import of almost all block-related info than frequent timeouts in realtime fetcher and handing blocks over to catchup fetcher which imports this data asynchronously anyway.	6 years ago
Paul Tsupikoff	c423a41e49	Flatten indexer module hierarchy and supervisor tree (#1718) ... * Add a defsupervisor macro with common supervisor functionality This macro allows removing more than a dozen of copy-pasted `supervisor.ex` modules. The `Indexer.Fetcher` module will be expanded with even more common fetcher functionality in upcoming PRs. * Flatten indexer module hierarchy and remove generated supervisors Modules are organized into 3 directories: `fetcher`, `transform` and `temporary`. Supervisors are replaced with `use Indexer.Fetcher` clause. `token_balances.ex` and block fetchers are left as-is for now, as they require more invasive code refactoring, which will be done in upcoming PRs. * Flatten supervisor tree There is no need in separate `Block.Supervisor` hierarchy, so its children are placed in the root supervisor tree. The tree itself is moved to `Indexer.Supervisor` module to reduce amount of directories. * Add CHANGELOG.md entry	6 years ago
Ayrat Badykov	4444f15e62	fix tests	6 years ago
goodsoft	0e15873fbc	Remove obsolete InvalidConsensus.Worker ... This worker only handled one of invalid consensus cases, i.e. when a parent block lost consensus, but the child one didn't. Even then it worked only when all blocks are reliably and sequentially imported (certainly not our case), and only once on indexer launch. Now this particular case is covered by previous commit, and we don't need this worker at all.	6 years ago
Ayrat Badykov	9487d16201	Update fetcher_test.exs	6 years ago
Ayrat Badykov	a6a0d4c5a9	fix test	6 years ago
Paul Tsupikoff	81557ab415	Add regression test for #1642	6 years ago
Ayrat Badykov	456c5549a5	do not fetch burn address balance	6 years ago
Ayrat Badykov	99e30c5a52	select queries by batches	6 years ago
Paul Tsupikoff	4ceb8a8ed2	Add regression test for #1584 ... When a block contains rewards for two uncle blocks to the same address, the Cardinality Violation error occurs due to rewards not being reduced.	6 years ago
Ayrat Badykov	2d51a7f76c	add logging	6 years ago
Ayrat Badykov	06544e089a	check if code is set	6 years ago
Ayrat Badykov	2abcc44354	check hashes in tests	6 years ago
Ayrat Badykov	cffd555137	add process to fix contract with code	6 years ago
goodsoft	7f43ad5850	Don't refetch internal transactions in FailedCreatedAddress fetcher ... We refetch all internal transactions anyway.	6 years ago
goodsoft	ea3af5e299	Use transaction-level fetch_internal_transactions for Geth ... Internal transaction fetcher is split into two APIs: * `fetch_block_internal_transactions` is used for Parity * `fetch_internal_transactions` is used for the rest of variants `Indexer.InternalTransaction.Fetcher` is updated to take care of that.	6 years ago
Paul Tsupikoff	e13f6f8a8e	Use trace_replayBlockTransactions API for faster tracing	6 years ago
Paul Tsupikoff	2a8def19e2	Allow executing smart contract functions in arbitrarily sized batches ... # Problem There are a lot of `getBalanceOf` token function calls during indexing blockchain. Each of them is sent as a separate JSONRPC request, which creates a lot of unnecessary overhead and clogs archive node workers, which could otherwise be busy performing other JSONRPC queries. Moreover, function name is used as the `id` parameter of JSONRPC queries, which doesn't allow us to simply put multiple requests into one batch. # Solution * Change `EthereumJSONRPC.execute_contract_functions` to accept a list of arbitrary function calls and return a list of results. * Handle `id` field generation in the `EthereumJSONRPC` package instead of leaking implementation details to `Explorer.SmartContract.Reader`. * Change `Token.BalanceReader.get_balance_of` to accept a list of parameter sets and return a list of balances. * Make one batch request instead of launching multiple concurrent tasks in `Indexer.TokenBalances.fetch_token_balances_from_blockchain` # Notes * Protocol details of calling smart contract functions is now fully encapsulated in `EthereumJSONRPC` package. * `SmartContract.Reader.query_contract` is left for backwards compatibility with other existing applications of smart contract requests, but can be fully deprecated at some later time.	6 years ago
Ayrat Badykov	714db43a14	fetch ERC721 token balances if senser is '0x0..0'	6 years ago