39 Commits (92de4ac6554041068e4cb36b63f37081dc41d072)

Author	SHA1	Message	Date
Paul Tsupikoff	6a5916764d	Consolidate POSDAO contract reading (#2371) ... Previously, staking contract reading was arbitrarily spread across Explorer.Staking.ContractState, Explorer.Staking.PoolsReader and Indexer.Fetcher.StakingPools. Using async fetcher infrastructure for staking contracts is inadequate, as blocks may arrive out of order, while we want fetching to be triggered by newer block arrival. Also, contract calls were not batched enough, and it was hard to follow their sequence across involved modules. Now, on each incoming block, which is newer than the last seen, we fully update not only global contract state, but all validators, pools and delegators. All requests are intelligently compiled into four batches. These are defined together to give better overview of performed actions. Chain.import infrastructure is still used for inserting data into DB to leverage smart batching of INSERT queries. Likelihood calculation is added: it zips results of getPoolsToBeElected and getPoolsLikelihood calls to make a map of addresses. Only POS_STAKING_CONTRACT env variable is required now. Other contract addresses are fetched from it during launch. ABIs are concatenated together during launch as well, as we don't support batch requests against distinct ABIs yet. This should be addressed in further PRs. Up-to-date ABIs copied from build artifacts of posdao-contracts repo. File names now correspond to contract names, README directs to their origin. * Fetch inactive delegators, mark pools and delegators deleted (#2205) * Fetch min stakes and token contract address from StakingAuRa (#2313) * Fetch block reward ratio using validatorRewardPercent getter (#2424) * Add missing `banned_until` field in ON CONFLICT clause for staking pools.	4 years ago
Victor Baranov	e9f1b6281a	Finalize bridged tokens	4 years ago
Victor Baranov	089b74aafe	Bridged tokens	4 years ago
Victor Baranov	067b67547a	Fix several tests	5 years ago
Victor Baranov	32d3bb85ed	Disable Pending transactions supervisor	5 years ago
Victor Baranov	f8cf0d194b	Revert realtime fetcher small skips feature	5 years ago
pasqu4le	e8c2fa554e	Allow Realtime Fetcher to wait for small skips ... Problem: Sometimes the realtime fetcher's `newHeads` websocket subscription skips a small amount of block numbers, just to be followed by those very same numbers that were skipped, without repeating the number that caused the skipping. This may be cause by out-of-order events or be a legitimate reorg, but in any case this causes close-in-time refetching which in turn leads to async insertions problems. Solution: allow the Realtime fetcher to keep track of a small (settable) skip and wait for the block number inside the skip to arrive before fetching the whole window. Obviously if it gets notified of the blocks following the skip (or it is forced into polling) the whole window gets fetched and inserted.	5 years ago
saneery	12ebc6fad8	Add LAST_BLOCK var to reindex range of blocks	5 years ago
saneery	1e4de6e41e	Add env to disable indexer supervisor	5 years ago
Ayrat Badykov	6e2572b8cc	mix format	6 years ago
Ayrat Badykov	857ac30058	use seconds	6 years ago
Ayrat Badykov	913147185c	allow to configure token metadata update interval	6 years ago
saneery	e9c61945e6	add staking pools fetcher	6 years ago
Ayrat Badykov	cb920e29f4	fix first block parameter	6 years ago
Paul Tsupikoff	c423a41e49	Flatten indexer module hierarchy and supervisor tree (#1718) ... * Add a defsupervisor macro with common supervisor functionality This macro allows removing more than a dozen of copy-pasted `supervisor.ex` modules. The `Indexer.Fetcher` module will be expanded with even more common fetcher functionality in upcoming PRs. * Flatten indexer module hierarchy and remove generated supervisors Modules are organized into 3 directories: `fetcher`, `transform` and `temporary`. Supervisors are replaced with `use Indexer.Fetcher` clause. `token_balances.ex` and block fetchers are left as-is for now, as they require more invasive code refactoring, which will be done in upcoming PRs. * Flatten supervisor tree There is no need in separate `Block.Supervisor` hierarchy, so its children are placed in the root supervisor tree. The tree itself is moved to `Indexer.Supervisor` module to reduce amount of directories. * Add CHANGELOG.md entry	6 years ago
Ayrat Badykov	11a288120f	decrease token metadata update interval	6 years ago
Ayrat Badykov	946b596abc	allow to set first indexing block	6 years ago
nervehammer	d9387ae495	add option to disable block rewards	6 years ago
zachdaniel	c97ce0c9d8	feat: make replaced transactions disabled	6 years ago
zachdaniel	949a971c25	fix: import clique genesis blocks ... Additionally resolves a few small issues that arose in doing so	6 years ago
Luke Imhoff	3c0adace16	Include import_id and transaction_id in logger metadata ... When `import_id` and `transaction_id` was added in Logger calls, they were not added to the metadata allowed to be shown and so weren't appearing in the logs.	6 years ago
Luke Imhoff	65197f5913	Add shrunk to metadata ... Allow separating shrunk vs unshrunk to monitor how often shrinking occurs.	6 years ago
Luke Imhoff	2569d2595e	Move missing_block_range_count and missing_block_count out of prose ... Being in metadata will allow it to be graphed by tools.	6 years ago
Luke Imhoff	1af8f89001	Move step out of prose and into metadata	6 years ago
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Luke Imhoff	c7d614e2ec	Include date in addition to time in log format	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Alex Garibay	e23ebab069	Add transform functions for block indexing	6 years ago
Luke Imhoff	cae25195e5	Indexer.Memory.Monitor ... `Indexer.Memory.Monitor` checks if the BEAM memory usage exceeds a set limit (defaults to 1 GiB) and if it does, it asks the process with the most memory that is registered as shrinkable to shrink.	6 years ago
Luke Imhoff	fdd1090c45	Re-enable Ecto loggers ... Allow Ecto loggers to log to the application-specific log **file**, `log/<Mix.env>/ecto.log` but raise `:console`'s level to `:info`, so that Ecto output does not flood `iex` and make it unusable.	6 years ago
Luke Imhoff	c032ea2291	Improved logging ... Have separate loggers for errors and per-application. * `:console` will continue to have all applications and levels. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` * `:error` is only for errors across all environments. It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/error.log`. * `:block_scout_web` is only when `:application` metadata is `:block_scout_web`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/block_scout_web.log`. * `:ethereum_jsonrpc` is only when `:application` metadata is `:ethereum_jsonrpc`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/ethereum_jsonrpc.log`. * `:explorer` is only when `:application` metadata is `:exploer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/explorer.log`. * `:indexer` is only when `:application` metadata is `:indexer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/indexer.log`.	6 years ago
Luke Imhoff	8f827ecfa6	Use Mox by default for local integration tests ... How to run without Mox is in the root `README.md`. Run all combinations of Variant and Transport on CircleCI to ensure that Mox matches HTTP behavior, so Mox can be used for fast local testing: * Geth, HTTP * Geth, Mox * Parity, HTTP * Parity, Mox Using Mox locally also lessens load on Sokol and the Infura API key the tests use for Geth under the assumption that we do far more local testing than CI testing. May help with #340, but does not fix the root cause, which is now known as `Sequence` not chunking ranges correctly on retry.	6 years ago
Luke Imhoff	466c1e76f6	Request transport module and options for all JSONRPC calls ... To lessen load on Sokol chain nodes during testing and to allow substituting different transports than HTTP, such as IPC, in the future, pass the transport module and its options to any call to `EthereumJSONRPC.json_rpc`. This will allow us to use a `mox` module in local testing, so we only do true chain integration testing in CircleCI.	6 years ago
Luke Imhoff	6b3fd80916	Extract Explorer.Indexer to Indexer ... Resolves #304	7 years ago
Luke Imhoff	27b4bf62d5	mix new indexer --sup	7 years ago