hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Trevor Porter	18d1d0d54e	Separate agent chain configurations, re-add some chains to infra, fix eclipsetestnet & solanatestnet configs (#3327 ) ### Description - solanatestnet and eclipsetestnet were accidentally moved in the agent JSON to not be in the `chains` object. This moves them back to the right place, and adds a test in `infra` to make sure that the agent JSON chains exactly match the environment chain names. There's opportunity for future improvement for exact consistency between TS artifacts / chain metadata and what exists in the agent JSONs, this is tracked here https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3325 - Adds back eclipsetestnet, solanatestnet, and mumbai to the list of testnet4 chains. These seem to have been accidentally removed - I guess we never deployed to mainnet3 since we started writing to aw-multisig.json. So this writes mainnet chains for the first time - Made some modest fixes to the address persistence logic: - Previously, it didn't handle AWS & GCP keys for the same role well. For non-EVM chains, we have a GCP version of a key for submitting transactions. These exist for relayers, kathy, and validators. This resulted in contention for the relayer key - sometimes the GCP key would get written, sometimes the AWS key. For validators, we'd write both the validator checkpoint signer (AWS) and the chain signer (GCP). Now we just flat out prefer the AWS key. We already only wrote the address persistence with EVM keys in mind, so this isn't a regression - Moves to very explicitly specifying which chains will be ran on the `hyperlane` context agents - Instead of sharing a list of chains for both agents and all contract management, now it's separate. This avoids us making quick fixes and removing e.g. non-EVM chains to make ethereum-specific contract tooling happy, but accidentally removing agent support for these chains - Each chain must be specified for each role in an `AgentChainConfig` indicating whether that chain is enabled. This being consistent with the environment's chain names is enforced at runtime & in a test that's ran in CI. Explicitly indicating whether a chain is supported or not and keeping this consistent will hopefully result in fewer unintentional changes to sets of configured chains - This also allows us during partner integrations to merge a PR with the contract deploy but without necessarily running agents yet. We can just set the chain as disabled to not deploy to it ### Drive-by changes - Removes the `disabled` property for chains in our infra - this is no longer a thing and isn't needed anymore ### Related issues Fixes #3326 Fixes #2572 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	b03c90191a	chore: increase relayer memory in helm chart (#3297 ) ### Description Updating the helm chart so we no longer exceed the requested memory. Checked the scraper and validators too and those have enough resources. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746 ) ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
josedev-union	76467daf3a	feat(helm): Support chain signer key as external secret (#2543 ) ### Description This PR sets chain signer keys in the external secret. related to https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2458 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> Co-authored-by: josedev-union <josedev-union@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	1 year ago
josedev-union	0d3dd6076d	helm: Support different signer key per a chain in relayer (#2555 ) ### Description This PR supports a different signer key per a chain in relayer. ### Drive-by changes Add env var `HYP_BASE_RELAYCHAINS` ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: josedev-union <josedev-union@users.noreply.github.com>	1 year ago
Mattie Conover	644db5eee1	Remove unused tracing logic (#2639 ) ### Description When I was working on #2215 I noticed that we are not using either zipkin or jaeger at all and possibly never have. Ripping these out to reduce dead code and also simplify our package dependencies. ### Drive-by changes None ### Related issues Discovered through #2215 ### Backward compatibility Yes - we were not using these features and did not document their existence ### Testing Manual - Verified that when running deploy-agents, the generated configs do not have jaeger or zipkin enabled on `main` and then tested to make sure they still work in this branch	1 year ago
Trevor Porter	c7ad0f4433	Fix deploying validators when there's an empty validator set (#2565 ) ### Description Turns out when there aren't any validators in the set, which is the case for solanadevnet & zbctestnet on the RC context (https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/typescript/infra/config/environments/testnet3/validators.ts#L195), the helm value `hyperlane.validator.configs` wasn't being overridden with CLI value args. So the `configs` in values.yaml, which has a length of 1 with some keys and empty values to illustrate (part of) the expected format of the config, was being used. So then in the Helm template we'd end up iterate through the configs of len 1 :( The fix is to just set the default `configs` to `[]` I also considered adding a check to just skip the chain if the validator set is 0, because there's usually no use in deploying them anyways. But we may be deploying to a chain that used to have a non-zero validator set that we now want to be 0, so we may as well just deploy it and deal with the fact that there will be some helm releases relating to a validator set of 0 🤷 ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility backward compatible ### Testing deployed	1 year ago
Mattie Conover	efad4e1326	Omniscient relayer infra (#2299 ) ### Description - Adds support for omniscient relayers - Updates config formats to allow better overideability by agent role - Enables deploying selected agents rather than having to deploy them all at once - Merges scraper scripts and configuration ### Drive-by changes Lots of cleanup was made in the infrastructure scripts and source as it was already being touched by the above. While this PR touches a lot of files, many of the changes are just because we have so many scripts and a small change to something they all use instantly can lead to many files changed. ### Related issues - Fixes hyperlane-xyz/issues#438 - Fixes hyperlane-xyz/issues#447 - Fixes #1664 ### Backward compatibility _Are these changes backward compatible?_ No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	1 year ago
josedev-union	8a12608220	feat(helm): Support configurable SecretStore type (#2257 ) ### Description This PR makes the SecretStore type as configurable for all external secrets instead of using fixed `ClusterSecretStore` type. ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ No `externalSecrets.clusterSecretStore` should be replaced with `externalSecrets.StoreName` in the values. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Signed-off-by: josedev-union <josedev-union@users.noreply.github.com> Co-authored-by: josedev-union <josedev-union@users.noreply.github.com>	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946 ) ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	599722a7ca	Load all configs by default (#1887 ) ### Description This removes `RUN_ENV` and `BASE_CONFIG` from the Rust config loading process and also from the helm charts and TS configs as appropriate. ### Drive-by changes - Flattens config dir structure - Removes the test config from the git repo. - Fixed a bug in the deploy script that caused the e2e pipeline to break when the config file was removed - Removes `environment` from the config files - Updates TS code to use `runEnv` for deployment - Updates TS code to type environment with `DeployEnvironment` instead of `string` ### Related issues - Relates to #1850 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual - Updated e2e test to verify these don't cause a config loading issue.	2 years ago
Mattie Conover	8105913dea	Improve agent logging (#1797 )	2 years ago
Mattie Conover	65b10f1376	Signer changes (#1557 ) * Signer and signable ethereum signer changes move signers into chain config (chains.rs) Undo removal of traits in hyperlane-core/traits/mod.rs fix chains.rs validator submit remove signers from hyperlane core Announcements and checkpoint Remove test_output trait_builder.rs hyperlane base settings mod base settings signers base settings mod trait builder multisig Undo checkpoint syncer change for now local storage s3 storage validator metadata builder validator submit relayer run locally configs config and deployment fmt * WIP * some refactoring of signing * fmt * Null signer in config * Fix after merge	2 years ago
Mattie Conover	1ff35f63e2	Domain config (#1553 ) * Domain config * implementation -> protocol	2 years ago
Asa Oines	ece89baee7	Relayer dynamically fetches signed checkpoints and merkle proofs (#1486 )	2 years ago
Mattie Conover	e8ed5f88ce	Deploy scraper for testnet3 (#1374 ) * How did those get there * abacus -> hyperlane * remove deprecated domains * Deleted dead code * Update sea-orm * Simplify settings post v2 changes * Split up settings * Update contexts * Fix scraper provider address regression during merge * Updated scraper deployment configs	2 years ago
Asa Oines	73aa067ae2	Add testnet3 environment (#1314 )	2 years ago
Asa Oines	f15632f26c	Finalize rename from abacus->hyperlane (#1312 ) * Rename abacus->hyperlane in agents * Rename in typescript * Rename abacus->hyperlane in dockerfile	2 years ago
Trevor Porter	ea5b584fe8	Support new Gelato relay API (#1041 ) * All compiles * Rename files fwd request -> sponsored call * More scaffolding to get the API key in there * Cleaning up * Rename task_status_call to task_status * Finish rename * rm useForDisabledOriginChains * Introduce TransactionSubmitterType * submitter type -> submission type * Pass around gelato_config instead of the sponsor api key * Final cleanup * Nit * Use default tx submission type * Some renames * Nits after some testing	2 years ago
Trevor Porter	a067a6235f	Minor fixes, deploy (#1051 ) Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	2 years ago
Trevor Porter	f544a5df03	Add gas payment enforcement policy to require a minimum payment (#1031 ) * Add gas payment enforcement policy * Add infra, prettier * Better logs, add IGPs into configs * Fix e2e * Fix clippy	2 years ago
webbhorn	02f9589fdc	Drop rust-Kathy (#713 )	2 years ago
Trevor Porter	75b116cd56	Add "contexts" to infrastructure tooling, this time with no circular dependency (#770 ) * Add contexts * Make validator configs and relayer configs optional * Introduce contexts * Get rid of coreConfig.agent, replace with coreConfig.agents * fix lint * Deployed, just needs some work with RPC urls * Add missing label in rust, disable undesired inboxes * Correctly disable chains, lol * Nit, rename chainNames to contextChainNames * nits * Fix getValidatorKeys * AWS flowcarbon testnet2; don't use default context * Better error when context is undefined * Add relayer funder cronjob context * Update flowcarbon testnet2 img * Fix helm parsing * whitelist * Make Contexts enum higher up * Make CheckpointSyncerType const * Move assertContext to avoid import cycle	2 years ago
Trevor Porter	84400b15c7	Revert "Add "contexts" to infrastructure tooling (#683 )" (#741 ) This reverts commit `e4a2811176`. Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Trevor Porter	e4a2811176	Add "contexts" to infrastructure tooling (#683 ) * Add contexts * Make validator configs and relayer configs optional * Introduce contexts * Get rid of coreConfig.agent, replace with coreConfig.agents * fix lint * Deployed, just needs some work with RPC urls * Add missing label in rust, disable undesired inboxes * Correctly disable chains, lol * Nit, rename chainNames to contextChainNames * nits * Fix getValidatorKeys * AWS flowcarbon testnet2; don't use default context * Better error when context is undefined * Add relayer funder cronjob context * Update flowcarbon testnet2 img * Fix helm parsing * whitelist * Make Contexts enum higher up * Make CheckpointSyncerType const	2 years ago
Trevor Porter	954d4bc866	Remote checkpointer infra (#673 ) * Remote checkpointer infra * rm more values * fix	2 years ago
Trevor Porter	d120a76c1e	No more cached checkpoints in agents (#550 ) * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * PR comments	3 years ago
Nam Chu Hoai	cc6c2db5d8	Testnet deploy (#369 ) * Got keys for testnet * Testnet deploy * Add rust configs * prettier * Nits, update governance config * new agent image * Rename secret reference * Have kathy also checkpoint * Only checkpoint when count > 1 * Move relayer to unique key for (source, remote) pair * prettier * Clean up get-aws-key-addresses * get GCP keys half working * let's clean this up * First steps at refactor * Migrate delete-keys over * support key rotation * Create and get key all working * update relayer external secret * Prettier * Update ethers-rs * Debugging testnet deployment (#370) * Dont fail on inbox checkpointing tx error * fix docker build * Debug * More instrumentation * Lint * Use the correct signers * Lint * prettier * suffix -> index, go back to 1 key per agent * rm relayer user * Create user and key identifiers * Move some fns to key-utils, fix gcp secret names * Correct kathy & checkpointer key names * Back to Promse.all in deploy-agents, clean * Don't require GCP keys to be created with agentconfig * fix * pure json response for get-key-addresses Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Trevor Porter	c7fa1272b4	Dev environment & updated infrastructure (#352 ) * First starting at adding some correct configs and messing with helm charts * Created local test docker container here * Adding back infrastructure for test * Add checkpointer to Dockerfile, a bunch of helm chart changes * use docker buildkit and correct image name in build.sh * Checkpointer deploy works * Create dev keys, add support for multiple validators when creating keys * Deploy dev * Add dev config * Deploy is working * Stepping toward a better config... * prettier * Use ChainAgentConfig * Use stricter typing, thank you Yorke * Add some trailing newlines where needed * S3 validators work * Almost entirely working * fix test env, bunch of cleanup * prettier * Clean * Add aws signers for validator statefulset * fix package-locks * Better values.yaml, rely on validators configmap * prettier * Make kathy key creation not conditional * Different local storage paths for test checkpoint syncers * Rename replicaChain * Make relayer types all mandatory * Fix output-agent-env-vars.ts * clean up * Try build cache * setup buildkit * Try GHA caching	3 years ago
Trevor Porter	fec07d21fe	Remove docs & all optics-related rust (#322 ) * optics -> abacus in infra * rm docs * rename contract-metrics prom metrics optics-* -> abacus-* * Rm watcher, updater, a bunch of optics things in abacus-base * rm agents/processor * Removing a whole lot of updater / home / replica stuff from abacus-core * Deleted a lot throughout -- some still commented out and a lot of cleanup needed * Slowly cleaning up, kathy is where the remaining errors are * Delete some commented out code * makes kathy an abacus agent * compiling woo * cargo fmt * compiling with all features * removing some update references * Ignore handles_missing_rpc_checkpoints * cargo fmt * Add signedCheckpoint.json * rm some references to updates, delete unnecessary abis * rm all references to home or replica * compiling, remove som e unused code * fix test warning * nit * New README - needs a better pass, but no optics references for now * mv rust/helm/optics-agent rust/helm/abacus-agent * Note on us not indexing updates * Get checkpointer working from local env var output	3 years ago
Nam Chu Hoai	23c3e4cd22	Add lookback for updates/messages in contract_sync (#144 ) * Apply contract_sync mod patch * Add IndexDataTypes * Fmt * Do not store updates/messages when encountering an invalid continuation If we store them we may end up invariably overwriting the latest_leaf_index database entry which keeps track of "contigousness". I.e. this would cause holes of size >1 to cause the agents being unable to recover * Index both messages and updates on the processor * remove comment * Followups for the contract-sync patch (#146) * Readd abacus_last_known_message_leaf_index to message indexing in the processor * Remove tipbuffer functionality * Additional contract_sync tests (#147) * Fix test case for when batch includes the already known updates/messages * Modify tests to include multiple ranges of missing updates/events	3 years ago
Trevor Porter	46bd6793c9	Move agents to external-secrets, no longer requiring a .env file (#145 ) * Use external-secrets for agents, working on dev, need to handle aws credentials and possibly others * Get AWS support working * Move toward not requiring AWS keys in a local .env * Use us-west-2 as AWS regions in config rather than requiring env vars * Pretty annoying refactor to asynchronously get some secrets from GCP that were previously env vars * Everything seems to be working * Don't pass in any secrets as helm values * Self nit * Fix * Prettier * Rm some unused dotenv	3 years ago
Nam Chu Hoai	767116f57b	Default to prometheus podAnnotations (#140 )	3 years ago
Nam Chu Hoai	e3c1b3bdcc	Support tip buffers for contract_sync (#138 ) * Support tip buffers for contract_sync * Pass IndexSettings * Add tipBuffer to deploy tooling * Run prettier * Deploy on dev	3 years ago
Nam Chu Hoai	bf72880763	Final changes for abacus agent deploy (#137 ) * Final changes for abacus agent deploy * Prettier	3 years ago
Nam Chu Hoai	d4c8720700	Rename rust packages (#128 ) * Rename rust packages * rustfmt * More rename * Rename update_abis * Rename metrics.rs	3 years ago
Nam Chu Hoai	47ead90aac	Modify deployment tooling for manual processing changes (#16 ) * Modify deployment tooling for manual processing changes * Add docker reference	3 years ago
Nam Chu Hoai	f5f1addc8f	Add Agent deployment tooling (#3 ) * Add staging-community environment * Agent deploy tooling * Finalize agent tooling * dev deploy * Add dev tooling * Dev deployment * Add env var output script * changes * PR comments * Fix missed replacements * PR comments	3 years ago
Conner Swann	440493c1e3	added metrics config to helm chart (#770 )	3 years ago
Conner Swann	f6f9404cb6	fixed podSecurityContext (#675 )	3 years ago
Conner Swann	4ed85d69f3	[Feature] Added docker workflow for image builds (#632 ) * added docker workflow, images produced on tag and commit * added back optics/deployment label to helm chart	3 years ago
Alex Crowe	ad363c7507	Helm Chart Update (#595 ) * Updated helm chart with additions for * Updated to use dedicated resources and pod lables/annotations * Added ConfigMap for common envs * Updated storage configuration to be more flexible * Added missing envs for db path, backtrace and loggings * Updated with conditional for tracer name to avoid startup failure * Updated AWS ENVs to correct names	3 years ago
Conner Swann	93b3f22201	[Feature] added AWS signers as option to Helm (#567 ) * added AWS signers as option * updated README * updated dockerfile	3 years ago
Conner Swann	9d2771621f	[Feature] Helm Chart for Agent Deployment (#394 ) * Helm implementation and build tooling	3 years ago

17 Commits (b379336b9187413211650a3d836b3ca48abd59f1)