The home for Hyperlane core contracts, sdk packages, and other infrastructure
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
hyperlane-monorepo/rust/helm/hyperlane-agent/templates/scraper-statefulset.yaml

88 lines
2.9 KiB

{{- if .Values.hyperlane.scraper.enabled }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
name: {{ include "agent-common.fullname" . }}-scraper3
labels:
{{- include "agent-common.labels" . | nindent 4 }}
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
app.kubernetes.io/component: scraper3
spec:
selector:
matchLabels:
{{- include "agent-common.selectorLabels" . | nindent 6 }}
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
app.kubernetes.io/component: scraper3
replicas: 1
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
serviceName: {{ include "agent-common.fullname" . }}-scraper3
template:
metadata:
annotations:
checksum/configmap: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.hyperlane.scraper.podAnnotations }}
{{ toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "agent-common.labels" . | nindent 8 }}
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
app.kubernetes.io/component: scraper3
{{- with .Values.podCommonLabels }}
{{ toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.hyperlane.scraper.podLabels }}
{{ toYaml . | nindent 8 }}
{{- end }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
terminationGracePeriodSeconds: 10
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: agent
securityContext:
{{- toYaml .Values.securityContext | nindent 10 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["./scraper"]
envFrom:
- configMapRef:
name: {{ include "agent-common.fullname" . }}
- secretRef:
name: {{ include "agent-common.fullname" . }}-secret
- secretRef:
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
name: {{ include "agent-common.fullname" . }}-scraper3-secret
env:
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
{{- $scraperChainNames := list }}
{{- range .Values.hyperlane.chains }}
{{- if not .disabled }}
{{- $scraperChainNames = append $scraperChainNames .name }}
{{- end }}
{{- end }}
- name: HYP_SCRAPER_CHAINSTOSCRAPE
value: {{ $scraperChainNames | join "," }}
{{- if .Values.hyperlane.tracing.uri }}
- name: HYP_BASE_TRACING_JAEGER_NAME
Scraper Refresh (#1932) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.
2 years ago
value: {{ include "agent-common.fullname" . }}-scraper3
{{- end }}
resources:
{{- toYaml .Values.hyperlane.scraper.resources | nindent 10 }}
ports:
- name: metrics
containerPort: {{ .Values.hyperlane.metrics.port }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}