Helm & Kubernetes

This guide explains how to deploy Rafiki using Helm charts on a Kubernetes cluster. Helm is a package manager for Kubernetes that allows you to define, install, and upgrade complex Kubernetes applications through Helm charts.

Rafiki uses the following key infrastructure components:

PostgreSQL: Used for storing application data, grants, and Open Payments resources
Redis: Used for caching and session management
TigerBeetle (recommended): High-performance accounting database for financial transaction processing and ledger management

Prerequisites

Before you begin, ensure you have the following:

Kubernetes cluster deployed
kubectl installed and configured to access your cluster
Helm installed

Install Rafiki using Helm

Add the Interledger Helm repository

Add the official Interledger Helm repository which contains the Rafiki charts:

helm repo add interledger https://interledger.github.io/charts/interledger
helm repo update

Verify available charts

You can list the available Rafiki charts:

helm search repo interledger/rafiki

This will show you the available chart versions for the different Rafiki components:

interledger/rafiki-auth
interledger/rafiki-backend
interledger/rafiki-frontend

Create values files

Create separate values files for each Rafiki service to customize your deployment. You can start with minimal configurations and add more settings as needed.

Basic auth service configuration

Create auth-values.yaml:

# Minimal auth service configuration
nodeEnv: production
logLevel: info

# Database configuration (update with your credentials)
databaseUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: DATABASE_URL

# Redis configuration
redisUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: REDIS_URL

# Identity server configuration
identityServer:
  domain: 'https://your-domain.com/idp'
  serverSecret:
    secretKeyRef:
      name: rafiki-secrets
      key: IDENTITY_SERVER_SECRET

# Cookie signing key
cookieKey:
  secretKeyRef:
    name: rafiki-secrets
    key: COOKIE_KEY

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3006
        targetPort: 3006
        protocol: TCP
        name: auth
      - port: 3003
        targetPort: 3003
        protocol: TCP
        name: admin
      - port: 3007
        targetPort: 3007
        protocol: TCP
        name: introspection
      - port: 3009
        targetPort: 3009
        protocol: TCP
        name: interaction

Basic backend service configuration

Create backend-values.yaml:

# Minimal backend service configuration
nodeEnv: production
logLevel: info
instanceName: 'your-rafiki-instance'

# Database configuration
databaseUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: DATABASE_URL

# Redis configuration
redisUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: REDIS_URL

# Auth service URLs
auth:
  grantUrl: http://rafiki-auth-service.default:3006
  introspectionUrl: http://rafiki-auth-service.default:3007

# ILP configuration
ilp:
  host: 'https://your-domain.com'
  address: test.your-instance
  connector: 'http://rafiki-backend-service.default:3002'
  streamSecret:
    secretKeyRef:
      name: rafiki-secrets
      key: STREAM_SECRET

# Webhook configuration
webhook:
  url: https://your-wallet.com/webhooks/rafiki

# Key configuration
key:
  id: your-unique-key-id
  pvk: '' # Will auto-generate if empty

# Webhook signature secret
webhookSignatureSecret:
  secretKeyRef:
    name: rafiki-secrets
    key: SIGNATURE_SECRET

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3000
        targetPort: 3000
        protocol: TCP
        name: open-payments
      - port: 3001
        targetPort: 3001
        protocol: TCP
        name: admin
      - port: 3002
        targetPort: 3002
        protocol: TCP
        name: connector

Basic frontend service configuration

Create frontend-values.yaml:

# Minimal frontend service configuration
nodeEnv: production
logLevel: info

# Service URLs
serviceUrls:
  GRAPHQL_URL: http://rafiki-backend-service.default:3001/graphql
  OPEN_PAYMENTS_URL: https://your-domain.com/

# Port configuration
port: 3010

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3010
        targetPort: 3010
        protocol: TCP
        name: http

# Kratos configuration (if using authentication)
kratos:
  enabled: false

Create Kubernetes secrets

Before installing the services, create the required secrets:

kubectl create secret generic rafiki-secrets \
  --from-literal=DATABASE_URL="postgresql://username:password@rafiki-postgresql:5432/rafiki" \
  --from-literal=REDIS_URL="redis://rafiki-redis:6379" \
  --from-literal=IDENTITY_SERVER_SECRET="your-identity-server-secret" \
  --from-literal=COOKIE_KEY="your-cookie-signing-key" \
  --from-literal=STREAM_SECRET="your-stream-secret" \
  --from-literal=SIGNATURE_SECRET="your-webhook-signature-secret"

Install the services

Install each Rafiki service using the appropriate values file:

# Install auth service
helm install rafiki-auth interledger/rafiki-auth -f auth-values.yaml

# Install backend service
helm install rafiki-backend interledger/rafiki-backend -f backend-values.yaml

# Install frontend service
helm install rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml

If you want to install to a specific namespace:

kubectl create namespace rafiki

helm install rafiki-auth interledger/rafiki-auth -f auth-values.yaml -n rafiki
helm install rafiki-backend interledger/rafiki-backend -f backend-values.yaml -n rafiki
helm install rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml -n rafiki

Verify the deployment

Check the status of your deployments:

# Check Helm releases
helm list

# Check running pods
kubectl get pods

# Check deployed services
kubectl get services

# Check logs for any issues
kubectl logs -l app.kubernetes.io/name=rafiki-auth
kubectl logs -l app.kubernetes.io/name=rafiki-backend
kubectl logs -l app.kubernetes.io/name=rafiki-frontend

Configure environment variables

Each Rafiki service can be configured via environment variables through Helm values. Below are the complete environment variable configurations for each service:

Auth service

The Rafiki auth service is responsible for handling Open Payments authorization and grant management. It implements the Open Payments authorization server specification and manages access tokens, grants, and interactions. The auth service connects to a PostgreSQL database to store grant-related data and uses Redis for session management.

Ports exposed:

3003 (admin) - Auth Admin API for managing grants and access tokens
3006 (auth) - Open Payments authorization server endpoint
3007 (introspection) - Access token introspection endpoint
3009 (interaction) - User interaction endpoint for grant flows

Required values

Helm variable	Default	Description
`auth.databaseUrl`	undefined	The URL of the Postgres database storing your Open Payments grant data. Can be provided directly as a value or referenced from a secret.
`auth.redisUrl`	undefined	The connection URL for Redis. Can be provided directly as a value or referenced from a secret.
`auth.cookieKey`	`changeme`	The koa KeyGrip key that is used to sign cookies for an interaction session. Can be provided directly as a value or referenced from a secret.
`auth.identityServer.domain`	`http://rafiki-backend/idp`	The URL of your IdP’s server, used by the authorization server to inform an Open Payments client of where to redirect the end-user to start interactions.
`auth.identityServer.serverSecret`	`changeme`	A shared secret between the authorization server and the IdP server; the authorization server will use the secret to secure its IdP-related endpoints. When the IdP server sends requests to the authorization server, the IdP server must provide the secret via an `x-idp-secret` header. Can be provided directly as a value or referenced from a secret.

Conditionally required values

Helm value name	Default	Description
`auth.trustProxy`	`true`	Must be set to `true` when running Rafiki behind a proxy. When `true`, the `X-Forwarded-Proto` header is used to determine if connections are secure.

Optional values

Helm value name	Default	Description
`auth.replicaCount`	`1`	The number of pods to run across the cluster.
`auth.nodeSelector`	`{}`	NodeSelector can be used to specify the nodes on which the pods should run on your cluster.
`auth.serviceAccount.create`	`false`	If true, a ServiceAccount will be created with the specified name and annotations.
`auth.serviceAccount.annotations`	`{}`	Annotations to add to the ServiceAccount.
`auth.serviceAccount.name`	`""`	The name of the ServiceAccount to use.
`auth.secrets.create`	`false`	If true, the chart will create a secret with the name of the release. If false, you must create the secret manually.
`auth.service.name`	`service`	The name of the service.
`auth.service.type`	`ClusterIP`	The type of Kubernetes service.
`auth.service.annotations`	`{}`	Annotations to add to the service.
`auth.service.ports`	Multiple ports	Service port configuration for auth (3006), admin (3003), introspection (3007), and interaction (3009) endpoints.
`auth.configmap`	`files/config.yaml`	Defines the location of the template that will be used to generate the ConfigMap.
`auth.secretsGenFile`	`files/secret.yaml`	Defines the location of the template that will be used to generate the default secret.
`auth.containers.name`	`server`	The name of the main container.
`auth.containers.readinessProbe.httpGet.path`	`/healthz`	The path for the readiness probe HTTP check.
`auth.containers.readinessProbe.httpGet.port`	`3007`	The port for the readiness probe HTTP check.
`auth.containers.livenessProbe.httpGet.path`	`/healthz`	The path for the liveness probe HTTP check.
`auth.containers.livenessProbe.httpGet.port`	`3007`	The port for the liveness probe HTTP check.
`auth.containers.resources.requests.memory`	`128Mi`	The minimum amount of memory required by the container.
`auth.containers.resources.requests.cpu`	`200m`	The minimum amount of CPU required by the container.
`auth.containers.resources.limits.memory`	`512Mi`	The maximum amount of memory the container can use.
`auth.containers.resources.limits.cpu`	`400m`	The maximum amount of CPU the container can use.
`auth.containers.ports`	Multiple ports	Container port configuration for auth (3006), admin (3003), introspection (3007), and interaction (3009) endpoints.
`auth.nodeEnv`	`production`	The type of node environment: `development`, `test`, or `production`.
`auth.logLevel`	`info`	Pino log level
`auth.authServerUrl`	`http://rafiki-auth:3006`	The public endpoint for your Rafiki instance’s public Open Payments routes.
`auth.interaction.quote`	`false`	When `true`, quote grants are interactive. Note that this is a string value.
`auth.interaction.cookieSameSite`	`lax`	The SameSite attribute for interaction cookies.
`auth.interaction.incomingPayment`	`false`	When `true`, incoming Open Payments grant requests are interactive. Note that this is a string value.
`auth.port.admin`	`3003`	The port of your Rafiki Auth Admin API server.
`auth.port.auth`	`3006`	The port of your Open Payments authorization server.
`auth.port.introspection`	`3007`	The port of your Open Payments access token introspection server.
`auth.grant.waitSeconds`	`5`	The wait time, in seconds, included in a grant request response (`grant.continue`).
`auth.accessToken.deletionDays`	`30`	The days until expired and/or revoked access tokens are deleted.
`auth.accessToken.expirySeconds`	`600` (10 minutes)	The expiry time, in seconds, for access tokens.
`auth.workers.cleanup`	`1`	The number of workers processing expired or revoked access tokens.
`auth.ingress.enabled`	`false`	When `true`, an Ingress resource is created to expose the application.
`auth.ingress.className`	`nginx`	The Ingress class name to use.
`auth.ingress.tls`	TLS configuration	TLS configuration for the Ingress, including secret name and hosts.
`auth.ingress.hosts`	Host configuration	Host configuration for the Ingress, including host, paths, and service details.

Backend service

The Rafiki backend service handles the core Open Payments functionality and business logic. It exposes the Open Payments resource server APIs for wallet addresses, incoming payments, outgoing payments, and quotes. The backend also includes an ILP connector for sending and receiving Interledger packets. It connects to PostgreSQL for storing Open Payments resources, Redis for caching, and optionally TigerBeetle for high-performance accounting.

Ports exposed:

3000 (openPayments) - Open Payments resource server API
3001 (admin) - Backend Admin API and GraphQL endpoint
3002 (connector) - ILP connector for sending and receiving packets
3005 (autoPeering) - Auto-peering service (when enabled)

Required values

Helm value name	Default	Description
`backend.auth.grantUrl`	`http://rafiki-auth.rafiki-auth:3006`	The endpoint on your Open Payments authorization server to grant a request.
`backend.auth.introspectionUrl`	`http://rafiki-auth.rafiki-auth:3007`	The endpoint on your Open Payments authorization server to introspect an access token.
`backend.databaseUrl`	undefined	The Postgres database URL of the database storing your resource data. Can be provided directly as a value or referenced from a secret.
`backend.redisUrl`	undefined	The Redis URL of the database handling ILP packet data. Can be provided directly as a value or referenced from a secret.
`backend.ilp.host`	`http://rafiki-backend:3000`	The public endpoint of your Open Payments resource server.
`backend.ilp.address`	`test.rafiki-backend`	The ILP address of your Rafiki instance.
`backend.ilp.connector`	`http://rafiki-backend:3002`	The ILP connector address where ILP packets are received.
`backend.ilp.streamSecret`	`changeme`	The seed secret to generate shared STREAM secrets. Can be provided directly as a value or referenced from a secret.
`backend.key.id`	`rafiki-override-this-value`	Your Rafiki instance’s client key ID.
`backend.key.pvk`	`''`	The private key pem file used. Must be provided as base64 encoded version of the pem file. If empty, a job will generate a new key pair.
`backend.webhook.url`	`http://wallet/webhooks/rafiki`	Your endpoint that consumes webhook events.
`backend.webhookSignatureSecret`	`changeme`	The secret to generate request header signatures for webhook event requests. Can be provided directly as a value or referenced from a secret.

Conditionally required values

Helm value name	Default	Description
`backend.instanceName`	`rafiki-backend-changeme`	Your Rafiki instance’s name used to communicate for auto-peering and/or telemetry. Required when auto-peering and/or telemetry is enabled.
`backend.trustProxy`	`true`	Must be set to `true` when running Rafiki behind a proxy. When `true`, the `X-Forwarded-Proto` header is used to determine if connections are secure.

Optional values

Helm value name	Default	Description
`backend.nodeEnv`	`production`	The type of node environment: `development`, `test`, or `production`.
`backend.logLevel`	`info`	Pino log level
`backend.useTigerbeetle`	`false`	When `true`, a TigerBeetle database is used for accounting. When `false`, a Postgres database is used.
`backend.secrets.create`	`false`	If true, the chart will create a secret with the name of the release. If false, you must create the secret manually.
`backend.rates.url`	`""`	The endpoint your Rafiki instance uses to request exchange rates.
`backend.webhook.timeout`	`200`	The time, in milliseconds, that WALLET_ADDRESS_WORKERS wait until checking the empty wallet address request queue again.
`backend.walletAddressRedirectHtmlPage`	`''`	This feature is experimental and may change in future versions.
`backend.slippage`	`0.01` (1%)	The accepted ILP rate fluctuation.
`backend.withdrawalThrottleDelay`	`''`	The delay in liquidity withdrawal processing. Unit ms.
`backend.lifetime.exchangeRate`	`15000`	The time, in milliseconds, the exchange rates you provide via the EXCHANGE_RATES_URL are valid.
`backend.lifetime.quote`	`300000` (5 minutes)	The time, in milliseconds, an Open Payments quote is valid for.
`backend.lifetime.webhook`	`200`	The time, in milliseconds, that your Rafiki instance will wait for a 200 response from your webhook endpoint. If a 200 response is not received, Rafiki will time out and try to send the webhook event again.
`backend.workers.incomingPayment`	`1`	The number of workers processing incoming payment requests.
`backend.workers.outgoingPayment`	`1`	The number of workers processing outgoing payment requests.
`backend.workers.paymentPointer`	`1`	The number of workers processing payment pointer requests.
`backend.workers.webhook`	`1`	The number of workers processing webhook events.
`backend.workerIdle`	`200`	The time, in milliseconds, that workers will wait until checking an empty queue again.
`backend.idempotency.keyTTL`	`86400000` (24 hours)	The TTL, in milliseconds, for idempotencyKey on GraphQL mutations on the Backend Admin API.
`backend.idempotency.keyLock`	`2000`	The TTL, in milliseconds, for idempotencyKey concurrency lock on GraphQL mutations on the Backend Admin API.
`backend.telemetry.enabled`	`false`	Enables the telemetry service on Rafiki.
`backend.telemetry.livenet`	`false`	When true, telemetry is enabled for livenet.
`backend.autoPeering.enabled`	`false`	Enables the auto-peering service on Rafiki.
`backend.port.admin`	`3001`	The port of your Backend Auth API server.
`backend.port.connector`	`3002`	The port of the ILP connector for sending packets via ILP over HTTP.
`backend.port.openPayments`	`3000`	The port of your Open Payments resource server.
`backend.port.autoPeering`	`3005`	If auto-peering is enabled, the server will use this port.
`backend.replicaCount`	`1`	Currently rafiki should run with a single replica, so this value is set to 1.
`backend.nodeSelector`	`{}`	Can be used to ensure rafiki-backend is deployed on a specific node or set of nodes.
`backend.serviceAccount.create`	`false`	If true, a ServiceAccount will be created with the specified name and annotations.
`backend.serviceAccount.annotations`	`{}`	Annotations to add to the ServiceAccount.
`backend.serviceAccount.name`	`""`	The name of the ServiceAccount to use.
`backend.service.name`	`service`	The name of the service.
`backend.service.annotations`	`{}`	Annotations to add to the service.
`backend.service.type`	`ClusterIP`	The type of Kubernetes service.
`backend.service.ports`	Multiple ports	Service port configuration for open-payments (3000), admin (3001), connector (3002), and auto-peering (3005) endpoints.
`backend.configmap`	`files/config.yaml`	The template for generating the default env configmap.
`backend.secretsGenFile`	`files/secret.yaml`	The template for generating the env secrets.
`backend.containers.name`	`server`	The name of the main container.
`backend.containers.resources.requests.memory`	`256Mi`	The minimum amount of memory required by the container.
`backend.containers.resources.requests.cpu`	`1`	The minimum amount of CPU required by the container.
`backend.containers.resources.limits.memory`	`2048Mi`	The maximum amount of memory the container can use.
`backend.containers.resources.limits.cpu`	`4`	The maximum amount of CPU the container can use.
`backend.containers.readinessProbe.httpGet.path`	`/healthz`	The path for the readiness probe HTTP check.
`backend.containers.readinessProbe.httpGet.port`	`3000`	The port for the readiness probe HTTP check.
`backend.containers.livenessProbe.httpGet.path`	`/healthz`	The path for the liveness probe HTTP check.
`backend.containers.livenessProbe.httpGet.port`	`3000`	The port for the liveness probe HTTP check.
`backend.containers.ports`	Multiple ports	Container port configuration for open-payments (3000), admin (3001), connector (3002), and auto-peering (3005) endpoints.
`backend.containers.volumeMounts`	Volume mount configuration	Mounts the keys volume at /mnt/keys as read-only.

Frontend service

The Rafiki frontend service provides an administrative web interface for managing your Rafiki instance. It offers a user-friendly dashboard for monitoring and managing wallet addresses, payments, peers, and other Open Payments resources. The frontend communicates with the Backend Admin API to perform administrative operations.

Ports exposed:

3010 (http) - Rafiki Admin web interface

Required values

Helm value name	Default	Description
`frontend.serviceUrls.GRAPHQL_URL`	undefined	URL for Rafiki’s GraphQL Auth Admin API
`frontend.serviceUrls.OPEN_PAYMENTS_URL`	undefined	Your Open Payments API endpoint
`frontend.port`	undefined	Port from which to host the Rafiki Remix app

Conditionally required values

The following values are required only when frontend.kratos.enabled is set to true.

Helm value name	Default	Description
`frontend.kratos.adminUrl`	undefined	The admin endpoint/container address for Kratos
`frontend.kratos.containerPublicUrl`	undefined	The URL for you to access the Kratos Docker container from within the Docker network. This is used for backend calls to Kratos.
`frontend.kratos.browserPublicUrl`	undefined	The URL for you to access the Kratos Docker container from a browser outside of the Docker network. This is used for calls from a browser (what you see in the Rafiki Admin UI) to the Kratos server on the backend.

Optional values

Helm value name	Default	Description
`frontend.nodeEnv`	`production`	The type of node environment: `development`, `test`, or `production`.
`frontend.logLevel`	`debug`	Pino log level
`frontend.replicaCount`	`1`	The number of pods to run across the cluster.
`frontend.nodeSelector`	`{}`	NodeSelector can be used to specify the nodes on which the pods should run on your cluster.
`frontend.serviceAccount.create`	`false`	If true, a ServiceAccount will be created with the specified name and annotations.
`frontend.serviceAccount.annotations`	`{}`	Annotations to add to the ServiceAccount.
`frontend.serviceAccount.name`	`""`	The name of the ServiceAccount to use.
`frontend.service.name`	`service`	The name of the service.
`frontend.service.type`	`ClusterIP`	The type of Kubernetes service.
`frontend.service.ports.port`	`3010`	The port exposed by the service.
`frontend.service.ports.targetPort`	`3010`	The port on the container that the service should forward traffic to.
`frontend.service.ports.protocol`	`TCP`	The protocol used by the port.
`frontend.service.ports.name`	`http`	The name of the port.
`frontend.configmap`	`files/config.yaml`	Defines the location of the template that will be used to generate the ConfigMap.
`frontend.containers.name`	`server`	The name of the main container.
`frontend.containers.envFrom`	ConfigMap reference	Environment variables sourced from the ConfigMap with the release name.
`frontend.containers.resources.requests.memory`	`128Mi`	The minimum amount of memory required by the container.
`frontend.containers.resources.requests.cpu`	`100m`	The minimum amount of CPU required by the container.
`frontend.containers.resources.limits.memory`	`512Mi`	The maximum amount of memory the container can use.
`frontend.containers.resources.limits.cpu`	`1`	The maximum amount of CPU the container can use.
`frontend.containers.ports.name`	`http`	The name of the container port.
`frontend.containers.ports.containerPort`	`3010`	The port number on the container.
`frontend.containers.ports.protocol`	`TCP`	The protocol used by the container port.
`frontend.rollingUpdate.maxSurge`	`100%`	The maximum number of pods that can be created above the desired number during a rolling update.
`frontend.rollingUpdate.maxUnavailable`	`50%`	The maximum number of pods that can be unavailable during a rolling update.
`frontend.kratos.enabled`	`false`	When `true`, Kratos authentication is enabled for the frontend.

Configure ingress

To expose Rafiki services outside the cluster, you’ll need to configure ingress. Here’s an example using NGINX Ingress Controller:

Install NGINX Ingress Controller

If you don’t already have an ingress controller installed:

# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Install the ingress controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
  --set controller.publishService.enabled=true

Configure ingress in your values files

Add ingress configuration to your service values files:

# In backend-values.yaml
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: rafiki-backend.your-domain.com
      paths:
        - path: /
          pathType: Prefix
          service:
            name: rafiki-backend-service
            port: 3000
  tls:
    - secretName: rafiki-backend-tls
      hosts:
        - rafiki-backend.your-domain.com

# In auth-values.yaml
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: rafiki-auth.your-domain.com
      paths:
        - path: /
          pathType: Prefix
          service:
            name: rafiki-auth-service
            port: 3006
  tls:
    - secretName: rafiki-auth-tls
      hosts:
        - rafiki-auth.your-domain.com

Configure DNS

Create DNS records pointing to your ingress controller’s external IP:

rafiki-backend.your-domain.com
rafiki-auth.your-domain.com
rafiki-frontend.your-domain.com

Configure dependencies

PostgreSQL

You can use an external PostgreSQL instance or deploy one using Helm:

# Add Bitnami repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install PostgreSQL
helm install rafiki-postgresql bitnami/postgresql \
  --set auth.postgresPassword=your-secure-password \
  --set auth.database=rafiki \
  --set primary.persistence.size=20Gi

Redis

Deploy Redis for caching and session management:

# Install Redis
helm install rafiki-redis bitnami/redis \
  --set auth.enabled=false \
  --set master.persistence.size=8Gi

TigerBeetle (recommended)

For high-performance accounting, deploy TigerBeetle:

# Add TigerBeetle repository
helm repo add tigerbeetle https://tigerbeetle.github.io/helm-charts

# Install TigerBeetle
helm install rafiki-tigerbeetle tigerbeetle/tigerbeetle \
  --set replicaCount=3 \
  --set persistence.size=20Gi

Upgrade Rafiki

To upgrade your Rafiki services to newer versions:

# Update the Helm repository
helm repo update

# Upgrade each service
helm upgrade rafiki-auth interledger/rafiki-auth -f auth-values.yaml
helm upgrade rafiki-backend interledger/rafiki-backend -f backend-values.yaml
helm upgrade rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml

Uninstall Rafiki

To uninstall Rafiki services:

helm uninstall rafiki-auth
helm uninstall rafiki-backend
helm uninstall rafiki-frontend

# Optionally uninstall dependencies
helm uninstall rafiki-postgresql
helm uninstall rafiki-redis
helm uninstall rafiki-tigerbeetle

Uninstalling will not delete Persistent Volume Claims (PVCs) by default. To delete all data:

kubectl delete pvc -l app.kubernetes.io/instance=rafiki-postgresql
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-redis
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-tigerbeetle

Troubleshooting

Check service health

Verify that all services are running correctly:

# Check pod status
kubectl get pods -l app.kubernetes.io/name=rafiki-auth
kubectl get pods -l app.kubernetes.io/name=rafiki-backend
kubectl get pods -l app.kubernetes.io/name=rafiki-frontend

# Check service endpoints
kubectl get endpoints

# Check for events
kubectl get events --sort-by=.metadata.creationTimestamp

Common issues and solutions

Database connection errors

Verify database connectivity:

kubectl logs -l app.kubernetes.io/name=rafiki-backend | grep -i database

Check PostgreSQL status:

kubectl get pods -l app.kubernetes.io/name=postgresql
kubectl logs -l app.kubernetes.io/name=postgresql

Test database connection:

kubectl run -it --rm debug --image=postgres:14 -- psql postgresql://username:password@rafiki-postgresql:5432/rafiki

Redis connection issues

Check Redis status:

kubectl get pods -l app.kubernetes.io/name=redis
kubectl logs -l app.kubernetes.io/name=redis

Test Redis connectivity:

kubectl run -it --rm debug --image=redis:7 -- redis-cli -h rafiki-redis ping

Service discovery problems

Verify service names and ports:

kubectl get services
kubectl describe service rafiki-backend-service

Check DNS resolution:

kubectl run -it --rm debug --image=busybox -- nslookup rafiki-backend-service

Authentication and authorization issues

Check auth service logs:

kubectl logs -l app.kubernetes.io/name=rafiki-auth

Verify secrets are properly mounted:

kubectl describe pod -l app.kubernetes.io/name=rafiki-auth

Test auth endpoints:

kubectl port-forward svc/rafiki-auth-service 3007:3007
curl http://localhost:3007/healthz

Ingress and networking issues

Check ingress controller status:

kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller

Verify ingress resources:

kubectl get ingress
kubectl describe ingress rafiki-backend-ingress

Check TLS certificates:

kubectl get secrets
kubectl describe secret rafiki-backend-tls

Resource and performance issues

Check resource usage:
Terminal window
```
kubectl top pods
kubectl top nodes
```

Monitor pod metrics:

kubectl describe pod -l app.kubernetes.io/name=rafiki-backend

Scale services if needed:

# Update replicaCount in values.yaml and upgrade
helm upgrade rafiki-backend interledger/rafiki-backend -f backend-values.yaml

Production considerations

Security

Use strong passwords: Replace all default passwords with cryptographically secure values
Enable TLS: Configure HTTPS for all external communications
Network policies: Implement Kubernetes network policies to restrict pod-to-pod communication
RBAC: Use Kubernetes Role-Based Access Control to limit cluster access
Secrets management: Consider using external secret management solutions like HashiCorp Vault
Image security: Use specific image tags and scan images for vulnerabilities
Pod security: Configure pod security standards and security contexts

High availability

Multi-replica deployments: Run multiple replicas of critical services
Pod disruption budgets: Configure disruption budgets to maintain availability during updates
Resource limits: Set appropriate CPU and memory limits
Health checks: Configure proper readiness and liveness probes
Load balancing: Use multiple ingress controller replicas

Monitoring and observability

Prometheus: Deploy Prometheus for metrics collection
Grafana: Set up Grafana dashboards for visualization
Logging: Implement centralized logging with ELK stack or similar
Alerting: Configure alerting rules for critical issues
Distributed tracing: Consider implementing distributed tracing for request flow analysis

Backup and disaster recovery

Database backups

PostgreSQL backup:

# Create a backup job
kubectl create job --from=cronjob/postgres-backup manual-backup-$(date +%Y%m%d-%H%M%S)

# Manual backup
kubectl exec -it rafiki-postgresql-0 -- pg_dump -U postgres rafiki > backup.sql

TigerBeetle backup:

# Create volume snapshots
kubectl create -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: tigerbeetle-snapshot-$(date +%Y%m%d-%H%M%S)
spec:
  source:
    persistentVolumeClaimName: tigerbeetle-data-0
  volumeSnapshotClassName: your-snapshot-class
EOF

Performance optimization

Resource allocation: Right-size CPU and memory requests/limits based on actual usage
Connection pooling: Configure appropriate database connection pool sizes
Caching: Optimize Redis configuration for your workload
TigerBeetle tuning: Configure TigerBeetle cluster size based on expected transaction volume

Upgrade strategies

Rolling updates: Use rolling update strategies to maintain availability
Blue-green deployments: Consider blue-green deployments for zero-downtime upgrades
Database migrations: Plan for database schema migrations
Backup before upgrade: Always backup data before major upgrades
Testing: Test upgrades in staging environments first