Keda Pub/Sub Scaler - Austin Poole

TLDR;
Assumptions
Getting Started
- Configuring Keda
Full Example
TroubleShooting

Keda Pub/Sub Scaler was an unnecessary challenge I had to face over the course of 3 days. If you were to cross reference these 3 sources:

https://cloud.google.com/kubernetes-engine/docs/tutorials/scale-to-zero-using-keda#setup-env
https://keda.sh/docs/2.10/scalers/gcp-pub-sub/
https://keda.sh/docs/2.14/authentication-providers/gcp-workload-identity/

You can come to a reasonable idea of what you need to do. As long as you read it thoroughly…

Or you can see a working example here 😀

TLDR;

Go to the full example

Assumptions

Workload Identity is turned on for you cluster
Your node pool has “GKE Metadata Server” enabled
Your GCP user has the permissions to create a workload identity for a Kubernetes Service Account
You’re using helm to install keda

Getting Started

To get Keda working you first need to get Custom Metrics Stackdriver Adapter working.

Please see my article on GCP Horizontal Pod Autoscaling with Pub/Sub to learn how to set that up.

Configuring Keda

After getting the custom metrics stackdriver adapter working now its time to install keda

helm install --repo  https://kedacore.github.io/charts --version 2.16.0 keda keda -n keda

Create a bash script to add a policy binding to the keda-operator KSA and call the script add-policy.sh.

PROJECT_ID=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_ID)")
PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
KEDA_NAMESPACE=keda

gcloud projects add-iam-policy-binding projects/$PROJECT_ID \
      --role=roles/monitoring.viewer \
      --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$KEDA_NAMESPACE/sa/keda-operator \
      --condition=None
  echo "Added workload identity to keda-operator"

That is all you need to do to enable Keda Pub/Sub Scaler. Continue on if you want a full example

Full Example

This script will

install keda via helm
add policy binding for the keda-operator and pubsub-sa KSA’s
create a topic/subscription
Deploys an app that reads from the pub/sub subscription
Creates Keda TriggerAuthentication and ScaledObject objects

PROJECT_ID=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_ID)")
SERVICE_ACCOUNT_NAME=custom-metrics-stackdriver
PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
KEDA_NAMESPACE=keda
APP_NAMESPACE=default
PUBSUB_TOPIC=echo
PUBSUB_SUBSCRIPTION=echo-read
YAML_FILE_NAME=test-app.yaml

create(){
  helm install keda kedacore/keda -n keda

  sleep 3

  gcloud projects add-iam-policy-binding projects/$PROJECT_ID \
      --role=roles/monitoring.viewer \
      --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$KEDA_NAMESPACE/sa/keda-operator \
      --condition=None
  echo "Added workload identity to keda-operator"

  gcloud pubsub topics create $PUBSUB_TOPIC
  sleep 5
  echo "Created $PUBSUB_TOPIC Topic"

  gcloud pubsub subscriptions create $PUBSUB_SUBSCRIPTION --topic=$PUBSUB_TOPIC
  echo "Created Subscription $PUBSUB_SUBSCRIPTION to Topic $PUBSUB_TOPIC"

  gcloud projects add-iam-policy-binding projects/$PROJECT_ID \
      --role=roles/pubsub.subscriber \
      --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$APP_NAMESPACE/sa/pubsub-sa
    echo "Added workload identity to to pubsub-sa"

cat > $YAML_FILE_NAME << EOL
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pubsub-sa
---
# [START gke_deployment_pubsub_with_workflow_identity_deployment_pubsub]
# [START container_pubsub_workload_identity_deployment]
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pubsub
spec:
  selector:
    matchLabels:
      app: pubsub
  template:
    metadata:
      labels:
        app: pubsub
    spec:
      serviceAccountName: pubsub-sa
      containers:
        - name: subscriber
          image: us-docker.pkg.dev/google-samples/containers/gke/pubsub-sample:v2
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-gcp-credentials
spec:
  podIdentity:
    provider: gcp
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: pubsub-scaledobject
spec:
  scaleTargetRef:
    name: pubsub #Deployment
  minReplicaCount: 1
  maxReplicaCount: 2
  triggers:
    - type: gcp-pubsub
      authenticationRef:
        name: keda-trigger-auth-gcp-credentials
      metadata:
        subscriptionName: "echo-read" # Required
        value: "5"
        activationValue: "5"
#        credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON
# [END container_pubsub_workload_identity_deployment]
# [END gke_deployment_pubsub_with_workflow_identity_deployment_pubsub]

EOL

  kubectl apply -f $YAML_FILE_NAME -n $APP_NAMESPACE
  echo "Deployed test application"
}


delete(){
  kubectl delete -f $YAML_FILE_NAME -n $APP_NAMESPACE
  gcloud projects remove-iam-policy-binding projects/$PROJECT_ID \
        --role=roles/pubsub.subscriber \
        --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$APP_NAMESPACE/sa/pubsub-sa \
        --condition=None
  gcloud pubsub subscriptions delete $PUBSUB_SUBSCRIPTION
  gcloud pubsub topics delete $PUBSUB_TOPIC
  gcloud projects remove-iam-policy-binding projects/$PROJECT_ID \
        --role=roles/monitoring.viewer \
        --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$KEDA_NAMESPACE/sa/keda-operator \
        --condition=None
  helm uninstall keda -n keda
}

create

In another window send messages to the topic

$ for i in {1..200}; do gcloud pubsub topics publish echo --message="Autoscaling #${i}"; done

Its going to take a few minutes for the scaling to occur.

Watch as the pod count goes up. Eventually you’ll see the targets start to go up

$ watch kubectl get hpa -n default

NAME                           REFERENCE           TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-pubsub-scaledobject   Deployment/pubsub   2/5 (avg)   1         2         2          10m

TroubleShooting

You see the dreaded <unkown>/5 error.
- This can happen for a variety of reasons. Its best to check the output of all the commands and make sure they all work. If any of them failed then the hpa setup will fail.

Table of Contents