If you're running PHP-FPM in Kubernetes you have likely found that adding a Horizontal Pod Autoscaler (HPA hereonin) that is based on the CPU ultisation metric isn't reliable. This is because web traffic by nature is somewhat spikey, and scaling based on a sustained high CPU % is in reality too late to scale.

There is a solution however by using Kubernetes Custom Metrics API. Now, this blog post is entirely focused on Google Kubernetes Engine, and we're using Stackdriver as our custom metrics server. I'll explain the setup details in this post. If you're not running GKE there will be a few steps you'll need to swap out.

First up, how should php-fpm be configured?

The general advice is to ensure the process manager should be configured to static, then you're pm.max_children value should be relatively low as each pod/container shouldn't run too many processes. You'll need to balance this based on your kubernetes container request/limit values for CPU and Memory.

Additionally we enable the php-fpm status page. This is important, as later on we will scrape the metrics from the status page!

Here's a sample of the configuration I use:

pm = static
pm.status_path = /status
pm.max_children = 10
; A child process will handle at least 200 requests before respawning.
; useful for memory leaks in 3rd party code.
pm.max_requests = 200

Then the kubernetes deployment configuration:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: phpfpm-demo
  labels:
    app: phpfpm-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: phpfpm-demo
  template:
    metadata:
      labels:
        app: phpfpm-demo
    spec:
      containers:
      - name: fpm
        image:  playsportsgroup/php:7.3-fpm-alpine-root
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
        resources:
          requests:
            cpu: 1
            memory: 1G
          limits:
            cpu: 1
            memory: 1G

Now, as it stands we have a deployment with a php-fpm container. Next up we'll need to begin scraping our metrics.

Scraping php-fpm status page metrics

The easiest way to do this is to use a Prometheous compatible exporter. I have personally used this one: https://github.com/hipages/php-fpm_exporter

We will need to add this container to our Deployment. So let's do that, our new deployment will look like this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: phpfpm-demo
  labels:
    app: phpfpm-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: phpfpm-demo
  template:
    metadata:
      labels:
        app: phpfpm-demo
    spec:
      containers:
      - name: fpm
        ... (omitted to keep it readible!)
      - name: phpfpm-exporter
        image: hipages/php-fpm_exporter
        ports:
          - name: exporter
            containerPort: 9253
        env:
          - name: PHP_FPM_SCRAPE_URI
            value: "tcp://localhost:9000/status"
          - name: PHP_FPM_FIX_PROCESS_COUNT
            value: "true"
        resources:
          requests:
            cpu: 10m
          limits:
            cpu: 10m

This works by scraping the status page over a TCP connection to the php-fpm container. The metrics will be exposed on the port 9253.

Now this isn't it. These metrics are currently not being sent anyway. The exporter has simply made them available! Next up we'll need to convert the Prometheous metrics into Stackdriver Metrics. This can be achieved with another container provided by Google.

... (continuing from the pevious)
      - name: prometheus-to-sd
        image: gcr.io/google-containers/prometheus-to-sd:v0.9.2
        ports:
          - name: profiler
            containerPort: 6060
        command:
          - /monitor
          - --stackdriver-prefix=custom.googleapis.com
          - --monitored-resource-type-prefix=k8s_
          - --source=:http://localhost:9253
          - --pod-id=$(POD_NAME)
          - --namespace-id=$(POD_NAMESPACE)
          - --cluster-location=[ADD_YOUR_CLUSTER_LOCATION_HERE]
        resources:
          requests:
            cpu: 10m
          limits:
            cpu: 10m
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

In the above example you'll need to define the cluster location yourself. I have found that within a Deployment this value cannot be automatically set. This is just the region that your cluster is in, so europe-west1 if you're using the Belgium region for example.

Let's take a look at the command arguments we are using here, as this deviates from GCP's documentation here slightly, again this is because we are using a Deployment here. GCP's guides only work when creating Pods directly.

--stackdriver-prefix=custom.googleapis.com is required to ensure the metric is tracked in stackdriver as a custom metric. Otherwise we won't be able to scale based on this!

--monitored-resource-type-prefix=k8s_ by default this appears to set the legacy prefix which is gke_. We don't want this as it doesn't work with Deployments!

--source=:http://localhost:9253 this specifies where the source of our metrics. The colon is defining the namespace, which in this case is empty. This is important because GKE doesn't support scaling on custom metrics that contain slashes, by adding a namespace it will add a slash. i.e. --source=mynamespace:http://localhost:9253 would create: custom.googleapis.com/mynamespace/some_metric, which is not desired!

--pod-id=$(POD_NAME) This applies the pod-id label in Stackdriver, allowing us to see the origin of metrics in relation to the pod.

--namespace-id=$(POD_NAMESPACE) Same as above, but for the kubernetes namespace your pod belongs to (note in my example deployment I have omitted the namespace which means it will default to default).

The last two arguments are using the environment variables defined on the container, these are coming from Kubernetes downward API. The Pod ID will be generated uniquely depending on the revision and number of replicas in your deployment, so we do not know this until runtime.

Ok. With that done we should now see metrics appearing in Stackdriver! Using the Metrics Exlorer you will be able to search for the metrics, they will all be prefixed with phpfpm_ - a full list can be found here: https://github.com/hipages/php-fpm_exporter

Configure cluster to access stackdriver metrics

Now, we have our metrics going to Stackdriver. But we won't be able to access these metrics inside Kubernetes to trigger our autoscaler just yet. We'll need to create the required authorisation roles to do this.

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin --user "$(gcloud config get-value account)"

Note: the above command will use your user account. I'd recommend using a service account instead that has the specific resources applied. Tying resources users is never a good approach.

With the authorisation role in place we can now deploy the Custom Metrics Stackdriver Adaptor to our cluster. This will make the stackdriver metrics available within your cluster.

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

Note: This deviates slightly from some of GCP's documentation again. They may tell you to apply a different file from the same github repo. That will be the legacy version which again, does not work with Deployments due to deployments using the k8s_ prefix for resources instead of the legacy gke_ (I lost a lot of hours to this...).

Create your autoscaler

Finally we're now ready to create our autoscaler. For this I'm going to use the phpfpm_active_processes metric. As we're using a statically defined number of processes we can trigger scaling when the number of active processes exceed a given average target value.

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: phpfpm-demo-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: phpfpm-demo
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metricName: phpfpm_active_processes
        targetAverageValue: 6

We have now defined an autoscaler with a minimum of 1 replica, which will increase to a max of 8. This will be triggered when the average value exceeds 6 active processes within phpfpm. Magical.

Useful links:

• https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling
• https://github.com/GoogleCloudPlatform/k8s-stackdriver
• https://github.com/craigmj/phpfpm_exporter