Using mcrouter and memcached as caching layer for Thanos Store

Figure 1. Photo by Dawid Zawiła on UnSplash

Introduction

Prometheus is the de-facto monitoring application for Kubernetes and cloud-native applications. By default, the TSDB (time series database) of Prometheus is stored on a file system. If you want to store the metrics on object storage for data retention or combine metrics from different Kubernetes clusters, there are some applications to check out. We have:

Thanos is one of the solutions I used in this article. Usually Thanos is deployed as a sidecar container with Prometheus. Thanos consists of several components. Some of the common components are:

Store Gateway

The Store Gateway implements the Store API. It communicates with the object storage to query metrics in there. It requires a small amount of local disk storage to store information about the remote object storage. According to this issue, using memcached as caching layer seems to reduce the amount of API requests made to the backend object storage. This is one the focus of this article.

Compactor

Even we have object storage, the amount of storage required to store the metrics is still large. To save storage space, we can compact (average) metrics in 5 minutes and 1 hour interval. This Compactor component would periodically compact the history data in the object storage.

Receiver

This component receive metrics from remote Prometheus instances.

Ruler

This component is similar to Alert Manager that is used with Prometheus. This could create alerts with respect to the metrics in Thanos.

Querier

This component receive queries and return metrics from Thanos backend components (Store Gateway and Prometheus).

Basic Thanos setup without memcached

Firstly, we would have a basic Thanos setup without memcached. Initially, I am using kube-prometheus stack without Thanos. Assuming we have a working kube-prometheus setup, then we add Thanos (kube-thanos) to it.

Here’s the additional configuration to the kube-prometheus jsonnet template:

        local t = (import 'kube-thanos/thanos.libsonnet');

local defaultPVCStorageClass = "cstor-pool2-r3";

local commonThanosConfig = {
  config+:: {
    local cfg = self,
    commonLabels+:: { 'app.identifier': 'thanos' },
    namespace: 'monitoring',
    version: 'v0.30.1',
    image: 'quay.io/thanos/thanos:' + cfg.version,
    imagePullPolicy: 'IfNotPresent',
    replicaLabels: ['prometheus_replica', 'rule_replica'],
    objectStorageConfig: {
      name: 'thanos-object-storage',
      key: 'thanos.yaml',
    },
    hashringConfigMapName: 'hashring-config',
    volumeClaimTemplate: {
      spec: {
        accessModes: ['ReadWriteOnce'],
        resources: {
          requests: {
            storage: '5Gi',
          },
        },
        storageClassName: defaultPVCStorageClass,
      },
    },
  },
},

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  (import 'github.com/thanos-io/thanos/mixin/config.libsonnet') +
  (import 'kube-prometheus/addons/all-namespaces.libsonnet') +
  {
     // .....
     prometheus+:: {
       // Thanos sidecar
       thanos: {
         objectStorageConfig: {
           name: commonThanosConfig.config.objectStorageConfig.name,
           key: commonThanosConfig.config.objectStorageConfig.key,
         },
         version: commonThanosConfig.config.version,
         image: commonThanosConfig.config.image,
       },
     },
    // .....
  };

local tc = t.compact (kp.values.common + commonThanosConfig.config {
  name: 'thanos-compact',
  replicas: 1,
  serviceMonitor: true,
  resources: { requests: { cpu: '1m', memory: '100Mi', }, limits: { cpu:1, memory: '1Gi' } },
  retentionResolutionRaw: '3d',
  retentionResolution5m: '10d',
  retentionResolution1h: '65d',
});

local ts = t.store(kp.values.common + commonThanosConfig.config {
  name: 'thanos-store',
  replicas: 1,
  serviceMonitor: true,
});

local tq = t.query(kp.values.common + commonThanosConfig.config {
  name: 'thanos-query',
  replicas: 1,
  serviceMonitor: true,
  stores: [
    'dnssrv+_grpc._tcp.%s.%s.svc.cluster.local' % [service.metadata.name, service.metadata.namespace]
    for service in [ts.service]
  ] + [
    'dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local:10901',
  ],
  autoDownsampling: true,
  ports: {
    grpc: 10901,
    http: 9090,
  },
});

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// ..... setting for kube-prometheus +

// additional settings for Thanos components
{ ['thanos-compact-' + name]: tc[name] for name in std.objectFields(tc) if tc[name] != null } +
{ ['thanos-store-' + name]: ts[name] for name in std.objectFields(ts) if ts[name] != null } +
{ ['thanos-query-' + name]: tq[name] for name in std.objectFields(tq) if tq[name] != null }

So, we have to prepare a secret called 'thanos-object-storage'. Inside the secret, it should contain the 'thanos.yaml' yaml file, in which the yaml file inside it should contain the credential and other information to access the backend object storage.

Here’s an example:

        ---
apiVersion: v1
kind: Secret
metadata:
  namespace: monitoring
  name: thanos-object-storage
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      insecure: false
      region: "ap-sydney-1"
      endpoint: "your-namespace.compat.objectstorage.ap-sydney-1.oraclecloud.com"
      bucket: "object-storage-thanos"
      access_key: "your-access-key"
      secret_key: "your-secret-key"
      aws_sdk_auth: false
      signature_version2: false

I am using the object storage from OCI, you need to create a group, bucket access policy, access key and secret key to access the bucket. Refer to my previous article about creating an OCI S3 bucket for storing backup for Velero.

Next, generate the manifests with the jsonnet template. Apply the manifests and restart deployments and stateful sets in the kube-prometheus stack.

Suppose the Grafana instance is deployed in the Kubernetes cluster. Then you should add a new data source in Grafana by this URL (check with the service created in your namespace):

http://thanos-query.{namespace}.svc.{cluster-name}:port-number

So it’s below URL for my case:

http://thanos-query.monitoring.svc.cluster.local:9090

Using mcrouter and memcached for caching

As we had mentioned before, when using memcached as caching layer, it seems to reduce the amount of API requests made to the backend object storage. This is to reduce cost if the object storage provider would incur cost for API invocations.

So, I would like to add memcached as the caching layer for Thanos Store. Adding a single memcached instance means no resilience, so we would like to add two memcached instances as a replica. An application called mcrouter by Facebook/Meta can act as the protocol router and replicate the changes to the two memcached instances.

But Facebook/Meta did not provide the actual container for mcrouter. Only sample Dockerfiles are provided. So I forked the Dockerfiles and scripts and created my own repository. I also run the container as non-root and support the ARM64 platform.

Setup two memcached instances

Here’s the setting I need for the two memcached instances:

        ---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: thanos-store-memcached
  labels:
    app: thanos-store-memcached

---
apiVersion: v1
kind: Service
metadata:
  name: thanos-store-memcached
  namespace: monitoring
  labels:
    app: thanos-store-memcached
spec:
  selector:
    app: thanos-store-memcached
  clusterIP: None
  ports:
    - name: thanos-store-memcached
      protocol: TCP
      port: 11211
      targetPort: 11211

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store-memcached
  namespace: monitoring
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: thanos-store-memcached
  serviceName: thanos-store-memcached
  template:
    metadata:
      labels:
        app: thanos-store-memcached
    spec:
      terminationGracePeriodSeconds: 15
      automountServiceAccountToken: false
      serviceAccountName: thanos-store-memcached
      securityContext:
        runAsUser: 11211
        runAsGroup: 11211
      containers:
        - name: memcached
          image: docker.io/library/memcached:1.6-bullseye
          args:  ["--memory-limit=448", "--conn-limit=2048"]
          ports:
            - containerPort: 11211
              protocol: TCP
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            runAsUser: 11211
            runAsGroup: 11211
            seccompProfile:
              type: RuntimeDefault
          resources:
            limits:
               memory: 480Mi
          livenessProbe:
            tcpSocket:
              port: 11211
            initialDelaySeconds: 30
            timeoutSeconds: 15
          readinessProbe:
            tcpSocket:
              port: 11211
            initialDelaySeconds: 10
            timeoutSeconds: 3

Setup the mcrouter

Here’s the setting I need for the mcrouter:

        ---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: monitoring
  name: thanos-store-memcached
  labels:
    app: thanos-store-memcached
data:
  config.json: |-
    {
      "pools": {
      "A": {
        "servers": [
          "thanos-store-memcached-0.thanos-store-memcached.monitoring.svc.cluster.local:11211",
          "thanos-store-memcached-1.thanos-store-memcached.monitoring.svc.cluster.local:11211"
        ]
      }
    },
      "route": {
        "type": "OperationSelectorRoute",
        "operation_policies": {
          "add": "AllSyncRoute|Pool|A",
          "delete": "AllSyncRoute|Pool|A",
          "get": "FailoverRoute|Pool|A",
          "set": "AllSyncRoute|Pool|A",
          "incr": "AllSyncRoute|Pool|A",
          "decr": "AllSyncRoute|Pool|A"
        }
      }
    }

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: thanos-store-mcrouter
  labels:
    app: thanos-store-mcrouter

---
apiVersion: v1
kind: Service
metadata:
  name: thanos-store-mcrouter
  namespace: monitoring
  labels:
    app: thanos-store-mcrouter
spec:
  selector:
    app: thanos-store-mcrouter
  type: ClusterIP
  ports:
    - name: thanos-store-mcrouter
      protocol: TCP
      port: 5000
      targetPort: 5000


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: monitoring
  name: thanos-store-mcrouter
  labels:
    app: thanos-store-mcrouter
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: thanos-store-mcrouter
  serviceName: thanos-store-mcrouter
  template:
    metadata:
      labels:
        app: thanos-store-mcrouter
    spec:
      terminationGracePeriodSeconds: 15
      automountServiceAccountToken: false
      serviceAccountName: thanos-store-mcrouter
      containers:
        - name: thanos-store-mcrouter
          image: registry.gitlab.com/patrickdung/docker-images/mcrouter-container:v0.3
          command: ["mcrouter"]
          args:
            - -p 5000
            - --config-file=/etc/mcrouter/config.json
          ports:
            - containerPort: 5000
              protocol: TCP
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            # Need to specify if the id used in the container is a numeric number
            # Need to match the id used in the container
            runAsUser: 20000
            runAsGroup: 20000
            seccompProfile:
              type: RuntimeDefault
          volumeMounts:
            - name: etc-mcrouter-config
              mountPath: /etc/mcrouter
            - name: var-mcrouter
              mountPath: /var/mcrouter
            - name: var-spool-mcrouter
              mountPath: /var/spool/mcrouter
          livenessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 30
            timeoutSeconds: 15
          readinessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 30
            timeoutSeconds: 15
      volumes:
        - name: etc-mcrouter-config
          configMap:
            name: thanos-store-memcached
            items:
              - key: config.json
                path: config.json
        - name: var-mcrouter
          emptyDir: {}
        - name: var-spool-mcrouter
          emptyDir: {}

Test if mcrouter is working

In this configuration, the mcrouter should send a request to both memcached instance. mcrouter itself acted like an memcached instance to the applications. Let’s try to test it:

        
# Connect to the mcrouter and create a key to be stored

$ telnet 10.43.121.145 5000
Trying 10.43.121.145...
Connected to 10.43.121.145.
Escape character is '^]'.
set key1 0 900 9
memcached
STORED
CLIENT_ERROR malformed request
Connection closed by foreign host.

# Query the two memcached instances:

$ telnet 10.42.1.186 11211
Trying 10.42.1.186...
Connected to 10.42.1.186.
Escape character is '^]'.
get key1
VALUE key1 0 9
memcached
END

$ telnet 10.42.2.143 11211
Trying 10.42.2.143...
Connected to 10.42.2.143.
Escape character is '^]'.
get key1
VALUE key1 0 9
memcached
END

Add a caching layer to Thanos Store

Replace below setting to the corresponding section for Thanos Store in the jsonnet template. We are making the request to the mcrouter instead to the individual memcached instances.

        local ts = t.store(kp.values.common + commonThanosConfig.config {
  name: 'thanos-store',
  replicas: 1,
  serviceMonitor: true,
  bucketCache: {
    type: 'memcached',
    config+: {
      addresses: ['dns+thanos-store-mcrouter.%s.svc.cluster.local:5000' % commonThanosConfig.config.namespace],
    },
  },
  indexCache: {
    type: 'memcached',
    config+: {
      addresses: ['dns+thanos-store-mcrouter.%s.svc.cluster.local:5000' % commonThanosConfig.config.namespace],
    },
  },
});

Generate the manifest again. Apply the manifests and restart deployments and stateful sets in the kube-prometheus stack. Then we should see the log of the thanos-store-0 pod that some requests were cached.

        level=info ts=2023-01-23T12:52:52.425001087Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=648.923062ms duration_ms=648 cached=90 returned=90 partial=0

If you run 'stats items' in the memcached instance, there should be items being cached:

        $ telnet 10.42.2.143 11211
Trying 10.42.2.143...
Connected to 10.42.2.143.
Escape character is '^]'.

stats items

STAT items:11:number 12
STAT items:11:number_hot 2
STAT items:11:number_warm 0
STAT items:11:number_cold 10
STAT items:11:age_hot 7219
STAT items:11:age_warm 0
STAT items:11:age 79219
STAT items:11:mem_requested 9528
STAT items:11:evicted 0
STAT items:11:evicted_nonzero 0
STAT items:11:evicted_time 0
STAT items:11:outofmemory 0
STAT items:11:tailrepairs 0
STAT items:11:reclaimed 778
STAT items:11:expired_unfetched 778
STAT items:11:evicted_unfetched 0
STAT items:11:evicted_active 0
STAT items:11:crawler_reclaimed 0
STAT items:11:crawler_items_checked 24219
STAT items:11:lrutail_reflocked 0
STAT items:11:moves_to_cold 8921
STAT items:11:moves_to_warm 0
STAT items:11:moves_within_lru 0
STAT items:11:direct_reclaims 0
STAT items:11:hits_to_hot 0
STAT items:11:hits_to_warm 0
STAT items:11:hits_to_cold 0
STAT items:11:hits_to_temp 0

References

An all in one example for kube-thanos

Using mcrouter and memcached as caching layer for Thanos Store

Introduction

Basic Thanos setup without memcached

Using mcrouter and memcached for caching

Setup two memcached instances

Setup the mcrouter

Test if mcrouter is working

Add a caching layer to Thanos Store

References

Share this article

Related articles

Perform backup and restore of K3S single master node due to problem with Leapp

Twitter responses: 2

Comments