Using mcrouter and memcached as caching layer for Thanos Store
Category: monitoring
Modified: Sat, 2023-Jan-28

Introduction
Prometheus is the de-facto monitoring application for Kubernetes and cloud-native applications. By default, the TSDB (time series database) of Prometheus is stored on a file system. If you want to store the metrics on object storage for data retention or combine metrics from different Kubernetes clusters, there are some applications to check out. We have:
Thanos is one of the solutions I used in this article. Usually Thanos is deployed as a sidecar container with Prometheus. Thanos consists of several components. Some of the common components are:
-
Store Gateway
The Store Gateway implements the Store API. It communicates with the object storage to query metrics in there. It requires a small amount of local disk storage to store information about the remote object storage. According to this issue, using memcached as caching layer seems to reduce the amount of API requests made to the backend object storage. This is one the focus of this article.
-
Compactor
Even we have object storage, the amount of storage required to store the metrics is still large. To save storage space, we can compact (average) metrics in 5 minutes and 1 hour interval. This Compactor component would periodically compact the history data in the object storage.
-
Receiver
This component receive metrics from remote Prometheus instances.
-
Ruler
This component is similar to Alert Manager that is used with Prometheus. This could create alerts with respect to the metrics in Thanos.
-
Querier
This component receive queries and return metrics from Thanos backend components (Store Gateway and Prometheus).
Basic Thanos setup without memcached
Firstly, we would have a basic Thanos setup without memcached. Initially, I am using kube-prometheus stack without Thanos. Assuming we have a working kube-prometheus setup, then we add Thanos (kube-thanos) to it.
Here’s the additional configuration to the kube-prometheus jsonnet template:
local t = (import 'kube-thanos/thanos.libsonnet');
local defaultPVCStorageClass = "cstor-pool2-r3";
local commonThanosConfig = {
config+:: {
local cfg = self,
commonLabels+:: { 'app.identifier': 'thanos' },
namespace: 'monitoring',
version: 'v0.30.1',
image: 'quay.io/thanos/thanos:' + cfg.version,
imagePullPolicy: 'IfNotPresent',
replicaLabels: ['prometheus_replica', 'rule_replica'],
objectStorageConfig: {
name: 'thanos-object-storage',
key: 'thanos.yaml',
},
hashringConfigMapName: 'hashring-config',
volumeClaimTemplate: {
spec: {
accessModes: ['ReadWriteOnce'],
resources: {
requests: {
storage: '5Gi',
},
},
storageClassName: defaultPVCStorageClass,
},
},
},
},
local kp =
(import 'kube-prometheus/main.libsonnet') +
(import 'github.com/thanos-io/thanos/mixin/config.libsonnet') +
(import 'kube-prometheus/addons/all-namespaces.libsonnet') +
{
// .....
prometheus+:: {
// Thanos sidecar
thanos: {
objectStorageConfig: {
name: commonThanosConfig.config.objectStorageConfig.name,
key: commonThanosConfig.config.objectStorageConfig.key,
},
version: commonThanosConfig.config.version,
image: commonThanosConfig.config.image,
},
},
// .....
};
local tc = t.compact (kp.values.common + commonThanosConfig.config {
name: 'thanos-compact',
replicas: 1,
serviceMonitor: true,
resources: { requests: { cpu: '1m', memory: '100Mi', }, limits: { cpu:1, memory: '1Gi' } },
retentionResolutionRaw: '3d',
retentionResolution5m: '10d',
retentionResolution1h: '65d',
});
local ts = t.store(kp.values.common + commonThanosConfig.config {
name: 'thanos-store',
replicas: 1,
serviceMonitor: true,
});
local tq = t.query(kp.values.common + commonThanosConfig.config {
name: 'thanos-query',
replicas: 1,
serviceMonitor: true,
stores: [
'dnssrv+_grpc._tcp.%s.%s.svc.cluster.local' % [service.metadata.name, service.metadata.namespace]
for service in [ts.service]
] + [
'dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local:10901',
],
autoDownsampling: true,
ports: {
grpc: 10901,
http: 9090,
},
});
{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// ..... setting for kube-prometheus +
// additional settings for Thanos components
{ ['thanos-compact-' + name]: tc[name] for name in std.objectFields(tc) if tc[name] != null } +
{ ['thanos-store-' + name]: ts[name] for name in std.objectFields(ts) if ts[name] != null } +
{ ['thanos-query-' + name]: tq[name] for name in std.objectFields(tq) if tq[name] != null }
So, we have to prepare a secret called 'thanos-object-storage'. Inside the secret, it should contain the 'thanos.yaml' yaml file, in which the yaml file inside it should contain the credential and other information to access the backend object storage.
Here’s an example:
---
apiVersion: v1
kind: Secret
metadata:
namespace: monitoring
name: thanos-object-storage
type: Opaque
stringData:
thanos.yaml: |
type: s3
config:
insecure: false
region: "ap-sydney-1"
endpoint: "your-namespace.compat.objectstorage.ap-sydney-1.oraclecloud.com"
bucket: "object-storage-thanos"
access_key: "your-access-key"
secret_key: "your-secret-key"
aws_sdk_auth: false
signature_version2: false
I am using the object storage from OCI, you need to create a group, bucket access policy, access key and secret key to access the bucket. Refer to my previous article about creating an OCI S3 bucket for storing backup for Velero.
Next, generate the manifests with the jsonnet template. Apply the manifests and restart deployments and stateful sets in the kube-prometheus stack.
Suppose the Grafana instance is deployed in the Kubernetes cluster. Then you should add a new data source in Grafana by this URL (check with the service created in your namespace):
So it’s below URL for my case:
Using mcrouter and memcached for caching
As we had mentioned before, when using memcached as caching layer, it seems to reduce the amount of API requests made to the backend object storage. This is to reduce cost if the object storage provider would incur cost for API invocations.
So, I would like to add memcached as the caching layer for Thanos Store. Adding a single memcached instance means no resilience, so we would like to add two memcached instances as a replica. An application called mcrouter by Facebook/Meta can act as the protocol router and replicate the changes to the two memcached instances.
But Facebook/Meta did not provide the actual container for mcrouter. Only sample Dockerfiles are provided. So I forked the Dockerfiles and scripts and created my own repository. I also run the container as non-root and support the ARM64 platform.
Setup two memcached instances
Here’s the setting I need for the two memcached instances:
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: monitoring
name: thanos-store-memcached
labels:
app: thanos-store-memcached
---
apiVersion: v1
kind: Service
metadata:
name: thanos-store-memcached
namespace: monitoring
labels:
app: thanos-store-memcached
spec:
selector:
app: thanos-store-memcached
clusterIP: None
ports:
- name: thanos-store-memcached
protocol: TCP
port: 11211
targetPort: 11211
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-store-memcached
namespace: monitoring
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
selector:
matchLabels:
app: thanos-store-memcached
serviceName: thanos-store-memcached
template:
metadata:
labels:
app: thanos-store-memcached
spec:
terminationGracePeriodSeconds: 15
automountServiceAccountToken: false
serviceAccountName: thanos-store-memcached
securityContext:
runAsUser: 11211
runAsGroup: 11211
containers:
- name: memcached
image: docker.io/library/memcached:1.6-bullseye
args: ["--memory-limit=448", "--conn-limit=2048"]
ports:
- containerPort: 11211
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 11211
runAsGroup: 11211
seccompProfile:
type: RuntimeDefault
resources:
limits:
memory: 480Mi
livenessProbe:
tcpSocket:
port: 11211
initialDelaySeconds: 30
timeoutSeconds: 15
readinessProbe:
tcpSocket:
port: 11211
initialDelaySeconds: 10
timeoutSeconds: 3
Setup the mcrouter
Here’s the setting I need for the mcrouter:
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: monitoring
name: thanos-store-memcached
labels:
app: thanos-store-memcached
data:
config.json: |-
{
"pools": {
"A": {
"servers": [
"thanos-store-memcached-0.thanos-store-memcached.monitoring.svc.cluster.local:11211",
"thanos-store-memcached-1.thanos-store-memcached.monitoring.svc.cluster.local:11211"
]
}
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"add": "AllSyncRoute|Pool|A",
"delete": "AllSyncRoute|Pool|A",
"get": "FailoverRoute|Pool|A",
"set": "AllSyncRoute|Pool|A",
"incr": "AllSyncRoute|Pool|A",
"decr": "AllSyncRoute|Pool|A"
}
}
}
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: monitoring
name: thanos-store-mcrouter
labels:
app: thanos-store-mcrouter
---
apiVersion: v1
kind: Service
metadata:
name: thanos-store-mcrouter
namespace: monitoring
labels:
app: thanos-store-mcrouter
spec:
selector:
app: thanos-store-mcrouter
type: ClusterIP
ports:
- name: thanos-store-mcrouter
protocol: TCP
port: 5000
targetPort: 5000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: monitoring
name: thanos-store-mcrouter
labels:
app: thanos-store-mcrouter
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
selector:
matchLabels:
app: thanos-store-mcrouter
serviceName: thanos-store-mcrouter
template:
metadata:
labels:
app: thanos-store-mcrouter
spec:
terminationGracePeriodSeconds: 15
automountServiceAccountToken: false
serviceAccountName: thanos-store-mcrouter
containers:
- name: thanos-store-mcrouter
image: registry.gitlab.com/patrickdung/docker-images/mcrouter-container:v0.3
command: ["mcrouter"]
args:
- -p 5000
- --config-file=/etc/mcrouter/config.json
ports:
- containerPort: 5000
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
# Need to specify if the id used in the container is a numeric number
# Need to match the id used in the container
runAsUser: 20000
runAsGroup: 20000
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: etc-mcrouter-config
mountPath: /etc/mcrouter
- name: var-mcrouter
mountPath: /var/mcrouter
- name: var-spool-mcrouter
mountPath: /var/spool/mcrouter
livenessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 30
timeoutSeconds: 15
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 30
timeoutSeconds: 15
volumes:
- name: etc-mcrouter-config
configMap:
name: thanos-store-memcached
items:
- key: config.json
path: config.json
- name: var-mcrouter
emptyDir: {}
- name: var-spool-mcrouter
emptyDir: {}
Test if mcrouter is working
In this configuration, the mcrouter should send a request to both memcached instance. mcrouter itself acted like an memcached instance to the applications. Let’s try to test it:
# Connect to the mcrouter and create a key to be stored
$ telnet 10.43.121.145 5000
Trying 10.43.121.145...
Connected to 10.43.121.145.
Escape character is '^]'.
set key1 0 900 9
memcached
STORED
CLIENT_ERROR malformed request
Connection closed by foreign host.
# Query the two memcached instances:
$ telnet 10.42.1.186 11211
Trying 10.42.1.186...
Connected to 10.42.1.186.
Escape character is '^]'.
get key1
VALUE key1 0 9
memcached
END
$ telnet 10.42.2.143 11211
Trying 10.42.2.143...
Connected to 10.42.2.143.
Escape character is '^]'.
get key1
VALUE key1 0 9
memcached
END
Add a caching layer to Thanos Store
Replace below setting to the corresponding section for Thanos Store in the jsonnet template. We are making the request to the mcrouter instead to the individual memcached instances.
local ts = t.store(kp.values.common + commonThanosConfig.config {
name: 'thanos-store',
replicas: 1,
serviceMonitor: true,
bucketCache: {
type: 'memcached',
config+: {
addresses: ['dns+thanos-store-mcrouter.%s.svc.cluster.local:5000' % commonThanosConfig.config.namespace],
},
},
indexCache: {
type: 'memcached',
config+: {
addresses: ['dns+thanos-store-mcrouter.%s.svc.cluster.local:5000' % commonThanosConfig.config.namespace],
},
},
});
Generate the manifest again. Apply the manifests and restart deployments and stateful sets in the kube-prometheus stack. Then we should see the log of the thanos-store-0 pod that some requests were cached.
level=info ts=2023-01-23T12:52:52.425001087Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=648.923062ms duration_ms=648 cached=90 returned=90 partial=0
If you run 'stats items' in the memcached instance, there should be items being cached:
$ telnet 10.42.2.143 11211
Trying 10.42.2.143...
Connected to 10.42.2.143.
Escape character is '^]'.
stats items
STAT items:11:number 12
STAT items:11:number_hot 2
STAT items:11:number_warm 0
STAT items:11:number_cold 10
STAT items:11:age_hot 7219
STAT items:11:age_warm 0
STAT items:11:age 79219
STAT items:11:mem_requested 9528
STAT items:11:evicted 0
STAT items:11:evicted_nonzero 0
STAT items:11:evicted_time 0
STAT items:11:outofmemory 0
STAT items:11:tailrepairs 0
STAT items:11:reclaimed 778
STAT items:11:expired_unfetched 778
STAT items:11:evicted_unfetched 0
STAT items:11:evicted_active 0
STAT items:11:crawler_reclaimed 0
STAT items:11:crawler_items_checked 24219
STAT items:11:lrutail_reflocked 0
STAT items:11:moves_to_cold 8921
STAT items:11:moves_to_warm 0
STAT items:11:moves_within_lru 0
STAT items:11:direct_reclaims 0
STAT items:11:hits_to_hot 0
STAT items:11:hits_to_warm 0
STAT items:11:hits_to_cold 0
STAT items:11:hits_to_temp 0
Comments
No. of comments: 0
Please read and agree the privacy policy before using the comment system.