本文是从零学习envoy笔记第四章 - Backend Utilization
  • 从零学习envoy笔记 - Gateway总览
  • 从零学习envoy笔记 - 学习EG API和Gateway API
  • 从零学习envoy笔记 - 快速部署实验环境
  • 从零学习envoy笔记 - 官方Task Backend Routing
  • 从零学习envoy笔记 - 官方Task Backend Utilization Load Balancing
  • 从零学习envoy笔记 - 官方Task Bandwidth Limit
  • 从零学习envoy笔记 - 官方Task Client Traffic Policy

BackendTrafficPolicy

Backend Utilization Load Balancing 就是根据服务利用率来动态调整负载均衡权重 (使用ORCA load metrics)

ORCA Open Request Cost Aggregation在 Envoy/gRPC 生态里通常是一种让 backend 向 L7 负载均衡器汇报自身负载的标准。gRPC 官方也把 ORCA 描述为一种传递 backend metrics 的开放标准。

使用官方示例完成实验

官方镜像没有仓库需要编译，这里使用 AI 生成一份使用了 ORCA 指标的 YAML 更好完成实验

yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205

cat <<YAML | k apply -n default -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-end-gray-orca
  namespace: default
  labels:
    app: gray-red-orca
    version: gray
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gray-red-orca
      version: gray
  template:
    metadata:
      labels:
        app: gray-red-orca
        version: gray
    spec:
      containers:
        - name: app
          image: python:3.12-alpine
          ports:
            - containerPort: 8080
          env:
            - name: COLOR
              value: gray
            - name: ORCA_CPU_UTILIZATION
              value: "0.1"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          command:
            - sh
            - -c
            - |
              cat > /tmp/server.py <<'PY'
              import os
              import json
              from http.server import BaseHTTPRequestHandler, HTTPServer

              COLOR = os.getenv("COLOR", "unknown")
              POD_NAME = os.getenv("POD_NAME", "unknown")
              CPU = os.getenv("ORCA_CPU_UTILIZATION", "0.5")

              class Handler(BaseHTTPRequestHandler):
                  protocol_version = "HTTP/1.1"

                  def do_GET(self):
                      body = json.dumps({
                          "color": COLOR,
                          "pod": POD_NAME,
                          "cpu_utilization": float(CPU),
                          "path": self.path
                      }).encode()

                      self.send_response(200)
                      self.send_header("content-type", "application/json")
                      self.send_header("content-length", str(len(body)))
                      self.send_header("connection", "close")
                      self.send_header(
                          "endpoint-load-metrics",
                          f"TEXT cpu_utilization={CPU},rps_fractional=1.0,eps=0"
                      )
                      self.end_headers()
                      self.wfile.write(body)

                  def log_message(self, format, *args):
                      return

              HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
              PY
              python /tmp/server.py              

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-end-red-orca
  namespace: default
  labels:
    app: gray-red-orca
    version: red
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gray-red-orca
      version: red
  template:
    metadata:
      labels:
        app: gray-red-orca
        version: red
    spec:
      containers:
        - name: app
          image: python:3.12-alpine
          ports:
            - containerPort: 8080
          env:
            - name: COLOR
              value: red
            - name: ORCA_CPU_UTILIZATION
              value: "0.9"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          command:
            - sh
            - -c
            - |
              cat > /tmp/server.py <<'PY'
              import os
              import json
              from http.server import BaseHTTPRequestHandler, HTTPServer

              COLOR = os.getenv("COLOR", "unknown")
              POD_NAME = os.getenv("POD_NAME", "unknown")
              CPU = os.getenv("ORCA_CPU_UTILIZATION", "0.5")

              class Handler(BaseHTTPRequestHandler):
                  protocol_version = "HTTP/1.1"

                  def do_GET(self):
                      body = json.dumps({
                          "color": COLOR,
                          "pod": POD_NAME,
                          "cpu_utilization": float(CPU),
                          "path": self.path
                      }).encode()

                      self.send_response(200)
                      self.send_header("content-type", "application/json")
                      self.send_header("content-length", str(len(body)))
                      self.send_header("connection", "close")
                      self.send_header(
                          "endpoint-load-metrics",
                          f"TEXT cpu_utilization={CPU},rps_fractional=1.0,eps=0"
                      )
                      self.end_headers()
                      self.wfile.write(body)

                  def log_message(self, format, *args):
                      return

              HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
              PY
              python /tmp/server.py              
---
apiVersion: v1
kind: Service
metadata:
  name: gray-red-orca
  namespace: default
spec:
  selector:
    app: gray-red-orca
  ports:
    - name: http
      port: 8080
      targetPort: 8080

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: gray-red-orca
  namespace: default
spec:
  parentRefs:
    - name: eg
  hostnames:
    - utilization.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: gray-red-orca
          port: 8080

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: gray-red-orca-utilization
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: gray-red-orca
  loadBalancer:
    type: BackendUtilization
    backendUtilization:
      blackoutPeriod: 1s
      weightUpdatePeriod: 500ms
      keepResponseHeaders: true
YAML

测试结果

text
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-cmdf6",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-h6cvf",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-h6cvf",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-cmdf6",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-8lt58",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-h6cvf",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-8lt58",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-plps5",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-cmdf6",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-8lt58",
  "cpu_utilization": 0.1,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-h6cvf",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "red",
  "pod": "envoy-end-red-orca-864d4b9b44-plps5",
  "cpu_utilization": 0.9,
  "path": "/"
}
$ curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq
{
  "color": "gray",
  "pod": "envoy-end-gray-orca-7fb8bbdc66-cmdf6",
  "cpu_utilization": 0.1,
  "path": "/"
}

批量测试

text
1
2
3
4
5

$ for i in $(seq 1 200); do   curl -s -H "Host: utilization.example.com" http://127.0.0.1:8888/ | jq -r '.pod'; done | sort | uniq -c
     92 envoy-end-gray-orca-7684ddcb7-dgpjt
     88 envoy-end-gray-orca-7684ddcb7-rxkhk
     10 envoy-end-red-orca-fbd446567-2rpnz
     10 envoy-end-red-orca-fbd446567-k828n

这里使用的策略是

yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
# 这里主要用它开启 BackendUtilization 负载均衡。
metadata:
  name: gray-red-orca-utilization
  namespace: default
spec:
  loadBalancer:
  # 配置负载均衡策略。
    backendUtilization:
    # BackendUtilization 的详细参数。
    # BackendUtilization 的意思是：根据后端上报的利用率指标动态调整流量比例。
    # 比如 gray 上报 cpu_utilization=0.1，red 上报 cpu_utilization=0.9，
    # Envoy 就会认为 gray 更空闲，red 更繁忙，于是更多请求会转发到 gray。

      blackoutPeriod: 1s
      # 黑窗期，表示 endpoint 开始上报负载指标后，等待 1 秒再信任这个指标。
      # 作用是避免刚启动、刚加入集群的后端指标不稳定，导致流量权重频繁抖动。
      # 官方默认通常更长，实验里设置 1s 是为了更快看到效果。

      keepResponseHeaders: true
      # 是否把后端返回的 ORCA 指标响应头保留下来。
      # true 表示客户端可以通过 curl -i 看到类似下面的 header：
      # endpoint-load-metrics: JSON {"cpu_utilization": 0.1}
      # 这个主要用于调试，方便确认后端确实返回了 ORCA metrics。

      weightUpdatePeriod: 500ms
      # 权重更新周期。
      # 表示 Envoy 每 500 毫秒根据后端上报的利用率重新计算一次 endpoint 权重。
      # 周期越短，权重调整越快；周期太短也可能带来更多计算和抖动。

    type: BackendUtilization
    # 指定负载均衡类型为 BackendUtilization。
    # 这表示不再只是普通 Round Robin，而是根据后端利用率动态加权。
    # 空闲的 backend 权重大，繁忙的 backend 权重小。

  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: gray-red-orca

通过这个个示例很好的了解了 BackendTrafficPolicy 使用，ORCA 原理是怎么实现的，其核心就是 Response Header

text
1

endpoint-load-metrics: TEXT cpu_utilization=0.1,rps_fractional=1.0,eps=0

就是通过 Python 设置了一个头来完成的。

环境清理

text
1
2
3
4

k delete backendtrafficpolicy gray-red-orca-utilization
k delete httproute gray-red-orca
k delete svc gray-red-orca
k delete deploy envoy-end-gray-orca envoy-end-red-orca