本文介绍分布式配置管理中心Apollo(阿波罗)多节点k8s部署

参考文档:

Apollo介绍

Apollo(阿波罗)是一款可靠的分布式配置管理中心,诞生于携程框架研发部,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。

  • 服务端基于Spring Boot和Spring Cloud开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器。
  • Java客户端不依赖任何框架,能够运行于所有Java运行时环境,同时对Spring/Spring Boot环境也有较好的支持。
  • .Net客户端不依赖任何框架,能够运行于所有.Net运行时环境。

这里不再赘述,详细参见官方文档。

Apollo各组件介绍

  • apollo-portal 提供管理界面 (部署一份k8s)
    • 提供Web界面供用户管理配置
    • 通过Meta Server获取Admin Service服务列表(IP+Port),通过IP+Port访问服务
    • 在Portal侧做load balance、错误重试
  • apollo-adminservice (部署各节点k8s)
    • 提供配置管理接口
    • 提供配置修改、发布等接口
    • 接口服务对象为apollo-portal
  • apollo-configservice (部署各节点k8s)
    • 提供配置获取接口
    • 提供配置更新推送接口
    • 接口服务对象为Apollo客户端
  • Meta Server (随apollo-configservice部署各节点k8s)
    • Portal通过域名访问Meta Server获取Admin Service服务列表(IP+Port)
    • Client通过域名访问Meta Server获取Config Service服务列表(IP+Port)
    • Meta Server从Eureka获取Config Service和Admin Service的服务信息,相当于是一个Eureka Client
    • 增设一个Meta Server的角色主要是为了封装服务发现的细节,对Portal和Client而言,永远通过一个Http接口获取Admin Service和Config Service的服务信息,而不需要关心背后实际的服务注册和发现组件
    • Meta Server只是一个逻辑角色,在部署时和Config Service是在一个JVM进程中的,所以IP、端口和Config Service一致
  • Eureka (随apollo-configservice部署各节点k8s,多数据中心分布式部署时需要弃用)
    • 基于Eureka和Spring Cloud Netflix提供服务注册和发现
    • Config Service和Admin Service会向Eureka注册服务,并保持心跳
    • 目前Eureka在部署时和Config Service是在一个JVM进程中的(通过Spring Cloud Netflix)
  • Client(使用服务的应用程序)
    • Apollo提供的客户端程序,为应用提供配置获取、实时更新等功能
    • 通过Meta Server获取Config Service服务列表(IP+Port),通过IP+Port访问服务
    • 在Client侧做load balance、错误重试

环境规划

各节点服务规划

环境 地址 需要部署的服务
SH service-apollo-portal-server.baseservice:8070:30001
http://47.xxx.xx.39:30001/system_info.html

service-apollo-config-server-sh.baseservice:8080:30002
service-apollo-admin-server-sh.baseservice:8090:30003
apollo-portal

apollo-configservice
apollo-adminservice
SG service-apollo-config-server-sg.baseservice:8080:30002
172.21.114.195:30002

service-apollo-admin-server-sg.baseservice:8090:30003
172.21.114.195:30003
apollo-configservice

apollo-adminservice
LDN(待加入) localhost:8080
localhost:8091
apollo-configservice
apollo-adminservice

各节点数据库规划

环境 地址 库名(字符集)
SH rm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306 apolloportaldb(UTF8MB4)
SH rm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306 apolloconfigdb(UTF8MB4)
SG rm-t4xxxxxx91nov.mysql.singapore.rds.aliyuncs.com:3306 apolloconfigdb(UTF8MB4)
LDN(待加入) xxxx:3306 apolloconfigdb(UTF8MB4)

上海节点部署

创建授权mysql库

createdb
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql
initmysql

部署服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-mysql-for-apollo-sh-env
labels:
app: service-mysql-for-apollo-sh-env
spec:
type: ExternalName
externalName: rm-xxxxxhmo7w90110.mysql.rds.aliyuncs.com
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-admin-server-sh
labels:
app: service-apollo-admin-server-sh
spec:
ports:
- protocol: TCP
port: 8090
targetPort: 8090
nodePort: 30003
selector:
app: pod-apollo-admin-server-sh
type: NodePort
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-meta-server-sh
labels:
app: service-apollo-meta-server-sh
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: pod-apollo-config-server-sh
type: ClusterIP
clusterIP: None
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-config-server-sh
labels:
app: service-apollo-config-server-sh
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
nodePort: 30002
selector:
app: pod-apollo-config-server-sh
type: NodePort
sessionAffinity: ClientIP

---
# configmap for apollo-admin-server-sh
kind: ConfigMap
apiVersion: v1
metadata:
namespace: baseservice
name: configmap-apollo-admin-server-sh
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
spring.datasource.username = wikifx
spring.datasource.password = Wikifx123
---
# configmap for apollo-config-server-sh
kind: ConfigMap
apiVersion: v1
metadata:
namespace: baseservice
name: configmap-apollo-config-server-sh
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
spring.datasource.username = wikifx
spring.datasource.password = Wikifx123
apollo.admin-service.url = http://service-apollo-admin-server-sh.baseservice:8090
apollo.config-service.url = http://service-apollo-config-server-sh.baseservice:8080
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: baseservice
name: deployment-apollo-admin-server-sh
labels:
app: deployment-apollo-admin-server-sh
spec:
replicas: 2
selector:
matchLabels:
app: pod-apollo-admin-server-sh
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-admin-server-sh
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-admin-server-sh
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-admin-server-sh
configMap:
name: configmap-apollo-admin-server-sh
containers:
- image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-admin-server-sh
ports:
- protocol: TCP
containerPort: 8090
volumeMounts:
- name: volume-configmap-apollo-admin-server-sh
mountPath: /apollo-adminservice/config/application-github.properties
subPath: application-github.properties
env:
- name: SPRING_PROFILES_ACTIVE
value: "github,kubernetes"
readinessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 10
periodSeconds: 5

livenessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
namespace: baseservice
name: statefulset-apollo-config-server-sh
labels:
app: statefulset-apollo-config-server-sh
spec:
serviceName: service-apollo-meta-server-sh
replicas: 2
selector:
matchLabels:
app: pod-apollo-config-server-sh
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-sh
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-config-server-sh
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-config-server-sh
configMap:
name: configmap-apollo-config-server-sh
containers:
- image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-config-server-sh
ports:
- protocol: TCP
containerPort: 8080
volumeMounts:
- name: volume-configmap-apollo-config-server-sh
mountPath: /apollo-configservice/config/application-github.properties
subPath: application-github.properties
env:
- name: SPRING_PROFILES_ACTIVE
value: "github,kubernetes"
# 添加环境变量禁用apollo原生Eureka
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 5

livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
# kubectl apply -f allinone.yaml
---

配置eureka地址(传统部署非常重要)
apolloconfig 在服务启动后会启动一个eureka服务注册中心,apolloadmin 和apolloconfig 都会注册其中,eurek的连接地址配置在数据库中配置或者在ConfigMap中配置:
格式为:http://<config server pod名>.<meta server 服务名>:<meta server端口号>/eureka/

  • 方式一:通过Spring Boot文件 application-github.properties配置(推荐)
    推荐此方式配置 eureka.service.url,因为可以通过ConfigMap的方式传入容器,无需再修改数据库的字段。如果该配置希望以数据库中为准,那么在 yaml 中直接删除该配置项即可。
  • 方式二:修改数据表 ApolloConfigDB.ServerConfig
    修改数据库表 ApolloConfigDB.ServerConfig的 eureka.service.url。
    install

Apollo 1.7.0版本开始增加了基于Kubernetes原生服务发现的部署模式,由于不再使用内置的Eureka,所以在整体部署上有很大简化。

以上配置禁用了Apollo原生的Eureka,采用了Kubernetes原生服务发现
ck

新加坡节点部署

同上,注意修改数据库连接串,以及对应节点的数据库账号和密码。
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-mysql-for-apollo-sg-env
labels:
app: service-mysql-for-apollo-sg-env
spec:
type: ExternalName
externalName: rm-xxxxxxov.mysql.singapore.rds.aliyuncs.com
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-admin-server-sg
labels:
app: service-apollo-admin-server-sg
spec:
ports:
- protocol: TCP
port: 8090
targetPort: 8090
nodePort: 30003
selector:
app: pod-apollo-admin-server-sg
type: NodePort
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-meta-server-sg
labels:
app: service-apollo-meta-server-sg
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: pod-apollo-config-server-sg
type: ClusterIP
clusterIP: None
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-config-server-sg
labels:
app: service-apollo-config-server-sg
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
nodePort: 30002
selector:
app: pod-apollo-config-server-sg
type: NodePort
sessionAffinity: ClientIP

---
kind: ConfigMap
apiVersion: v1
metadata:
namespace: baseservice
name: configmap-apollo-admin-server-sg
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
spring.datasource.username = wikifx
spring.datasource.password = Wikifx2022
---
# configmap for apollo-config-server-sg
kind: ConfigMap
apiVersion: v1
metadata:
namespace: baseservice
name: configmap-apollo-config-server-sg
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
spring.datasource.username = wikifx
spring.datasource.password = Wikifx2022
apollo.admin-service.url = http://172.21.114.195:30003
apollo.config-service.url = http://172.21.114.195:30002
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: baseservice
name: deployment-apollo-admin-server-sg
labels:
app: deployment-apollo-admin-server-sg
spec:
replicas: 2
selector:
matchLabels:
app: pod-apollo-admin-server-sg
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-admin-server-sg
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-admin-server-sg
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-admin-server-sg
configMap:
name: configmap-apollo-admin-server-sg
containers:
- image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-admin-server-sg
ports:
- protocol: TCP
containerPort: 8090
volumeMounts:
- name: volume-configmap-apollo-admin-server-sg
mountPath: /apollo-adminservice/config/application-github.properties
subPath: application-github.properties
env:
- name: SPRING_PROFILES_ACTIVE
value: "github,kubernetes"
readinessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 10
periodSeconds: 5

livenessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
namespace: baseservice
name: statefulset-apollo-config-server-sg
labels:
app: statefulset-apollo-config-server-sg
spec:
serviceName: service-apollo-meta-server-sg
replicas: 2
selector:
matchLabels:
app: pod-apollo-config-server-sg
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-sg
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-config-server-sg
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-config-server-sg
configMap:
name: configmap-apollo-config-server-sg
containers:
- image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-config-server-sg
ports:
- protocol: TCP
containerPort: 8080
volumeMounts:
- name: volume-configmap-apollo-config-server-sg
mountPath: /apollo-configservice/config/application-github.properties
subPath: application-github.properties
env:
- name: SPRING_PROFILES_ACTIVE
value: "github,kubernetes"
# 添加环境变量禁用apollo原生Eureka
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 5

livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
# kubectl apply -f allinone.yaml

指定Meta Server返回的apollo-configservice和apollo-adminservice地址。(多数据中心k8s部署必要)

1
2
3
4
5
6
7
# configmap for apollo-config-server-sg
kind: ConfigMap
apiVersion: v1
...
apollo.admin-service.url = http://172.21.114.195:30003
apollo.config-service.url = http://172.21.114.195:30002
#这里的地址为线上K8S集群前端的SLB地址

以上配置禁用了Apollo原生的Eureka,采用了Kubernetes原生服务发现
ck

ApolloPortal部署

创建授权mysql库

create
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloportaldb.sql
init

开始部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-mysql-for-portal-server
labels:
app: service-mysql-for-portal-server
spec:
type: ExternalName
externalName: rm-uf6xxxxxxw90110.mysql.rds.aliyuncs.com

---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
namespace: baseservice
name: configmap-apollo-portal-server
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-portal-server.baseservice:3306/apolloportaldb?characterEncoding=utf8
# mysql username
spring.datasource.username = wikifx
# mysql password
spring.datasource.password = Wikifx123
# enabled environments
apollo.portal.envs = sh,sg
apollo-env.properties: |
sh.meta=http://service-apollo-config-server-sh.baseservice:8080
sg.meta=http://172.21.114.195:30002

---
kind: Service
apiVersion: v1
metadata:
namespace: baseservice
name: service-apollo-portal-server
labels:
app: service-apollo-portal-server
spec:
ports:
- protocol: TCP
port: 8070
targetPort: 8070
nodePort: 30001
selector:
app: pod-apollo-portal-server
type: NodePort
# portal session 保持
sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: baseservice
name: deployment-apollo-portal-server
labels:
app: deployment-apollo-portal-server
spec:
# 2 个实例
replicas: 2
selector:
matchLabels:
app: pod-apollo-portal-server
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-portal-server
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-portal-server
topologyKey: kubernetes.io/hostname

volumes:
- name: volume-configmap-apollo-portal-server
configMap:
name: configmap-apollo-portal-server
items:
- key: application-github.properties
path: application-github.properties
- key: apollo-env.properties
path: apollo-env.properties
containers:
- image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-portal-1.9.0-SNAPSHOT
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-portal-server
ports:
- protocol: TCP
containerPort: 8070
volumeMounts:
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal/config/application-github.properties
subPath: application-github.properties
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal/config/apollo-env.properties
subPath: apollo-env.properties
env:
- name: APOLLO_PORTAL_SERVICE_NAME
value: "service-apollo-portal-server.baseservice"
readinessProbe:
tcpSocket:
port: 8070
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8070
# 120s 内, server 未启动则重启 container
initialDelaySeconds: 120
periodSeconds: 15
dnsPolicy: ClusterFirst
restartPolicy: Always

# kubectl apply -f allinone.yaml

检查

ck
ck
ck