本文介绍分布式配置管理中心Apollo(阿波罗)多节点k8s部署
参考文档:
Apollo介绍
Apollo(阿波罗)是一款可靠的分布式配置管理中心,诞生于携程框架研发部,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。
- 服务端基于Spring Boot和Spring Cloud开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器。
- Java客户端不依赖任何框架,能够运行于所有Java运行时环境,同时对Spring/Spring Boot环境也有较好的支持。
- .Net客户端不依赖任何框架,能够运行于所有.Net运行时环境。
这里不再赘述,详细参见官方文档。
Apollo各组件介绍
- apollo-portal 提供管理界面 (部署一份k8s)
- 提供Web界面供用户管理配置
- 通过Meta Server获取Admin Service服务列表(IP+Port),通过IP+Port访问服务
- 在Portal侧做load balance、错误重试
- apollo-adminservice (部署各节点k8s)
- 提供配置管理接口
- 提供配置修改、发布等接口
- 接口服务对象为apollo-portal
- apollo-configservice (部署各节点k8s)
- 提供配置获取接口
- 提供配置更新推送接口
- 接口服务对象为Apollo客户端
- Meta Server (随apollo-configservice部署各节点k8s)
- Portal通过域名访问Meta Server获取Admin Service服务列表(IP+Port)
- Client通过域名访问Meta Server获取Config Service服务列表(IP+Port)
- Meta Server从Eureka获取Config Service和Admin Service的服务信息,相当于是一个Eureka Client
- 增设一个Meta Server的角色主要是为了封装服务发现的细节,对Portal和Client而言,永远通过一个Http接口获取Admin Service和Config Service的服务信息,而不需要关心背后实际的服务注册和发现组件
- Meta Server只是一个逻辑角色,在部署时和Config Service是在一个JVM进程中的,所以IP、端口和Config Service一致
- Eureka (随apollo-configservice部署各节点k8s,多数据中心分布式部署时需要弃用)
- 基于Eureka和Spring Cloud Netflix提供服务注册和发现
- Config Service和Admin Service会向Eureka注册服务,并保持心跳
- 目前Eureka在部署时和Config Service是在一个JVM进程中的(通过Spring Cloud Netflix)
- Client(使用服务的应用程序)
- Apollo提供的客户端程序,为应用提供配置获取、实时更新等功能
- 通过Meta Server获取Config Service服务列表(IP+Port),通过IP+Port访问服务
- 在Client侧做load balance、错误重试
环境规划
各节点服务规划
环境 |
地址 |
需要部署的服务 |
SH |
service-apollo-portal-server.baseservice:8070:30001 http://47.xxx.xx.39:30001/system_info.html
service-apollo-config-server-sh.baseservice:8080:30002 service-apollo-admin-server-sh.baseservice:8090:30003 |
apollo-portal
apollo-configservice apollo-adminservice |
SG |
service-apollo-config-server-sg.baseservice:8080:30002 172.21.114.195:30002
service-apollo-admin-server-sg.baseservice:8090:30003 172.21.114.195:30003 |
apollo-configservice
apollo-adminservice |
LDN(待加入) |
localhost:8080 localhost:8091 |
apollo-configservice apollo-adminservice |
各节点数据库规划
环境 |
地址 |
库名(字符集) |
SH |
rm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306 |
apolloportaldb(UTF8MB4) |
SH |
rm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306 |
apolloconfigdb(UTF8MB4) |
SG |
rm-t4xxxxxx91nov.mysql.singapore.rds.aliyuncs.com:3306 |
apolloconfigdb(UTF8MB4) |
LDN(待加入) |
xxxx:3306 |
apolloconfigdb(UTF8MB4) |
上海节点部署
创建授权mysql库
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql
部署服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229
| kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-mysql-for-apollo-sh-env labels: app: service-mysql-for-apollo-sh-env spec: type: ExternalName externalName: rm-xxxxxhmo7w90110.mysql.rds.aliyuncs.com --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-admin-server-sh labels: app: service-apollo-admin-server-sh spec: ports: - protocol: TCP port: 8090 targetPort: 8090 nodePort: 30003 selector: app: pod-apollo-admin-server-sh type: NodePort sessionAffinity: ClientIP --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-meta-server-sh labels: app: service-apollo-meta-server-sh spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: pod-apollo-config-server-sh type: ClusterIP clusterIP: None sessionAffinity: ClientIP --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-config-server-sh labels: app: service-apollo-config-server-sh spec: ports: - protocol: TCP port: 8080 targetPort: 8080 nodePort: 30002 selector: app: pod-apollo-config-server-sh type: NodePort sessionAffinity: ClientIP
---
kind: ConfigMap apiVersion: v1 metadata: namespace: baseservice name: configmap-apollo-admin-server-sh data: application-github.properties: | spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8 spring.datasource.username = wikifx spring.datasource.password = Wikifx123 ---
kind: ConfigMap apiVersion: v1 metadata: namespace: baseservice name: configmap-apollo-config-server-sh data: application-github.properties: | spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8 spring.datasource.username = wikifx spring.datasource.password = Wikifx123 apollo.admin-service.url = http://service-apollo-admin-server-sh.baseservice:8090 apollo.config-service.url = http://service-apollo-config-server-sh.baseservice:8080 --- kind: Deployment apiVersion: apps/v1 metadata: namespace: baseservice name: deployment-apollo-admin-server-sh labels: app: deployment-apollo-admin-server-sh spec: replicas: 2 selector: matchLabels: app: pod-apollo-admin-server-sh strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app: pod-apollo-admin-server-sh spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - pod-apollo-admin-server-sh topologyKey: kubernetes.io/hostname volumes: - name: volume-configmap-apollo-admin-server-sh configMap: name: configmap-apollo-admin-server-sh containers: - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT securityContext: privileged: true imagePullPolicy: IfNotPresent name: container-apollo-admin-server-sh ports: - protocol: TCP containerPort: 8090 volumeMounts: - name: volume-configmap-apollo-admin-server-sh mountPath: /apollo-adminservice/config/application-github.properties subPath: application-github.properties env: - name: SPRING_PROFILES_ACTIVE value: "github,kubernetes" readinessProbe: tcpSocket: port: 8090 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8090 initialDelaySeconds: 120 periodSeconds: 10 dnsPolicy: ClusterFirst restartPolicy: Always --- kind: StatefulSet apiVersion: apps/v1 metadata: namespace: baseservice name: statefulset-apollo-config-server-sh labels: app: statefulset-apollo-config-server-sh spec: serviceName: service-apollo-meta-server-sh replicas: 2 selector: matchLabels: app: pod-apollo-config-server-sh updateStrategy: type: RollingUpdate template: metadata: labels: app: pod-apollo-config-server-sh spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - pod-apollo-config-server-sh topologyKey: kubernetes.io/hostname volumes: - name: volume-configmap-apollo-config-server-sh configMap: name: configmap-apollo-config-server-sh containers: - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT securityContext: privileged: true imagePullPolicy: IfNotPresent name: container-apollo-config-server-sh ports: - protocol: TCP containerPort: 8080 volumeMounts: - name: volume-configmap-apollo-config-server-sh mountPath: /apollo-configservice/config/application-github.properties subPath: application-github.properties env: - name: SPRING_PROFILES_ACTIVE value: "github,kubernetes" readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 120 periodSeconds: 10 dnsPolicy: ClusterFirst restartPolicy: Always
---
|
配置eureka地址(传统部署非常重要)
apolloconfig 在服务启动后会启动一个eureka服务注册中心,apolloadmin 和apolloconfig 都会注册其中,eurek的连接地址配置在数据库中配置或者在ConfigMap中配置:
格式为:http://<config server pod名>.<meta server 服务名>:<meta server端口号>/eureka/
- 方式一:通过Spring Boot文件 application-github.properties配置(推荐)
推荐此方式配置 eureka.service.url,因为可以通过ConfigMap的方式传入容器,无需再修改数据库的字段。如果该配置希望以数据库中为准,那么在 yaml 中直接删除该配置项即可。
- 方式二:修改数据表 ApolloConfigDB.ServerConfig
修改数据库表 ApolloConfigDB.ServerConfig的 eureka.service.url。
Apollo 1.7.0版本开始增加了基于Kubernetes原生服务发现的部署模式,由于不再使用内置的Eureka,所以在整体部署上有很大简化。
以上配置禁用了Apollo原生的Eureka,采用了Kubernetes原生服务发现
新加坡节点部署
同上,注意修改数据库连接串,以及对应节点的数据库账号和密码。
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226
| kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-mysql-for-apollo-sg-env labels: app: service-mysql-for-apollo-sg-env spec: type: ExternalName externalName: rm-xxxxxxov.mysql.singapore.rds.aliyuncs.com --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-admin-server-sg labels: app: service-apollo-admin-server-sg spec: ports: - protocol: TCP port: 8090 targetPort: 8090 nodePort: 30003 selector: app: pod-apollo-admin-server-sg type: NodePort sessionAffinity: ClientIP --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-meta-server-sg labels: app: service-apollo-meta-server-sg spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: pod-apollo-config-server-sg type: ClusterIP clusterIP: None sessionAffinity: ClientIP --- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-config-server-sg labels: app: service-apollo-config-server-sg spec: ports: - protocol: TCP port: 8080 targetPort: 8080 nodePort: 30002 selector: app: pod-apollo-config-server-sg type: NodePort sessionAffinity: ClientIP
--- kind: ConfigMap apiVersion: v1 metadata: namespace: baseservice name: configmap-apollo-admin-server-sg data: application-github.properties: | spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8 spring.datasource.username = wikifx spring.datasource.password = Wikifx2022 ---
kind: ConfigMap apiVersion: v1 metadata: namespace: baseservice name: configmap-apollo-config-server-sg data: application-github.properties: | spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8 spring.datasource.username = wikifx spring.datasource.password = Wikifx2022 apollo.admin-service.url = http://172.21.114.195:30003 apollo.config-service.url = http://172.21.114.195:30002 --- kind: Deployment apiVersion: apps/v1 metadata: namespace: baseservice name: deployment-apollo-admin-server-sg labels: app: deployment-apollo-admin-server-sg spec: replicas: 2 selector: matchLabels: app: pod-apollo-admin-server-sg strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app: pod-apollo-admin-server-sg spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - pod-apollo-admin-server-sg topologyKey: kubernetes.io/hostname volumes: - name: volume-configmap-apollo-admin-server-sg configMap: name: configmap-apollo-admin-server-sg containers: - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT securityContext: privileged: true imagePullPolicy: IfNotPresent name: container-apollo-admin-server-sg ports: - protocol: TCP containerPort: 8090 volumeMounts: - name: volume-configmap-apollo-admin-server-sg mountPath: /apollo-adminservice/config/application-github.properties subPath: application-github.properties env: - name: SPRING_PROFILES_ACTIVE value: "github,kubernetes" readinessProbe: tcpSocket: port: 8090 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8090 initialDelaySeconds: 120 periodSeconds: 10 dnsPolicy: ClusterFirst restartPolicy: Always --- kind: StatefulSet apiVersion: apps/v1 metadata: namespace: baseservice name: statefulset-apollo-config-server-sg labels: app: statefulset-apollo-config-server-sg spec: serviceName: service-apollo-meta-server-sg replicas: 2 selector: matchLabels: app: pod-apollo-config-server-sg updateStrategy: type: RollingUpdate template: metadata: labels: app: pod-apollo-config-server-sg spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - pod-apollo-config-server-sg topologyKey: kubernetes.io/hostname volumes: - name: volume-configmap-apollo-config-server-sg configMap: name: configmap-apollo-config-server-sg containers: - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT securityContext: privileged: true imagePullPolicy: IfNotPresent name: container-apollo-config-server-sg ports: - protocol: TCP containerPort: 8080 volumeMounts: - name: volume-configmap-apollo-config-server-sg mountPath: /apollo-configservice/config/application-github.properties subPath: application-github.properties env: - name: SPRING_PROFILES_ACTIVE value: "github,kubernetes" readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 120 periodSeconds: 10 dnsPolicy: ClusterFirst restartPolicy: Always
|
指定Meta Server返回的apollo-configservice和apollo-adminservice地址。(多数据中心k8s部署必要)
1 2 3 4 5 6 7
| # configmap for apollo-config-server-sg kind: ConfigMap apiVersion: v1 ... apollo.admin-service.url = http://172.21.114.195:30003 apollo.config-service.url = http://172.21.114.195:30002 #这里的地址为线上K8S集群前端的SLB地址
|
以上配置禁用了Apollo原生的Eureka,采用了Kubernetes原生服务发现
ApolloPortal部署
创建授权mysql库
初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloportaldb.sql
开始部署
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
| kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-mysql-for-portal-server labels: app: service-mysql-for-portal-server spec: type: ExternalName externalName: rm-uf6xxxxxxw90110.mysql.rds.aliyuncs.com
---
kind: ConfigMap apiVersion: v1 metadata: namespace: baseservice name: configmap-apollo-portal-server data: application-github.properties: | spring.datasource.url = jdbc:mysql://service-mysql-for-portal-server.baseservice:3306/apolloportaldb?characterEncoding=utf8 # mysql username spring.datasource.username = wikifx # mysql password spring.datasource.password = Wikifx123 # enabled environments apollo.portal.envs = sh,sg apollo-env.properties: | sh.meta=http://service-apollo-config-server-sh.baseservice:8080 sg.meta=http://172.21.114.195:30002
--- kind: Service apiVersion: v1 metadata: namespace: baseservice name: service-apollo-portal-server labels: app: service-apollo-portal-server spec: ports: - protocol: TCP port: 8070 targetPort: 8070 nodePort: 30001 selector: app: pod-apollo-portal-server type: NodePort sessionAffinity: ClientIP
--- kind: Deployment apiVersion: apps/v1 metadata: namespace: baseservice name: deployment-apollo-portal-server labels: app: deployment-apollo-portal-server spec: replicas: 2 selector: matchLabels: app: pod-apollo-portal-server strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app: pod-apollo-portal-server spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - pod-apollo-portal-server topologyKey: kubernetes.io/hostname volumes: - name: volume-configmap-apollo-portal-server configMap: name: configmap-apollo-portal-server items: - key: application-github.properties path: application-github.properties - key: apollo-env.properties path: apollo-env.properties containers: - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-portal-1.9.0-SNAPSHOT securityContext: privileged: true imagePullPolicy: IfNotPresent name: container-apollo-portal-server ports: - protocol: TCP containerPort: 8070 volumeMounts: - name: volume-configmap-apollo-portal-server mountPath: /apollo-portal/config/application-github.properties subPath: application-github.properties - name: volume-configmap-apollo-portal-server mountPath: /apollo-portal/config/apollo-env.properties subPath: apollo-env.properties env: - name: APOLLO_PORTAL_SERVICE_NAME value: "service-apollo-portal-server.baseservice" readinessProbe: tcpSocket: port: 8070 initialDelaySeconds: 10 periodSeconds: 5 livenessProbe: tcpSocket: port: 8070 initialDelaySeconds: 120 periodSeconds: 15 dnsPolicy: ClusterFirst restartPolicy: Always
|
检查
Apollo In Kubernetes(多数据中心)
转载前请阅读本站 版权协议,文章著作权归 饼铛 所有,转载请注明出处。