Apollo In Kubernetes（多数据中心）

2022年4月4日OPSOPSKubernetesApollo--

本文介绍分布式配置管理中心Apollo（阿波罗）多节点k8s部署

参考文档:

  • https://github.com/apolloconfig/apollo
  • apollo分布式部署指南
  • https://github.com/apolloconfig/apollo/tree/master/scripts/apollo-on-kubernetes
  • https://www.apolloconfig.com/#/zh/design/apollo-design

Apollo介绍

Apollo（阿波罗）是一款可靠的分布式配置管理中心，诞生于携程框架研发部，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。

  • 服务端基于Spring Boot和Spring Cloud开发，打包后可以直接运行，不需要额外安装Tomcat等应用容器。
  • Java客户端不依赖任何框架，能够运行于所有Java运行时环境，同时对Spring/Spring Boot环境也有较好的支持。
  • .Net客户端不依赖任何框架，能够运行于所有.Net运行时环境。

这里不再赘述，详细参见官方文档。

Apollo各组件介绍

  • apollo-portal 提供管理界面 (部署一份k8s)

    • 提供Web界面供用户管理配置
    • 通过Meta Server获取Admin Service服务列表（IP+Port），通过IP+Port访问服务
    • 在Portal侧做load balance、错误重试

  • apollo-adminservice (部署各节点k8s)

    • 提供配置管理接口
    • 提供配置修改、发布等接口
    • 接口服务对象为apollo-portal

  • apollo-configservice (部署各节点k8s)

    • 提供配置获取接口
    • 提供配置更新推送接口
    • 接口服务对象为Apollo客户端

  • Meta Server (随apollo-configservice部署各节点k8s)

    • Portal通过域名访问Meta Server获取Admin Service服务列表（IP+Port）
    • Client通过域名访问Meta Server获取Config Service服务列表（IP+Port）
    • Meta Server从Eureka获取Config Service和Admin Service的服务信息，相当于是一个Eureka Client
    • 增设一个Meta Server的角色主要是为了封装服务发现的细节，对Portal和Client而言，永远通过一个Http接口获取Admin Service和Config Service的服务信息，而不需要关心背后实际的服务注册和发现组件
    • Meta Server只是一个逻辑角色，在部署时和Config Service是在一个JVM进程中的，所以IP、端口和Config Service一致

  • Eureka (随apollo-configservice部署各节点k8s,多数据中心分布式部署时需要弃用)

    • 基于Eureka和Spring Cloud Netflix提供服务注册和发现
    • Config Service和Admin Service会向Eureka注册服务，并保持心跳
    • 目前Eureka在部署时和Config Service是在一个JVM进程中的（通过Spring Cloud Netflix）

  • Client（使用服务的应用程序）

    • Apollo提供的客户端程序，为应用提供配置获取、实时更新等功能
    • 通过Meta Server获取Config Service服务列表（IP+Port），通过IP+Port访问服务
    • 在Client侧做load balance、错误重试

环境规划

各节点服务规划

环境地址需要部署的服务
SHservice-apollo-portal-server.baseservice:8070:30001
http://47.xxx.xx.39:30001/system_info.html

service-apollo-config-server-sh.baseservice:8080:30002
service-apollo-admin-server-sh.baseservice:8090:30003		apollo-portal

apollo-configservice
apollo-adminservice
SGservice-apollo-config-server-sg.baseservice:8080:30002
172.21.114.195:30002

service-apollo-admin-server-sg.baseservice:8090:30003
172.21.114.195:30003		apollo-configservice

apollo-adminservice
LDN(待加入)localhost:8080
localhost:8091		apollo-configservice
apollo-adminservice

各节点数据库规划

环境地址库名(字符集)
SHrm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306apolloportaldb(UTF8MB4)
SHrm-uxxxxxmo7w90110.mysql.rds.aliyuncs.com:3306apolloconfigdb(UTF8MB4)
SGrm-t4xxxxxx91nov.mysql.singapore.rds.aliyuncs.com:3306apolloconfigdb(UTF8MB4)
LDN(待加入)xxxx:3306apolloconfigdb(UTF8MB4)

上海节点部署

创建授权mysql库

createdb 初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql initmysql

部署服务

# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-mysql-for-apollo-sh-env
  labels:
    app: service-mysql-for-apollo-sh-env
spec:
  type: ExternalName
  externalName: rm-xxxxxhmo7w90110.mysql.rds.aliyuncs.com
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-admin-server-sh
  labels:
    app: service-apollo-admin-server-sh
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
      nodePort: 30003
  selector:
    app: pod-apollo-admin-server-sh
  type: NodePort
  sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-meta-server-sh
  labels:
    app: service-apollo-meta-server-sh
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-sh
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-config-server-sh
  labels:
    app: service-apollo-config-server-sh
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30002
  selector:
    app: pod-apollo-config-server-sh 
  type: NodePort
  sessionAffinity: ClientIP

---
# configmap for apollo-admin-server-sh
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: baseservice
  name: configmap-apollo-admin-server-sh
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
    spring.datasource.username = wikifx
    spring.datasource.password = Wikifx123
---
# configmap for apollo-config-server-sh
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: baseservice
  name: configmap-apollo-config-server-sh
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sh-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
    spring.datasource.username = wikifx
    spring.datasource.password = Wikifx123
    apollo.admin-service.url = http://service-apollo-admin-server-sh.baseservice:8090
    apollo.config-service.url = http://service-apollo-config-server-sh.baseservice:8080
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: baseservice
  name: deployment-apollo-admin-server-sh
  labels:
    app: deployment-apollo-admin-server-sh
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pod-apollo-admin-server-sh
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-sh
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-sh
              topologyKey: kubernetes.io/hostname
      volumes:
        - name: volume-configmap-apollo-admin-server-sh
          configMap:
            name: configmap-apollo-admin-server-sh
      containers:
        - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-sh
          ports:
            - protocol: TCP
              containerPort: 8090
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-sh
              mountPath: /apollo-adminservice/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "github,kubernetes"
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10  
      dnsPolicy: ClusterFirst
      restartPolicy: Always
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: baseservice
  name: statefulset-apollo-config-server-sh
  labels:
    app: statefulset-apollo-config-server-sh
spec:
  serviceName: service-apollo-meta-server-sh
  replicas: 2
  selector:
    matchLabels:
      app: pod-apollo-config-server-sh
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-sh
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-sh
              topologyKey: kubernetes.io/hostname
      volumes:
        - name: volume-configmap-apollo-config-server-sh
          configMap:
            name: configmap-apollo-config-server-sh
      containers:
        - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-config-server-sh
          ports:
            - protocol: TCP
              containerPort: 8080
          volumeMounts:
            - name: volume-configmap-apollo-config-server-sh
              mountPath: /apollo-configservice/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "github,kubernetes"
            # 添加环境变量禁用apollo原生Eureka
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds:  120
            periodSeconds: 10
      dnsPolicy: ClusterFirst
      restartPolicy: Always
# kubectl apply -f allinone.yaml
---

配置eureka地址（传统部署非常重要） apolloconfig 在服务启动后会启动一个eureka服务注册中心，apolloadmin 和apolloconfig 都会注册其中，eurek的连接地址配置在数据库中配置或者在ConfigMap中配置： 格式为：http://<config server pod名>.<meta server 服务名>:<meta server端口号>/eureka/

  • 方式一：通过Spring Boot文件 application-github.properties配置（推荐） 推荐此方式配置 eureka.service.url，因为可以通过ConfigMap的方式传入容器，无需再修改数据库的字段。如果该配置希望以数据库中为准，那么在 yaml 中直接删除该配置项即可。
  • 方式二：修改数据表 ApolloConfigDB.ServerConfig 修改数据库表 ApolloConfigDB.ServerConfig的 eureka.service.url。 install

Apollo 1.7.0版本开始增加了基于Kubernetes原生服务发现的部署模式，由于不再使用内置的Eureka，所以在整体部署上有很大简化。

以上配置禁用了Apollo原生的Eureka，采用了Kubernetes原生服务发现 ck

新加坡节点部署

同上，注意修改数据库连接串，以及对应节点的数据库账号和密码。 初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloconfigdb.sql

kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-mysql-for-apollo-sg-env
  labels:
    app: service-mysql-for-apollo-sg-env
spec:
  type: ExternalName
  externalName: rm-xxxxxxov.mysql.singapore.rds.aliyuncs.com
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-admin-server-sg
  labels:
    app: service-apollo-admin-server-sg
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
      nodePort: 30003
  selector:
    app: pod-apollo-admin-server-sg
  type: NodePort
  sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-meta-server-sg
  labels:
    app: service-apollo-meta-server-sg
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-sg
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-config-server-sg
  labels:
    app: service-apollo-config-server-sg
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30002
  selector:
    app: pod-apollo-config-server-sg 
  type: NodePort
  sessionAffinity: ClientIP

---
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: baseservice
  name: configmap-apollo-admin-server-sg
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
    spring.datasource.username = wikifx
    spring.datasource.password = Wikifx2022
---
# configmap for apollo-config-server-sg
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: baseservice
  name: configmap-apollo-config-server-sg
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-sg-env.baseservice:3306/apolloconfigdb?characterEncoding=utf8
    spring.datasource.username = wikifx
    spring.datasource.password = Wikifx2022
    apollo.admin-service.url = http://172.21.114.195:30003
    apollo.config-service.url = http://172.21.114.195:30002
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: baseservice
  name: deployment-apollo-admin-server-sg
  labels:
    app: deployment-apollo-admin-server-sg
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pod-apollo-admin-server-sg
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-sg
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-sg
              topologyKey: kubernetes.io/hostname
      volumes:
        - name: volume-configmap-apollo-admin-server-sg
          configMap:
            name: configmap-apollo-admin-server-sg
      containers:
        - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-adminservice-1.9.0-SNAPSHOT
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-sg
          ports:
            - protocol: TCP
              containerPort: 8090
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-sg
              mountPath: /apollo-adminservice/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "github,kubernetes"
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10  
      dnsPolicy: ClusterFirst
      restartPolicy: Always
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: baseservice
  name: statefulset-apollo-config-server-sg
  labels:
    app: statefulset-apollo-config-server-sg
spec:
  serviceName: service-apollo-meta-server-sg
  replicas: 2
  selector:
    matchLabels:
      app: pod-apollo-config-server-sg
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-sg
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-sg
              topologyKey: kubernetes.io/hostname
      volumes:
        - name: volume-configmap-apollo-config-server-sg
          configMap:
            name: configmap-apollo-config-server-sg
      containers:
        - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-configservice-1.9.0-SNAPSHOT
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-config-server-sg
          ports:
            - protocol: TCP
              containerPort: 8080
          volumeMounts:
            - name: volume-configmap-apollo-config-server-sg
              mountPath: /apollo-configservice/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "github,kubernetes"
            # 添加环境变量禁用apollo原生Eureka
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds:  120
            periodSeconds: 10
      dnsPolicy: ClusterFirst
      restartPolicy: Always
# kubectl apply -f allinone.yaml

指定Meta Server返回的apollo-configservice和apollo-adminservice地址。(多数据中心k8s部署必要)

# configmap for apollo-config-server-sg
kind: ConfigMap
apiVersion: v1
...
    apollo.admin-service.url = http://172.21.114.195:30003
    apollo.config-service.url = http://172.21.114.195:30002
#这里的地址为线上K8S集群前端的SLB地址

以上配置禁用了Apollo原生的Eureka，采用了Kubernetes原生服务发现 ck

ApolloPortal部署

创建授权mysql库

create 初始化sql: https://github.com/apolloconfig/apollo/blob/master/scripts/sql/apolloportaldb.sql init

开始部署

# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-mysql-for-portal-server
  labels:
    app: service-mysql-for-portal-server
spec:
  type: ExternalName
  externalName: rm-uf6xxxxxxw90110.mysql.rds.aliyuncs.com

---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: baseservice
  name: configmap-apollo-portal-server
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://service-mysql-for-portal-server.baseservice:3306/apolloportaldb?characterEncoding=utf8
    # mysql username
    spring.datasource.username = wikifx
    # mysql password
    spring.datasource.password = Wikifx123
    # enabled environments
    apollo.portal.envs = sh,sg
  apollo-env.properties: |
    sh.meta=http://service-apollo-config-server-sh.baseservice:8080
    sg.meta=http://172.21.114.195:30002

---
kind: Service
apiVersion: v1
metadata:
  namespace: baseservice
  name: service-apollo-portal-server
  labels:
    app: service-apollo-portal-server
spec:
  ports:
    - protocol: TCP
      port: 8070
      targetPort: 8070
      nodePort: 30001
  selector:
    app: pod-apollo-portal-server
  type: NodePort
  # portal session 保持
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: baseservice
  name: deployment-apollo-portal-server
  labels:
    app: deployment-apollo-portal-server
spec:
  # 2 个实例
  replicas: 2
  selector:
    matchLabels:
      app: pod-apollo-portal-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-portal-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-portal-server
              topologyKey: kubernetes.io/hostname
      
      volumes:
        - name: volume-configmap-apollo-portal-server
          configMap:
            name: configmap-apollo-portal-server
            items:
              - key: application-github.properties
                path: application-github.properties
              - key: apollo-env.properties
                path: apollo-env.properties
      containers:
        - image: registry.cn-shanghai.aliyuncs.com/wikifx/base:apollo-portal-1.9.0-SNAPSHOT
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-portal-server
          ports:
            - protocol: TCP
              containerPort: 8070
          volumeMounts:
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal/config/application-github.properties
              subPath: application-github.properties
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal/config/apollo-env.properties
              subPath: apollo-env.properties
          env:
            - name: APOLLO_PORTAL_SERVICE_NAME
              value: "service-apollo-portal-server.baseservice"  
          readinessProbe:
            tcpSocket:
              port: 8070
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 8070
            # 120s 内, server 未启动则重启 container
            initialDelaySeconds: 120
            periodSeconds: 15
      dnsPolicy: ClusterFirst
      restartPolicy: Always

# kubectl apply -f allinone.yaml

检查

ckckck

Apollo In Kubernetes（多数据中心）
转载前请阅读本站 版权协议，文章著作权归 饼铛 所有，转载请注明出处。