使用场景参考官网文档部署流程本文分别部署默认 IPIP 模式与 IPIP CrossSubnet 模式分别在请求同网段、不同网段时进行抓包对比1.通过脚本快速生成 IPIP 默认模式#!/bin/bash set -v # 1. Prepare NoCNI environment cat EOF | HTTP_PROXY HTTPS_PROXY http_proxy https_proxy kind create cluster --namecalico-ipip --imageburlyluo/kindest:v1.27.3 --config- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true podSubnet: 10.244.0.0/16 nodes: - role: control-plane kubeadmConfigPatches: - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.5.10 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.5.11 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.8.10 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.8.11 EOF # 2. Remove taints controller_node_ipkubectl get node -o wide --no-headers | grep -E control-plane|bpf1 | awk -F {print $6} kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule- kubectl get nodes -o wide ./2-setup-clab.sh # 3. Collect startup message controller_node_name$(kubectl get nodes -o jsonpath{range .items[*]}{.metadata.name}{\n}{end} | grep control-plane) if [ -n $controller_node_name ]; then timeout 1 docker exec -t $controller_node_name bash -c cat EOF /root/monitor_startup.sh #!/bin/bash ip -ts monitor all /root/startup_monitor.txt 21 EOF chmod x /root/monitor_startup.sh /root/monitor_startup.sh else echo No such controller_node! fi # 4. Install CNI[Calico v3.23.2] kubectl apply -f calico.yaml其中 2-setup-clab.sh 的作用是通过 containerlab 创建四个容器给他们设置 IP 后分别与 kind 创建的四个容器共享网络命名空间这样 k8s 集群就能使用 kind 参数指定的 node-ip 了#!/bin/bash set -v for br in br-pool0 br-pool1; do ip link set $br down /dev/null 21 ip link delete $br ip link add $br type bridge ip link set $br up done cat EOF clab.yaml | containerlab destroy -t clab.yaml --cleanup - name: calico-ipip topology: nodes: gw0: kind: linux image: hub.deepflow.yunshan.net/network-demo/vyos:1.4.9 cmd: /sbin/init binds: - /lib/modules:/lib/modules - ./startup-conf/gw0-boot.cfg:/opt/vyatta/etc/config/config.boot br-pool0: kind: bridge br-pool1: kind: bridge server1: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-control-plane exec: - ip addr add 10.1.5.10/24 dev net0 - ip route replace default via 10.1.5.1 server2: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker exec: - ip addr add 10.1.5.11/24 dev net0 - ip route replace default via 10.1.5.1 server3: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker2 exec: - ip addr add 10.1.8.10/24 dev net0 - ip route replace default via 10.1.8.1 server4: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker3 exec: - ip addr add 10.1.8.11/24 dev net0 - ip route replace default via 10.1.8.1 links: - endpoints: [br-pool0:br-pool0-net0, server1:net0] mtu: 1500 - endpoints: [br-pool0:br-pool0-net1, server2:net0] mtu: 1500 - endpoints: [br-pool1:br-pool1-net0, server3:net0] mtu: 1500 - endpoints: [br-pool1:br-pool1-net1, server4:net0] mtu: 1500 - endpoints: [gw0:eth1, br-pool0:br-pool0-net2] mtu: 1500 - endpoints: [gw0:eth2, br-pool1:br-pool1-net2] mtu: 1500 EOFgw0 中 startup-conf/gw0-boot.cfg 文件的作用就是让 10.1.5.0/24 和 10.1.8.0/24 两个子网能互通两个子网的默认网关都在 gw0 上gw0 直接转发就行interfaces { ethernet eth1 { address 10.1.5.1/24 duplex auto speed auto } ethernet eth2 { address 10.1.8.1/24 duplex auto speed auto } loopback lo { } } nat { source { rule 100 { outbound-interface { name eth0 } source { address 10.1.0.0/16 } translation { address masquerade } } } } system { config-management { commit-revisions 100 } console { device ttyS0 { speed 9600 } } host-name gw0 login { user vyos { authentication { encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/ plaintext-password } } } time-zone UTC }## calico yaml # Auto-detect the BGP IP address. - name: IP value: autodetect # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: Always # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: Never # Enable or Disable VXLAN on the default IPv6 IP pool. - name: CALICO_IPV6POOL_VXLAN value: Never2.通过脚本快速生成 IPIP CrossSubnet 模式其余部署脚本一致仅在 calico CALICO_IPV4POOL_IPIP 模式中有差异## calico yaml # Auto-detect the BGP IP address. - name: IP value: autodetect # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: CrossSubnet # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: Never # Enable or Disable VXLAN on the default IPv6 IP pool. - name: CALICO_IPV6POOL_VXLAN value: Never创建测试 Pod本质是 Nginx用于后续请求抓包使用apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: nginx name: pod spec: replicas: 4 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: burlyluo/nettool:latest name: nettoolbox env: - name: NETTOOL_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: privileged: true affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: nginx topologyKey: kubernetes.io/hostname查询部署结果1.查询 IPIP 默认模式部署结果rootnetwork-demo:~# docker ps --format {{.Names}} clab-calico-ipip-server2 clab-calico-ipip-server4 clab-calico-ipip-server1 clab-calico-ipip-server3 clab-calico-ipip-gw0 calico-ipip-worker calico-ipip-worker2 calico-ipip-control-plane calico-ipip-worker3在主机上看到创建的 br-pool0-net0 网卡与 containerlab 创建的容器中 net0 网卡对应。在 kind 生成的 docker 容器中也能看到相同的网卡说明已经共享了同一个网络空间rootnetwork-demo:~# ip -d link show br-pool0-net0 198: br-pool0-net0if197: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue master br-pool0 state UP mode DEFAULT group default link/ether aa:c1:ab:1c:c9:1c brd ff:ff:ff:ff:ff:ff link-netns clab-calico-ipip-server1 promiscuity 1 allmulti 1 minmtu 68 maxmtu 65535 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.c6:58:98:9d:5f:ea designated_root 8000.c6:58:98:9d:5f:ea hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on bcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off locked off addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 rootnetwork-demo:~# docker exec -it clab-calico-ipip-server1 ip -d link show net0 197: net0if198: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:bd:45:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535 veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 rootnetwork-demo:~# docker exec -it calico-ipip-control-plane ip -d link show net0 197: net0if198: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:bd:45:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535 veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535rootnetwork-demo:~# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system calico-kube-controllers 1/1 Running 0 16m 10.244.51.196 calico-ipip-control-plane kube-system calico-node-64f6p 1/1 Running 0 16m 10.1.5.10 calico-ipip-control-plane kube-system calico-node-p4ks7 1/1 Running 0 16m 10.1.5.11 calico-ipip-worker kube-system calico-node-pjbc7 1/1 Running 0 16m 10.1.8.11 calico-ipip-worker3 kube-system calico-node-r6rk2 1/1 Running 0 16m 10.1.8.10 calico-ipip-worker2 kube-system coredns-5d78c9869d-jx4lx 1/1 Running 0 17m 10.244.51.194 calico-ipip-control-plane kube-system coredns-5d78c9869d-mrf2d 1/1 Running 0 17m 10.244.51.195 calico-ipip-control-plane kube-system etcd-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-apiserver-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-controller-manager-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-proxy-4svbw 1/1 Running 0 17m 10.1.8.10 calico-ipip-worker2 kube-system kube-proxy-4zw9q 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-proxy-5nnfn 1/1 Running 0 17m 10.1.8.11 calico-ipip-worker3 kube-system kube-proxy-b69xp 1/1 Running 0 17m 10.1.5.11 calico-ipip-worker kube-system kube-scheduler-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane rootnetwork-demo:~# kubectl describe pods -n kube-system calico-node-64f6p | grep CALICO_IPV4POOL CALICO_IPV4POOL_IPIP: Always