memcg相关接口cgroup v2, 内核版本6.19memory.max memory.high memory.low memory.min示意图max上限硬限制超过了就会OOMhigh上限软限制尽量限制low下线软保护min: 下线硬保护例如一个cgroup设置max10G high是8Glow是4Gmin是2G这个代表什么意思呢如果cgroup中使用的内存超过了10G那么会发生OOM会挑一个内存占用多的进程将其杀死使内存一定限制在10G以下。如果内存使用在8-10G直接那么在用户态系统调用返回的时候会触发一次内存回收尽量将内存控制在8G以内但是这个不是强限制的只能说尽可能。如果使用的内存在2-4G之间那么如果触发了内存回收父节点memory.reclaim/kswapd那么回收过程中是尽量保护这个cgroup也就是前面内存扫描的时候会跳过这个cgroup。尽量去回收其他cgroup内存。但是如果尝试了其他cgroup没法达到回收期望值还是会回收这个cgroup的内存所以只能说尽量保护。如果使用低于2G是硬保护。就算其他cgroup没法达到回收期望值只能OOM也不能回收这个cgroup的内存达到一个硬保护的效果。注意cgroup级别的水位线与全局的水位线是反过来的全局水位线是指reserve多少cgroup水位是指用了多少。************-memory.max * * * * * *-memory.high * * * * * * * mem * * * * *-memory.low * * * * * *-memory.min * * ************这里提个问题对于low min限制多个children的和大于父节点那么限制效果是什么实现机制high 触发内存回收每次发生pagefault的时候就会使用新的物理内存就会触发memcg计数。memcg计数入口是try_charge_memcg。注意malloc是申请虚拟内存这个时候物理内存没有被使用。只有写内存地址发生pagefault才会触发计数触发超过high就是执行一个回收由内核直接回收/返回用户态的时候回收不管是哪种对用户态都有延迟。但是不保证成功且只try_to_free_mem_cgroup_pages一次。try_charge_memcg // 这个函数比较复杂我们只关注high的情况 ... mem_high page_counter_read(memcg-memory) READ_ONCE(memcg-memory.high); // 如果使用量超过high swap_high page_counter_read(memcg-swap) READ_ONCE(memcg-swap.high); // 如果使用量超过swap_high if (!in_task()) { // 处于中断上下文中不能触发回收因为太耗时了所以起了一个kworker if (mem_high) { schedule_work(memcg-high_work); break; } continue; } if (mem_high || swap_high) { current-memcg_nr_pages_over_high batch; set_notify_resume(current); // 设置通知返回用户态时候会触发回收设置了标志位TIF_NOTIFY_RESUME break; } if (current-memcg_nr_pages_over_high MEMCG_CHARGE_BATCH !(current-flags PF_MEMALLOC) gfpflags_allow_blocking(gfp_mask)) __mem_cgroup_handle_over_high(gfp_mask); // 直接同步回收防止继续占用内存如果这里回收成功返回用户态可以不回收。返回到用户态出口exit_to_user_mode_loop,检测到有TIF_NOTIFY_RESUME标志位触发high回收流程exit_to_user_mode_loop if (ti_work _TIF_NOTIFY_RESUME) // 有标志位 resume_user_mode_work(regs); mem_cgroup_handle_over_high(GFP_KERNEL); reclaim_high try_to_free_mem_cgroup_pages // 尝试回收内存到highlow min这两个实现的逻辑还是挺简单的// 所有对内存回收最终都是到这个入口,扫描所有zone进行回收 do_try_to_free_pageszonelist sc // sc-memcg_low_reclaim 0; 首次扫描还没重试不会reclaim low以下的cgroup retry: shrink_zones shrink_node shrink_node_memcgs // for 循环每个cgroup if (mem_cgroup_below_min(target_memcg, memcg)) { // 低于min的直接跳过 /* * Hard protection. * If there is no reclaimable memory, OOM. */ continue; } else if (mem_cgroup_below_low(target_memcg, memcg)) { // 低于low /* * Soft protection. * Respect the protection only as long as * there is an unprotected supply * of reclaimable memory from other cgroups. */ if (!sc-memcg_low_reclaim) { // 如果不回收low直接跳过但是设置了memcg_low_skipped说明已经跳过一次下次来的回收了。 sc-memcg_low_skipped 1; continue; } memcg_memory_event(memcg, MEMCG_LOW); } ... 如果回收成功就返回了回收不成功尝试更狠的回收比如全扫描比如回收low以下的值 if (!sc-memcg_full_walk) { sc-priority initial_priority; sc-memcg_full_walk 1; goto retry; } /* Untapped cgroup reserves? Dont OOM, retry. */ if (sc-memcg_low_skipped) { sc-priority initial_priority; sc-force_deactivate 0; sc-memcg_low_reclaim 1; sc-memcg_low_skipped 0; goto retry; }从中可以看出开始时并不会去扫描low以下的cgroup尽量保护他们但是实在回收不上来那没办法了还是得回收。所以是尽可能保护。但是对于min不管怎么都不能回收它爱咋咋的反正就是不能回收。回答问题这个是children小于父节点的elow等于min(usage, low)* Example * Lets have global and As reclaim in parallel: * | * A (low2G, usage 3G, max 3G, children_low_usage 1.5G) * |\ * | C (low 1G, usage 2.5G) * B (low 1G, usage 0.5G) * * For the global reclaim * A.elow A.low * B.elow min(B.usage, B.low) because children_low_usage A.elow * C.elow min(C.usage, C.low) * * With the effective values resetting we have A reclaim * A.elow 0 * B.elow B.low * C.elow C.low如果大于呢* Example * Lets have global and As reclaim in parallel: * | * A (low2G, usage 3G, max 3G, children_low_usage 1.5G) * |\ * | C (low 2G, usage 2.5G) * B (low 1G, usage 1G)通过函数计算page_counter_calculate_protection if (parent root) { // root就是等于设置值 counter-emin READ_ONCE(counter-min); counter-elow READ_ONCE(counter-low); return; } effective_protection if (siblings_protected parent_effective) return protected * parent_effective / siblings_protected; // 按比例分 // 即B elow:2*1/3G, A elow:2*2/3G有效值计算static unsigned long effective_protection(unsigned long usage, unsigned long parent_usage, unsigned long setting, unsigned long parent_effective, unsigned long siblings_protected, bool recursive_protection) { unsigned long protected; unsigned long ep; protected min(usage, setting); /* * If all cgroups at this level combined claim and use more * protection than what the parent affords them, distribute * shares in proportion to utilization. * * We are using actual utilization rather than the statically * claimed protection in order to be work-conserving: claimed * but unused protection is available to siblings that would * otherwise get a smaller chunk than what they claimed. */ if (siblings_protected parent_effective) return protected * parent_effective / siblings_protected; /* * Ok, utilized protection of all children is within what the * parent affords them, so we know whatever this child claims * and utilizes is effectively protected. * * If there is unprotected usage beyond this value, reclaim * will apply pressure in proportion to that amount. * * If there is unutilized protection, the cgroup will be fully * shielded from reclaim, but we do return a smaller value for * protection than what the group could enjoy in theory. This * is okay. With the overcommit distribution above, effective * protection is always dependent on how memory is actually * consumed among the siblings anyway. */ ep protected; /* * If the children arent claiming (all of) the protection * afforded to them by the parent, distribute the remainder in * proportion to the (unprotected) memory of each cgroup. That * way, cgroups that arent explicitly prioritized wrt each * other compete freely over the allowance, but they are * collectively protected from neighboring trees. * * Were using unprotected memory for the weight so that if * some cgroups DO claim explicit protection, we dont protect * the same bytes twice. * * Check both usage and parent_usage against the respective * protected values. One should imply the other, but they * arent read atomically - make sure the division is sane. */ if (!recursive_protection) return ep; if (parent_effective siblings_protected parent_usage siblings_protected usage protected) { unsigned long unclaimed; unclaimed parent_effective - siblings_protected; unclaimed * usage - protected; unclaimed / parent_usage - siblings_protected; ep unclaimed; } return ep;举例说明再来一个 undercommit recursive_protection 的例子 root └── parent (memory.low 1000MB, usage 1500MB) ├── child_A (memory.low 100MB, usage 600MB) └── child_B (memory.low 100MB, usage 900MB) parent.elow 1000MB children_low_usage: child_A: min(600, 100) 100 child_B: min(900, 100) 100 siblings_protected 200 200 1000parent_effectiveundercommit有 800MB 剩余保护没人认领。 先确认进入 recursive_protection 分支的条件 - parent_effective (1000) siblings_protected (200) ✓ - parent_usage (1500) siblings_protected (200) ✓ 计算浮动保护分配 - usage protected → 各 child 都满足 ✓ 计算浮动保护分配意思是把还 ep 剩余阈值*没被保护的 / 没有被保护总额 unclaimed parent_effective - siblings_protected 1000 - 200 800 child_A: ep protected 100 unclaimed_share 800 * (600 - 100) / (1500 - 200) 800 * 500 / 1300 307MB child_A.elow 100 307 407MB child_B: ep protected 100 unclaimed_share 800 * (900 - 100) / (1500 - 200) 800 * 800 / 1300 492MB child_B.elow 100 492 592MB 两个孩子只各声明了 100MB 保护但父亲有 1000MB 的有效保护。剩余的 800MB 按各自未受保护的用量比例分配下去让整个子树作为整体享受父亲的保护免受外部回收压力。