ClusterMonitor Cluster Heartbeat Detection Function Description【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcommFunction DescriptionClusterMonitoris the cluster-level heartbeat monitoring module of HCCL (Huawei Collective Communication Library) under thecoll_communicator_mgr/dfxpath, mainly used forcontinuous detection of reachability and liveness of all participating nodes and abnormal propagation after the establishment of a collective communication domain (Communicator).Core capabilities include:Node UID identification: Each Rank is encoded as a uniqueClusterUIDType(up to 2048 bytes) vianetInstId localId, used for cross-layer (device / server / pod / superPod) node identification.Dual Ring link establishment: Within eachnetLayerplane, nodes are sorted bylocalIdornetInstIdto form a Ring topology; each Rank establishes heartbeat socket connections with the left and right neighbors on the ring (only one neighbor when there are only 2 nodes).Async link establishment: Each remote UID starts an independent threadCreateLinkWithRemotePonitforSocketCreate status polling, controlled byHCCL_CONNECT_TIMEOUT.Periodic heartbeat send/receive: The backgroundMonitorThreadtraverses all sockets atBROADCAST_INTERVALintervals, firstSocketSendNbto send heartbeats thenSocketRecvNbto receive heartbeats;lostNumaccumulates everyHEARTBEAT_COUNTcycles, and whenHCCL_LOST_THRESHOLD(30s) is reached, the status is determined asLOST.Abnormal status propagation: NodeLOSTorCQE_ERRstatus is broadcast to other neighbors via the Ring link (SetStatus→errRankQueue_→ProcessExceptionEvent→SendFrame).Error information reporting: Through theGetCqeErrInfoFromTaskExceptioncallback registered via__attribute__((constructor)), CQE errors from AICPU/CCU tasks are collected intocqeErrInfo_, and triggered viaSetStatus(..., CLUSTER_MONITOR_CQE_ERR, true)to broadcast; the query interfaceGetErrStatusVecFromCluserMonitorformats error descriptions by priority (CQE_ERR LOST) and returns them to the upper layer.The module belongs to the DFX (Design For X) category and is a key component of HCCL for providing network disconnection / peer CoreDump observability at the cluster level.Directory Descriptioncluster_monitor/ ├── CMakeLists.txt # Build script, only adds cluster_monitor.cc to the hcomm target ├── cluster_monitor.h # Class and data structure definitions (ClusterMonitor, Frame, SockCtx, UID, etc.) └── cluster_monitor.cc # Module implementation, including heartbeat main thread, async link establishment, frame send/receive, exception handlingDirectory features:Located incoll_communicator_mgr/dfx/, belongs to DFX monitoring functionality;Module autonomous: singleton (GetInstance(u32 deviceId)), does not depend on other files in the directory;External dependencies:ring_buffer,reference_map,hcclCommSocket,hccl_communicator,hcclCommDfx,coll_comm,log,comm_addr_logger, etc.Flow Description (Mermaid Sequence Diagram)Overall Registration / Link Establishment / Heartbeat / Deregistration Main FlowCQE Exception Reporting and Broadcast SequenceData Description (Mermaid Class Diagram)Interface DescriptionPublic API (External)InterfaceDescriptionstatic ClusterMonitor GetInstance(u32 deviceId)Get module singleton by device (held indirectly viaCollCommMgr).HcclResult RegisterToClusterMonitor(HcclComm comm)Register a communicator: build UID context, calculate Ring connection set, push intoclusterLinkContext_waiting for background link establishment; first registration startsMonitorThread.HcclResult UnRegisterToClusterMonitor(hccl::CollComm* collComm)Deregister a communicator: clear reference counts for that commId inclusterLinkContext_,commIdMap_,monitorLinkStatusMap_,uid2SocketRefMap_; triggersDeInitwhen the last commId is deregistered.void GetCqeErrInfoFromTaskException(u32 remoteLocalId, uint16_t status, std::string localEid, std::string remoteEid, std::string remoteInsId)Called by the AICPU/CCU CQE error callback, records CQE errors and propagates them via broadcast.std::vectorstd::string GetErrStatusVecFromCluserMonitor()DrainerrStatusQueue_, format error descriptions by priority (CQE_ERR LOST), for upper layer queries.HcclResult RunMonitorThread()Explicitly start the background heartbeat thread (MonitorThread).HcclResult DeInit()Stop the heartbeat thread, destroy sockets, clean up all internal containers; idempotent.void SetStatus(crimer, informer, status, needBroadcasttrue)Set / update a nodes status, push intoerrRankQueue_if broadcast is needed.ClusterUIDType FormatUID(ClusterUIDCxt cxt)Assemble UID usingnetInstId/localId.std::string FormatConnTag(role, uidPair)Generate a socket tag in the formatHeartBeat_src_to_dst.Internal / Private InterfacesInterfaceDescriptionGetRemEndpointDescs / GetRemEndpointDescsPerLayerEnumerate all ranks in eachnetLayerfromRankGraph, generateUIDContext, initializeuid2FrameStatusMap_,commIdMap_.InsertClusterMonitorCxtGiven a peer UID, query RankGraph to obtain link and devicePort, decide SERVER/CLIENT role and constructSocketDesc.GetSamePlaneRankSelect left and right neighbors within the same plane according to Ring topology (only one neighbor when size2).GetConnectRankMerge netLayer0 plane (sorted bylocalId) with1plane (sorted bynetInstId), form rings respectively.CreateHBLinksAsyncBackground thread entry, traverseclusterLinkContext_to start / restart independent link establishment threads for each remUID.CreateLinkWithRemotePonitLink establishment thread entry for a single remUID:SocketCreate→ pollSocketGetStatus→ initializerecvBuffer→ register intouid2SocketRefMap_.CreateTransportHandleWrapSocketCreateto avoid duplicate creation.SendFrame/RecvFrame/ParseFrameNon-blocking heartbeat frame send (with partial send resume), non-blocking receive (with ring buffer), validity check and status update.MonitorThreadBackground main loop:CreateHBLinksAsync→ send heartbeat accumulatelostNumeveryHEARTBEAT_COUNTcycles → receive heartbeat → handlelostNumthreshold exceeded →ProcessExceptionEvent.DelErrorSocketDestroy sockets marked as abnormal inerrorSocket_.ProcessExceptionEventConsumeerrRankQueue_, broadcast abnormal frames to all neighbors where rem ! informer and status OK.PrintEvents / MakeErrMsgFormatClusterMonitorFramequeue into readable string vectors.Cross-module Callback Registration (__attribute__((constructor)))ClusterMonitorCallBackInit() { RegisterGetAicpuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); RegisterGetCcuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); RegisterAicpuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); RegisterCcuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); }The module registers two callbacks with the AICPU/CCU framework at load time:Exception Entry(error reporting) andError Query(error formatting export), which are the only coupling points between the module and the upper-layer Task exception system.Usage Limitations (Supported Scenarios and Constraint Specifications)Supported ScenariosMulti-Rank communicator: rankSize 2 communicators; when rankSize 1,RegisterToClusterMonitordirectly returnsHCCL_SUCCESSwith aWARNINGlog and no link establishment.Multi-plane topology: Supports dual Ring link establishment fornetLayer 0(within same server/device, sorted bylocalId) andnetLayer 1(cross-server/pod/superPod, sorted bynetInstId).Cross-communicator shared socket: Viahccl::ReferenceMapcounting, only one socket is created when multiple communicators connect to the same remote node; reference count--on deregistration, truly destroyed only when it reaches zero.Ring fault propagation: Single nodeLOST/CQE_ERRstatus can spread to other nodes along the Ring link, enabling querying cluster-wide exceptions from any node.CQE error capture: ReceivesTaskexceptions via AICPU/CCU callbacks, formatted as readable logs with local / remoteinstanceId / localId / Eid.Toggle switch: The environment variableHCCL_DFS_CONFIG.cluster_heartbeatcan disable heartbeat registration / CQE error capture chain.HCCL v2 communicator: Depends onHcclCommunicator::GetRankGraphV2andRankGraph, only effective whenCommunicatorV2exists; otherwise fails withCHK_PTR_NULLinGetRemEndpointDescs.Constraint SpecificationsCategoryConstraintThread model1MonitorThread(Hccl_HeartBeat) NLinkThreads (hb );threadLock_protectscommIdMap_ / uid2SocketRefMap_ / uid2FrameStatusMap_ / monitorLinkStatusMap_ / errRankQueue_ / errStatusQueue_;clusertMonitorLinkMtx_protectsclusterLinkContext_.LifecycleModule singleton is held byCollCommMgr;DeInitis triggered by the last commId deregistration or~ClusterMonitor, idempotent (guarded byisDeInit_).Timeout controlLink establishment timeout taken fromEnvConfig::GetSocketConfig().GetLinkTimeOut()(i.e.,HCCL_CONNECT_TIMEOUT); heartbeat loss thresholdlostThreshold_ HCCL_LOST_THRESHOLD(30s).Port validitydevicePort/rmtPortmust be Hccl::MAX_VALUE_TCPPORT, otherwise returnsHCCL_E_PARA.Frame sizeClusterMonitorFramecontains 4 UIDs of 2048 bytes each status dual timestamps 256 bytes reserved, fixed total length (sizeof(ClusterMonitorFrame));recvBuffer.Initcapacity isBASE_NUMBER * frameSize(approximately 2x).UID lengthHcclClusterMonitorUID.idis fixed at 2048 bytes, requiringnetInstId / localIdto not exceed 2048 bytes.Role decisionSERVER/CLIENT is determined bylocalIpAddr remoteIpAddr; when local is SERVER, fill locallistenPort, otherwise fill peerrmtPort, must be consistent with SocketConfigs listener strategy.Broadcast strategyProcessExceptionEventonly broadcasts to neighbors where rem ! informerandstatus OK, avoiding loop storms; non-OK neighbors are already self-aware.Status priorityError query order: CQE_ERR LOST (call order withinPrintEvents).Device scopeSingleton maintained per device; each device has independent heartbeats in multi-device scenarios without mutual interference.Platform dependenciesDepends onHcclCommunicator(v2),RankGraph, and Socket abstraction layer (SocketCreate/SendNb/RecvNb/GetStatus/Destroy); v1 communicator path is not supported.Environment switchWhenclusterHeartBeatEnable false, no new sockets are created during registration (commIdMap_ markers are retained), CQE error callbacks directlyreturn.Error propagation pathAbnormal frames spread gradually through the Ring link; propagation delay ≈BROADCAST_INTERVAL× Ring hops; not instantaneous synchronization.Resource releaseDeInit/UnRegisterboth performSocketDestroyand clear reference mappings; multiple calls are safe (guarded byisDeInit_,while(uid2SocketRefMap_.erase(rem)) {}spin).【免费下载链接】hcommHCOMMHuawei Communication是HCCL的通信基础库提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考