Administrator
Published on 2023-11-06 / 2 Visits
0

Yarn状态机

一、基本概念

状态机由一组状态组成,这些状态大体分为三类:初始状态、中间状态和最终状态。状态机首先由初始状态A开始运行,经过一系列的中间状态后到达最终状态,并在最终状态退出,从而形成一个有向无环图。其状态处理的逻辑是收到一个事件,触发状态A到状态B的转换,而转换操作是由事件对应的hook完成的。

以下是Yarn中的状态机的组成部分:

  1. ResourceManager中状态机:

RMApp:用于维护一个Application的生命周期,实现类 - RMAppImpl
RMAppAttempt:用于维护一次试探运行的生命周期,实现类 - RMAppAttemptImpl
RMContainer:用于维护一个已分配的资源最小单位Container的生命周期,实现类 - RMContainerImpl
RMNode:用于维护一个NodeManager的生命周期,实现类 - RMNodeImpl

  1. NodeManager中状态机:

Application:用于维护节点上一个Application的生命周期,实现类 - ApplicationImpl
Container:用于维护节点上一个容器的生命周期,实现类 - ContainerImpl
LocalizedResource:用于维护节点上资源本地化的生命周期,没有使用接口即实现类 - LocalizedResource

  1. MapReduce中的状态机:Job,Task,TaskAttempt。

在Yarn提供了一种方法将上面的状态机可视化,图片如下:
ResourceManager.png
ResourceManager相关的
NodeManager.pngNodeManager相关的
MapReduce.png
MapReduce相关的

二、状态机的构建

状态机的核心类是StateMachineFactory,主要是构建其中的一个名字叫Map<STATE, Map<EVENTTYPE,Transition<OPERAND, STATE, EVENTTYPE, EVENT>>> stateMachineTable的属性,其使用了泛型,这里面存了状态机的元信息。后续调用完全是根据这个Map来运行的。这个map的组成,从from到to端,第一个STATE是from端的状态。从一个状态转移可以有多个事件触发,其中的每一个事件可以有一个Transition,每个Transition就是有一个OPERAND操作。
我们以RMAppImpl中的状态机进行举例说明

  private static final StateMachineFactory<RMAppImpl,
                                           RMAppState,
                                           RMAppEventType,
                                           RMAppEvent> stateMachineFactory
                               = new StateMachineFactory<RMAppImpl,
                                           RMAppState,
                                           RMAppEventType,
                                           RMAppEvent>(RMAppState.NEW)

![ResourceManager.png](/upload/ResourceManager.png)

     // Transitions from NEW state
    .addTransition(RMAppState.NEW, RMAppState.NEW,
        RMAppEventType.NODE_UPDATE, new RMAppNodeUpdateTransition())
    .addTransition(RMAppState.NEW, RMAppState.NEW_SAVING,
        RMAppEventType.START, new RMAppNewlySavingTransition())
    .addTransition(RMAppState.NEW, EnumSet.of(RMAppState.SUBMITTED,
            RMAppState.ACCEPTED, RMAppState.FINISHED, RMAppState.FAILED,
            RMAppState.KILLED, RMAppState.FINAL_SAVING),
        RMAppEventType.RECOVER, new RMAppRecoveredTransition())
    .addTransition(RMAppState.NEW, RMAppState.KILLED, RMAppEventType.KILL,
        new AppKilledTransition())
    .addTransition(RMAppState.NEW, RMAppState.FINAL_SAVING,
        RMAppEventType.APP_REJECTED,
        new FinalSavingTransition(new AppRejectedTransition(),
          RMAppState.FAILED))
    ...
	.installTopology();

可以看到StateMachineFactory提供了addTransition方法来让用户添加各种状态转移,最后通过installTopology完成一个拓扑结构的搭建。addTransition提供了五种重载方法供用户使用,分别为

addTransition(STATE preState, STATE postState, EVENTTYPE eventType)
addTransition(STATE preState, STATE postState, Set<EVENTTYPE> eventTypes)
addTransition(
      STATE preState, STATE postState, Set<EVENTTYPE> eventTypes,
      SingleArcTransition<OPERAND, EVENT> hook)
addTransition(STATE preState, STATE postState,
                        EVENTTYPE eventType,
                        SingleArcTransition<OPERAND, EVENT> hook)
addTransition(STATE preState, Set<STATE> postStates,
                        EVENTTYPE eventType,
                        MultipleArcTransition<OPERAND, EVENT, STATE> hook)

由其上的addTransition方法可以看出定义了三种状态转换方式,分别是

  • preState通过某个事件转换为postState,也就是状态机在preState时,接收到Event事件后,执行对应的hook,并在执行完成后将当前的状态转换为postState。addTransition(STATE preState, STATE postState, EVENTTYPE eventType, SingleArcTransition<OPERAND, EVENT> hook)
  • preState通过多个事件转换为postState,也就是状态机在preState时,接收到某些Event事件后,执行对应的hook,并在执行完成后将当前的状态转换为postState。addTransition(STATE preState, STATE postState, Set eventTypes, SingleArcTransition<OPERAND, EVENT> hook)
  • preState通过某个事件转换为多个postState,也就是状态机在preState时,接收到Event事件后,执行对应的hook,并在执行完成后将返回hook的返回值所表示的状态。addTransition(STATE preState, Set postStates, EVENTTYPE eventType, MultipleArcTransition<OPERAND, EVENT, STATE> hook)

那么根据以上所述,在RMAppImpl中如果接收到了RMAppEventType.START事件那么会用对应的状态转移函数RMAppNewlySavingTransition来将RMAppState.NEW转换成RMAppState.NEW_SAVING。如果收到了RMAppState.SUBMITTED,RMAppState.ACCEPTED, RMAppState.FINISHED, RMAppState.FAILED, RMAppState.KILLED, RMAppState.FINAL_SAVING其中的一个事件那么会使用对应的状态转移函数RMAppRecoveredTransition的返回值来作为转移后的状态。