逐题说明考什么、怎么想、如何代入，以及容易出错的地方。每题末尾附英文答案要点；简答每个有效得分点 1 分，计算题允许保留正确方法分。
先做题再看解析。标为“进一步理解”的补充不额外计分。
在线值(1,5)，目标值(7,2)，Double 的未来项究竟读取哪个数？
先在在线表圈动作下标，再在目标表读同一个下标。不要直接把两个表各取一个最大数。
Online next-state values are (1,5), while target-network values are (7,2). Which value is used for the bootstrap term in Double DQN?
A. 2
B. 7
C. 5
D. 1
先看在线网络：第二动作 5 最大，所以选择 a₂。再到目标网络查同一动作的值，得到 2。题目只问自举部分所读的 Q，不要求另外加奖励。
B 是标准 DQN 的目标网络最大值；C 是在线网络最大值本身，没切换评价网络；D 对应在线表非最优动作。
随机环境的 Bellman 最优备份，先平均什么、再最大化什么？
可控的是动作，不可控的是该动作后的随机结果。每个动作先得到一个期望分数，再从动作分数中选最大。
For an MDP with stochastic next states, which operation order is correct in a Bellman optimality backup?
A. Choose the best next-state outcome, then average actions.
B. Average all actions equally and ignore the policy.
C. For each action take the expected reward plus discounted next value, then maximize over actions.
D. Multiply the immediate reward by γ before any averaging.
先算“如果指定做某动作”的期望结果，再比较这些动作哪个最好。环境随机性不由智能体选择。
A 把不可控结果也挑最好的，会过于乐观；B 是某一随机策略的评估，不是最优控制；D 奖励时序不对，下一步收到的奖励不折扣。
四种说法中哪一条不属于 GLIE 定义？题目特别问 NOT。
先在草稿写出无限探索、极限贪心两个条件；再看选项是否偷偷把“无限时间性质”换成“有限回合精确学会”。
Which statement is NOT part of the definition of GLIE?
A. All required state-action pairs are visited infinitely often.
B. The exact optimal policy is known after a finite number of episodes.
C. The policy becomes greedy in the limit.
D. It describes long-run exploration and exploitation properties.
注意题目问 NOT。GLIE 是无限时间意义的性质，并不承诺训练有限回合后已经知道精确最优策略。
A 与 C 是两个核心条件；D 正确描述其类别。要由这些性质得到具体算法收敛结论，还要加步长、有限状态动作空间等相应假设。
γ=0 时，回报公式还剩下什么？
把第一个奖励与后面奖励分开。只有后面项带γ，第一项权重是1。
If γ=0, which quantity does the return Gₜ equal?
A. Rₜ₊₁
B. The sum of all future rewards.
C. V(Sₜ₊₁)
D. Zero for every task.
γ=0 时，后面项都消失，第一项系数仍是 1。因此只关心下一次即时奖励，不是所有回报都为零。
B 忽略折扣；C 是价值估计；D 把第一项也错误乘了零。
离散动作 DQN 输入一个状态后，输出的数分别代表什么？
比较“动作有多好”与“动作被选中的概率”。DQN 先打价值分，再由贪心或探索规则选动作，这两个步骤不是同一个输出。
In a standard discrete-action DQN, what does the network output for one input state?
A. A probability distribution that must sum to one.
B. Only the next state.
C. The exact transition matrix.
D. An estimated Q value for each available action.
DQN 输出每个离散动作的估计价值。Q 可正可负，通常不要求和为 1，然后用贪心或 ε-greedy 选动作。
A 更像随机策略网络输出；B、C 是模型相关信息。Q 值为 (2,5) 不代表动作概率为 (2/7,5/7)，除非另外明确规定某种概率转换规则。
5个动作只有1个最高，ε=0.1。问选到任何非贪心动作的总概率。
先算每个动作的探索份额，再数共有几个非贪心动作。探索时也可能抽中贪心动作，所以总非贪心概率不等于ε。
With five actions, a unique greedy action and ε=0.1, what is the probability of choosing a nongreedy action in total?
A. 0.08
B. 0.10
C. 0.02
D. 0.90
每个非贪心动作得到 ε/5=0.02，一共有四个，所以总概率是 4×0.02=0.08。贪心动作概率为 0.9+0.02=0.92。
B 将探索概率误认为“选到非贪心的概率”，但探索也可能选到贪心；C 只算一个非贪心动作；D 是利用分支的概率。
Actor–critic 中 critic 主要做什么？
把 actor 想成决定怎么走的人，critic 想成估计这次选择长期效果的评分员。评分有误差，所以并不保证每次动作最优。
In actor–critic, what is the critic’s main role?
A. Choose actions without a policy.
B. Estimate value information used to guide policy improvement.
C. Guarantee every selected action is optimal.
D. Replace all rewards by zero.
critic 估计 V、Q 或相关评价信号，帮助 actor 判断刚才的动作表现是否好于预期。actor 才负责参数化和更新行为策略。
A 抹掉了 actor 的作用；C critic 本身是学习到的估计，会有误差；D 破坏了学习目标。critic 的批评不是语言评价，而是可用于梯度更新的数值信号。
同一状态在一回合出现两次，对应完整回报4和10，first-visit MC 用哪一个？
“首次”修饰访问发生的位置，不是只拿第一笔奖励。找到第一次访问，然后从那一刻算完整未来回报。
A state occurs twice in one episode, with returns 4 and 10 at its two visits. Which return is used by first-visit MC for that state in this episode?
A. 10 only
B. 7
C. Both returns as two updates
D. 4 only
首次访问 MC 每回合对该状态只使用第一次出现对应的完整回报，所以用 4。
A 是最后一次访问；B 是这两次访问的平均值，在特定每次访问样本均值计算中可能出现；C 是每次访问更新。first visit 不是“只拿第一笔即时奖励”，仍然要算从首次访问开始的完整未来回报。
小车下一位置受速度影响，只记录位置够不够？
想象同一位置的一辆车向左、一辆向右。相同控制动作不一定得到相同后续分布，所以状态需要能区分相关动态信息。
For a moving cart whose future position depends on its velocity, which claim is most reasonable?
A. Position alone must always be Markov.
B. Position and velocity can provide a more sufficient state than position alone.
C. Rewards alone always determine the next position.
D. A random policy automatically makes position Markov.
同一位置的车可能正向左或向右移动，即使施加相同动作，下一位置也可能不同。速度保留了预测未来所需的动态信息。
A 忽略了速度；C 没有足够动力学信息；D 改变策略不能凭空补上丢失的状态变量。说“更充分”也不表示任意复杂系统加速度这一个变量就必然完全 Markov，还需考虑具体环境。
线性价值函数的特征为(1,2)，权重为(3,−1)，求预测值。
把它当两项评分：第一特征乘第一权重、第二特征乘第二权重，最后相加。这里没有概率归一化。
A linear value approximator uses x(s)=(1,2) and w=(3,-1). What is v̂(s,w)?
A. 5
B. -1
C. 1
D. 3
这是点积：对应位置相乘再求和。A 忽略了第二个权重的负号；B 仅是某个权重；D 只取第一项。特征维度不是动作数量，也不是回报步数。
分别区分 reward/return、V/Q、max/argmax、k/t、α/ε，一共五组。
每组都用“它回答什么问题”解释。别只写中文译名：例如max回答分数多少，argmax回答选择谁。
Briefly distinguish reward from return, state value from action value, and max from argmax. Finally distinguish the iteration counter k from environment time t and the learning rate α from the exploration parameter ε. [5 points]
① Reward 是一次转移收到的即时数值；return 是从某时刻开始的未来折扣奖励和。② V(s) 尚未指定第一动作，按策略平均；Q(s,a) 指定第一步做 a，之后再按相应策略继续。
③ max 返回最大的数，argmax 返回达到该数的动作或下标。④ k 表示算法更新轮数，t 表示真实/采样轨迹的时间。⑤ α 控制估计向目标移动多少，ε 控制 ε-greedy 选动作时的探索概率。
例如 Q=(2,6)，max=6，argmax=第二动作；α=0.1 表示只改动目标差距的十分之一，不表示探索概率 10%。
解释首次/每次访问 MC、样本均值/常数步长，以及重复状态为何让在线TD的顺序重要。
这里有三个独立选择：用哪些访问产生的回报、如何平均这些回报、什么时候改表。不要因为都出现“每次”就把它们混在一起。
Explain first-visit versus every-visit MC. Distinguish sample-mean and constant-step-size estimation. Explain why online TD must specify the order of transitions when a state repeats. [5 points]
① First-visit 每回合每个状态只使用首次访问的回报。② Every-visit 每次访问对应的回报都使用。
③ Sample mean 以已有回报的算术平均估计，递推步长是 1/N。④ Constant α 用 V←V+α(G−V)，一般更重视近期样本，并非普通算术平均。
⑤ 在线 TD 每条转移后立即改表，后面的目标可能读到前面刚更新的值，重复状态时处理顺序就影响有限次更新结果。
例如旧 V=0，依次目标 2、6，α=0.5：得到 1 再到 3.5；样本均值则为 4。这两种估计不能互换。
把回放池、目标网络、Double、PER、Dueling 分别对应到主要作用，并各给一个细节。
按训练流程排位置：收集和存经验→选哪些经验训练→网络怎样表示Q→怎样构造目标。这样每个名字都有落脚点。
Match each mechanism to its main purpose and give one useful detail: (i) replay buffer, (ii) target network, (iii) Double DQN, (iv) prioritized replay, (v) dueling network. [5 points]
| 机制 | 主要作用 | 关键细节 |
|---|---|---|
| Replay buffer | 重用经验、减轻时间相关性 | 保存 (s,a,r,s′,done)，随机抽小批量 |
| Target network | 减慢自举目标变化 | θ⁻ 延迟从在线参数同步 |
| Double DQN | 缓解最大化过估计 | 在线选动作、目标网络评价 |
| Prioritized replay | 把更多采样分给值得学习的经验 | 常用绝对 TD error 作优先级并配重要性权重 |
| Dueling network | 共享状态价值并区分动作优势 | V+A−mean(A) |
这些方法改变的是不同环节，因此可以组合。比如“使用 dueling 架构、Double 目标、优先回放”的 agent 并不矛盾。
两动作单步 bandit：a₁概率是sigmoid(θ)，初始θ=0，实际选a₁获4分，基线2，α=0.1。求对数概率导数、更新后的θ及概率变化方向。
先算当前概率；再算这个概率对参数的敏感方向；用“本次回报−基线”决定沿正还是反方向；最后乘学习率。θ是参数，不是奖励或概率。
In a one-state two-action bandit, πθ(a₁)=σ(θ) and πθ(a₂)=1−σ(θ), where σ is the logistic function. An episode consists of one action. At θ=0 the sampled action is a₁ and its reward is 4. Use baseline b=2 and learning rate α=0.1. Compute ∂logπθ(a₁)/∂θ, the REINFORCE parameter update, and the direction of the probability change. Explain the role of the baseline. [5 points]
① θ=0 时 σ(0)=1/2。（1 分）② d logσ(θ)/dθ=σ′/σ=1−σ，因此梯度为 0.5。（1 分）
③ 新参数 0.1。（1 分）④ sigmoid 单调递增，所以 a₁ 概率从 0.5 上升，约为 0.5250。（1 分，方向正确即可）⑤ 基线将表现和参考水平比较，可降低梯度估计方差；动作无关时不改变期望梯度。（1 分）
这不是把奖励直接当概率，也不是令 θ=4。θ 是策略参数，要沿梯度方向小步更新。
区分无模型学习与规划，说明Dyna的真实和模拟两部分，并解释模型错误为何可能被更多规划放大。
先问数据是亲自遇到的，还是模型想象出的。模型生成的经验也能用相同价值更新式处理，但它是否可信取决于模型。
Distinguish model-free learning from planning. Outline the real-experience and simulated-experience parts of Dyna. Why can additional planning be harmful when the learned model is inaccurate? [5 points]
① Model-free 价值/策略学习直接用实际交互样本更新，不要求显式环境转移模型。② Planning 使用模型推演结果来改进价值或策略。
③ Dyna 接到真实 (s,a,r,s′) 后，可以直接进行一次学习更新，同时更新模型。④ 再抽取状态动作，通过模型生成模拟转移，进行额外学习更新。
⑤ 若模型系统性预测错奖励或下一状态，更多规划会重复放大错误，可能让价值或策略偏离真实环境的好解。
每点 1 分。Replay 是重放已经发生的数据，Dyna planning 是通过模型生成结果，这两个来源要分开。
γ=1，每个非终止转移扣1；A各半留A或去B；B以0.25回A、0.75结束。求转移矩阵、真实价值、A→A→B→T样本回报，并解释负价值对应的步数。
无折扣且每步扣1，所以总回报就是负的剩余步数。先解期望方程，再把结果翻译成平均还要走几步。无限允许的最长路径不等于无限期望步数。
γ=1. States A and B are nonterminal. Each transition from either state has reward -1. A moves to A or B with probability 0.5 each. B moves to A with probability 0.25 and to terminal T with probability 0.75. T is absorbing with zero reward.
(a) Write the transition matrix and the Bellman equations. [2]
(b) Solve for v(A),v(B). [2]
(c) For the observed episode A → A → B → T, compute all three returns. [2]
(d) Interpret -v(A) and -v(B), and explain why finite values are possible even though γ=1. [2]
这是一条会吸收终止且期望步数有限的有限链。每走一步扣 1，价值可以用负的期望剩余步数理解。
矩阵行按 (A,B,T)：(0.5,0.5,0)、(0.25,0,0.75)、(0,0,1)。
第一式整理为 vA=-2+vB；代入第二式得 vA=-3+0.25vA，因此 vA=-4，vB=-2。
三笔奖励均为 -1，且不折扣：G₀=-3、G₁=-2、G₂=-1。A 首次出现的样本回报 -3 不必等于平均值 -4。
从 A 起期望还需 4 步到终点，从 B 起为 2 步。各非终止状态都可到达 T，且不存在封闭的非终止类；本有限链具有有限期望吸收时间，所以 γ=1 时仍有有限总回报。
S的a得2去U、b得1结束；U的c得0回S、d得3结束；γ=0.5，旧策略固定a/c。求旧值、所有Q、新策略值和最优性。
旧策略确定，所以不要擅自给动作各乘1/2。旧值先按循环联立求解；然后评价没选过的b、d，才知道是否值得换。
γ=0.5. At S, a gives reward 2 and goes to U; b gives reward 1 and terminates. At U, c gives reward 0 and goes to S; d gives reward 3 and terminates. The initial deterministic policy is π=(a,c).
(a) Write its Bellman equations. [2]
(b) Solve for vπ(S),vπ(U). [2]
(c) Compute all four qπ values and give a greedy improvement. [2]
(d) Evaluate the improved policy and test whether it is optimal. [2]
策略 (a,c) 表示 S 必做 a、U 必做 c。环境动作也确定，整个旧策略是循环。
代入 vS=2+0.25vS，得 vS=8/3，vU=4/3。
S 的 a=2+0.5×4/3=8/3，b=1，仍选 a；U 的 c=0.5×8/3=4/3，d=3，改选 d。新策略 (a,d)。
U 直接终止得 3；S 得 2+0.5×3=3.5。重新比较 U 的 c=0.5×3.5=1.75<3，S 的 a=3.5>1，策略稳定且最优。
同一环境：S的a得0回S，b得1去U；U的c得2结束，d得−1回S；γ=0.5。V评估各半随机策略，W进行价值迭代，二者初值都零。
准备两张表。V每轮按动作概率平均，W每轮动作取max；两者都读自己的上一轮。最后一问指定用V₂求Q，不能误读成W₂。
γ=0.5. At S, a gives reward 0 and returns to S; b gives reward 1 and goes to U. At U, c gives reward 2 and terminates; d gives reward -1 and goes to S. Let Vₖ be random-policy evaluation with probability 0.5 per action. Let Wₖ be value iteration. Start both at zero.
(a) Write the update formulas for V and W at S. [1]
(b) Compute V₁,V₂ for (S,U). [2]
(c) Compute W₁,W₂ for (S,U). [2]
(d) Using V₂ as successor values, calculate all four one-step action values and the greedy policy. [2]
(e) Explain why the two sequences differ even though they use the same environment. [2]
![V_{k+1}(S)=0.5[0.5V_k(S)]+0.5[1+0.5V_k(U)]](data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiIHN0YW5kYWxvbmU9Im5vIj8+DQo8IURPQ1RZUEUgc3ZnIFBVQkxJQyAiLS8vVzNDLy9EVEQgU1ZHIDEuMS8vRU4iDQogICJodHRwOi8vd3d3LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPg0KPHN2ZyB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgd2lkdGg9IjI4OHB0IiBoZWlnaHQ9IjE2cHQiIHZpZXdCb3g9IjAgMCAyODggMTYiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmVyc2lvbj0iMS4xIj4NCiA8bWV0YWRhdGE+DQogIDxyZGY6UkRGIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6Y2M9Imh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL25zIyIgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4NCiAgIDxjYzpXb3JrPg0KICAgIDxkYzp0eXBlIHJkZjpyZXNvdXJjZT0iaHR0cDovL3B1cmwub3JnL2RjL2RjbWl0eXBlL1N0aWxsSW1hZ2UiLz4NCiAgICA8ZGM6ZGF0ZT4yMDI2LTExLTEwVDIwOjMxOjQ2LjA0ODI1MTwvZGM6ZGF0ZT4NCiAgICA8ZGM6Zm9ybWF0PmltYWdlL3N2Zyt4bWw8L2RjOmZvcm1hdD4NCiAgICA8ZGM6Y3JlYXRvcj4NCiAgICAgPGNjOkFnZW50Pg0KICAgICAgPGRjOnRpdGxlPk1hdHBsb3RsaWIgdjMuMTEuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88L2RjOnRpdGxlPg0KICAgICA8L2NjOkFnZW50Pg0KICAgIDwvZGM6Y3JlYXRvcj4NCiAgIDwvY2M6V29yaz4NCiAgPC9yZGY6UkRGPg0KIDwvbWV0YWRhdGE+DQogPGRlZnM+DQogIDxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+KntzdHJva2UtbGluZWpvaW46IHJvdW5kOyBzdHJva2UtbGluZWNhcDogYnV0dH08L3N0eWxlPg0KIDwvZGVmcz4NCiA8ZyBpZD0iZmlndXJlXzEiPg0KICA8ZyBpZD0icGF0Y2hfMSI+DQogICA8cGF0aCBkPSJNIDAgMTYgDQpMIDI4OCAxNiANCkwgMjg4IDAgDQpMIDAgMCANCnoNCiIgc3R5bGU9ImZpbGw6ICNmZmZmZmYiLz4NCiAgPC9nPg0KICA8ZyBpZD0idGV4dF8xIj4NCiAgIDwhLS0gJFZfe2srMX0oUyk9MC41WzAuNVZfayhTKV0rMC41WzErMC41Vl9rKFUpXSQgLS0+DQogICA8ZyBzdHlsZT0iZmlsbDogIzE3M2U0NiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAxMikgc2NhbGUoMC4xNyAtMC4xNykiPg0KICAgIDxkZWZzPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM5IiBkPSJNIDk3MiAtOTcgDQpMIDEzMjIgMzQzMSANClEgMTM0NyAzNzAwIDEzNDcgMzc5NCANClEgMTM0NyAzOTM0IDEyNDIgNDAyOCANClEgMTEzOCA0MTIyIDkxMyA0MTIyIA0KTCA4MDAgNDEyMiANCkwgODMxIDQyMzggDQpMIDI1MDMgNDIzOCANCkwgMjQ2OSA0MTIyIA0KUSAyMTk3IDQxMTkgMjA5NSA0MDY2IA0KUSAxOTk0IDQwMTMgMTkzNCAzODg0IA0KUSAxOTAzIDM4MTYgMTg2OSAzNDk0IA0KTCAxNjE5IDk2NiANCkwgMzI3MiAzMjIyIA0KUSAzNTQxIDM1ODggMzYwOSAzNzMxIA0KUSAzNjU2IDM4MjggMzY1NiAzOTAwIA0KUSAzNjU2IDM5ODQgMzU4NCA0MDQ2IA0KUSAzNTEzIDQxMDkgMzMxNiA0MTIyIA0KTCAzMzUzIDQyMzggDQpMIDQ2MjggNDIzOCANCkwgNDU5NyA0MTIyIA0KUSA0NDQ0IDQxMDkgNDM1MyA0MDYzIA0KUSA0MTgxIDM5ODEgNDAxMSAzODE1IA0KUSAzODQxIDM2NTAgMzQ3MiAzMTQ3IA0KTCAxMDg0IC05NyANCkwgOTcyIC05NyANCnoNCiIgdHJhbnNmb3JtPSJzY2FsZSgwLjAxNTYyNSkiLz4NCiAgICAgPHBhdGggaWQ9IlRpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC00ZSIgZD0iTSAxODMxIDQ0NDQgDQpMIDk0NCAxMzcyIA0KTCAxMzAwIDE2NzUgDQpRIDE4ODQgMjE2OSAyMDU5IDI0MDYgDQpRIDIxMTYgMjQ4NCAyMTE2IDI1MjUgDQpRIDIxMTYgMjU1MyAyMDk3IDI1NzggDQpRIDIwNzggMjYwMyAyMDIzIDI2MTggDQpRIDE5NjkgMjYzNCAxODMxIDI2MzQgDQpMIDE3MjIgMjYzNCANCkwgMTc1NiAyNzUzIA0KTCAzMTIyIDI3NTMgDQpMIDMwOTEgMjYzNCANClEgMjg1NiAyNjIyIDI2ODYgMjU2MSANClEgMjUxNiAyNTAwIDIzNTYgMjM4MSANClEgMjA0MSAyMTQ3IDE5NDEgMjA1NiANCkwgMTY3MiAxODE5IA0KUSAxNzQxIDE1OTQgMTg0MSAxMTkxIA0KUSAyMDA5IDUyMiAyMDY5IDM5NyANClEgMjEyOCAyNzIgMjE5MSAyNzIgDQpRIDIyNDQgMjcyIDIzMTYgMzI1IA0KUSAyNDc4IDQ0NCAyNjYzIDcwMyANCkwgMjc3MiA2MjUgDQpRIDI0MzEgMTk3IDIyNTAgNjEgDQpRIDIwNjkgLTc1IDE5MTkgLTc1IA0KUSAxNzg4IC03NSAxNzE2IDI4IA0KUSAxNTkxIDIxMyAxMzAwIDE0ODEgDQpMIDg3MiAxMTA5IA0KTCA1NTAgMCANCkwgNjMgMCANCkwgMTEwMCAzNTY2IA0KTCAxMTg4IDM4NjkgDQpRIDEyMDMgMzk0NyAxMjAzIDQwMDYgDQpRIDEyMDMgNDA4MSAxMTQ1IDQxMzEgDQpRIDEwODggNDE4MSA5NzggNDE4MSANClEgOTA2IDQxODEgNzUwIDQxNTAgDQpMIDc1MCA0MjU5IA0KTCAxODMxIDQ0NDQgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC1lIiBkPSJNIDE2NzIgNDQxIA0KTCAxNjcyIDIwMDAgDQpMIDExNiAyMDAwIA0KTCAxMTYgMjI1NiANCkwgMTY3MiAyMjU2IA0KTCAxNjcyIDM4MDkgDQpMIDE5MjIgMzgwOSANCkwgMTkyMiAyMjU2IA0KTCAzNDg0IDIyNTYgDQpMIDM0ODQgMjAwMCANCkwgMTkyMiAyMDAwIA0KTCAxOTIyIDQ0MSANCkwgMTY3MiA0NDEgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC0xNCIgZD0iTSA3NTAgMzgyMiANCkwgMTc4MSA0MzI1IA0KTCAxODg0IDQzMjUgDQpMIDE4ODQgNzQ3IA0KUSAxODg0IDM5MSAxOTE0IDMwMyANClEgMTk0NCAyMTYgMjAzNyAxNjkgDQpRIDIxMzEgMTIyIDI0MTkgMTE2IA0KTCAyNDE5IDAgDQpMIDgyNSAwIA0KTCA4MjUgMTE2IA0KUSAxMTI1IDEyMiAxMjEyIDE2NyANClEgMTMwMCAyMTMgMTMzNCAyODkgDQpRIDEzNjkgMzY2IDEzNjkgNzQ3IA0KTCAxMzY5IDMwMzQgDQpRIDEzNjkgMzQ5NyAxMzM4IDM2MjggDQpRIDEzMTYgMzcyOCAxMjU4IDM3NzUgDQpRIDEyMDAgMzgyMiAxMTE5IDM4MjIgDQpRIDEwMDMgMzgyMiA3OTcgMzcyNSANCkwgNzUwIDM4MjIgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC1iIiBkPSJNIDE5ODggLTEyNTMgDQpMIDE5ODggLTEzNjkgDQpRIDE1MTYgLTExMzEgMTIwMCAtODEzIA0KUSA3NTAgLTM1OSA1MDYgMjU2IA0KUSAyNjMgODcyIDI2MyAxNTM0IA0KUSAyNjMgMjUwMyA3NDEgMzMwMSANClEgMTIxOSA0MTAwIDE5ODggNDQ0NCANCkwgMTk4OCA0MzEzIA0KUSAxNjAzIDQxMDAgMTM1NiAzNzMxIA0KUSAxMTA5IDMzNjMgOTg3IDI3OTcgDQpRIDg2NiAyMjMxIDg2NiAxNjE2IA0KUSA4NjYgOTQ3IDk2OSA0MDAgDQpRIDEwNTAgLTMxIDExNjUgLTI5MiANClEgMTI4MSAtNTUzIDE0NzYgLTc5MyANClEgMTY3MiAtMTAzNCAxOTg4IC0xMjUzIA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM2IiBkPSJNIDQ0IC05NyANCkwgMzk0IDE0NzUgDQpMIDUwMCAxNDc1IA0KUSA0NzggMTI0NyA0NzggMTA5NyANClEgNDc4IDY2OSA3NTEgNDAwIA0KUSAxMDI1IDEzMSAxNDY2IDEzMSANClEgMTg3NSAxMzEgMjA4NyAzODIgDQpRIDIzMDAgNjM0IDIzMDAgOTYzIA0KUSAyMzAwIDExNzUgMjIwMyAxMzUwIA0KUSAyMDU2IDE2MDkgMTQxOSAyMjYzIA0KUSAxMTA5IDI1NzUgMTAyMiAyNzI4IA0KUSA4NzggMjk4MSA4NzggMzI1OSANClEgODc4IDM3MDMgMTIwOSA0MDE4IA0KUSAxNTQxIDQzMzQgMjA1OSA0MzM0IA0KUSAyMjM0IDQzMzQgMjM5MSA0MzAwIA0KUSAyNDg4IDQyODEgMjc0NCA0MTc1IA0KUSAyOTI1IDQxMDMgMjk0NCA0MDk3IA0KUSAyOTg4IDQwODggMzA0MSA0MDg4IA0KUSAzMTMxIDQwODggMzE5NyA0MTM0IA0KUSAzMjYzIDQxODEgMzM1MCA0MzM0IA0KTCAzNDY5IDQzMzQgDQpMIDMxNDQgMjkyOCANCkwgMzAzOCAyOTI4IA0KUSAzMDUwIDMxMTYgMzA1MCAzMjMxIA0KUSAzMDUwIDM2MDkgMjgwMCAzODUwIA0KUSAyNTUwIDQwOTEgMjE0MSA0MDkxIA0KUSAxODE2IDQwOTEgMTYxMiAzOTAwIA0KUSAxNDA5IDM3MDkgMTQwOSAzNDU5IA0KUSAxNDA5IDMyNDEgMTUzOSAzMDQyIA0KUSAxNjY5IDI4NDQgMjEzNiAyMzk1IA0KUSAyNjAzIDE5NDcgMjc0MCAxNjk1IA0KUSAyODc4IDE0NDQgMjg3OCAxMTU5IA0KUSAyODc4IDgzOCAyNzA4IDUzNiANClEgMjUzOCAyMzQgMjIxOSA2OCANClEgMTkwMCAtOTcgMTUyMiAtOTcgDQpRIDEzMzQgLTk3IDExNzEgLTYyIA0KUSAxMDA5IC0yOCA2NTMgMTA2IA0KUSA1MzEgMTUzIDQ1MCAxNTMgDQpRIDI2NiAxNTMgMTYzIC05NyANCkwgNDQgLTk3IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtYyIgZD0iTSAxNDQgNDMxMyANCkwgMTQ0IDQ0NDQgDQpRIDYxOSA0MjA5IDkzNCAzODkxIA0KUSAxMzgxIDM0MzQgMTYyNSAyODIwIA0KUSAxODY5IDIyMDYgMTg2OSAxNTQxIA0KUSAxODY5IDU3MiAxMzkyIC0yMjYgDQpRIDkxNiAtMTAyNSAxNDQgLTEzNjkgDQpMIDE0NCAtMTI1MyANClEgNTI4IC0xMDM4IDc3NiAtNjcwIA0KUSAxMDI1IC0zMDMgMTE0NSAyNjQgDQpRIDEyNjYgODMxIDEyNjYgMTQ0NyANClEgMTI2NiAyMTEzIDExNjMgMjY2MyANClEgMTA4NCAzMDk0IDk2NyAzMzUzIA0KUSA4NTAgMzYxMyA2NTYgMzg1MyANClEgNDYzIDQwOTQgMTQ0IDQzMTMgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC0yMCIgZD0iTSAxMTYgMjc1OSANCkwgMzQ4OCAyNzU5IA0KTCAzNDg4IDI1MDMgDQpMIDExNiAyNTAzIA0KTCAxMTYgMjc1OSANCnoNCk0gMTE2IDE3NDEgDQpMIDM0ODggMTc0MSANCkwgMzQ4OCAxNDg0IA0KTCAxMTYgMTQ4NCANCkwgMTE2IDE3NDEgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC0xMyIgZD0iTSAyMzEgMjA5NCANClEgMjMxIDI4MTkgNDUwIDMzNDIgDQpRIDY2OSAzODY2IDEwMzEgNDEyMiANClEgMTMxMyA0MzI1IDE2MTMgNDMyNSANClEgMjEwMCA0MzI1IDI0ODggMzgyOCANClEgMjk3MiAzMjEzIDI5NzIgMjE1OSANClEgMjk3MiAxNDIyIDI3NTkgOTA2IA0KUSAyNTQ3IDM5MSAyMjE3IDE1OCANClEgMTg4OCAtNzUgMTU4MSAtNzUgDQpRIDk3NSAtNzUgNTcyIDY0MSANClEgMjMxIDEyNDQgMjMxIDIwOTQgDQp6DQpNIDg0NCAyMDE2IA0KUSA4NDQgMTE0MSAxMDU5IDU4OCANClEgMTIzOCAxMjIgMTU5MSAxMjIgDQpRIDE3NTkgMTIyIDE5NDAgMjczIA0KUSAyMTIyIDQyNSAyMjE2IDc4MSANClEgMjM1OSAxMzE5IDIzNTkgMjI5NyANClEgMjM1OSAzMDIyIDIyMDkgMzUwNiANClEgMjA5NyAzODY2IDE5MTkgNDAxNiANClEgMTc5MSA0MTE5IDE2MDkgNDExOSANClEgMTM5NyA0MTE5IDEyMzEgMzkyOCANClEgMTAwNiAzNjY5IDkyNSAzMTEyIA0KUSA4NDQgMjU1NiA4NDQgMjAxNiANCnoNCiIgdHJhbnNmb3JtPSJzY2FsZSgwLjAxNTYyNSkiLz4NCiAgICAgPHBhdGggaWQ9IlRpbWVzTmV3Um9tYW5QU01ULTExIiBkPSJNIDgwMCA2MDYgDQpRIDk0NyA2MDYgMTA0NyA1MDQgDQpRIDExNDcgNDAzIDExNDcgMjU5IA0KUSAxMTQ3IDExNiAxMDQ1IDE0IA0KUSA5NDQgLTg4IDgwMCAtODggDQpRIDY1NiAtODggNTU0IDE0IA0KUSA0NTMgMTE2IDQ1MyAyNTkgDQpRIDQ1MyA0MDYgNTU0IDUwNiANClEgNjU2IDYwNiA4MDAgNjA2IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtMTgiIGQ9Ik0gMjc3OCA0MjM4IA0KTCAyNTM0IDM3MDYgDQpMIDEyNTkgMzcwNiANCkwgOTgxIDMxMzggDQpRIDE4MDkgMzAxNiAyMjk0IDI1MjIgDQpRIDI3MDkgMjA5NyAyNzA5IDE1MjIgDQpRIDI3MDkgMTE4OCAyNTczIDkwMyANClEgMjQzOCA2MTkgMjIzMSA0MTkgDQpRIDIwMjUgMjE5IDE3NzIgOTcgDQpRIDE0MTMgLTc1IDEwMzQgLTc1IA0KUSA2NTMgLTc1IDQ3OSA1NCANClEgMzA2IDE4NCAzMDYgMzQxIA0KUSAzMDYgNDI4IDM3OCA0OTUgDQpRIDQ1MCA1NjMgNTU5IDU2MyANClEgNjQxIDU2MyA3MDIgNTM4IA0KUSA3NjMgNTEzIDkwOSA0MDkgDQpRIDExNDQgMjQ3IDEzODQgMjQ3IA0KUSAxNzUwIDI0NyAyMDI2IDUyMyANClEgMjMwMyA4MDAgMjMwMyAxMTk3IA0KUSAyMzAzIDE1ODEgMjA1NiAxOTE0IA0KUSAxODA5IDIyNDcgMTM3NSAyNDI4IA0KUSAxMDM0IDI1NjkgNDQ3IDI1OTEgDQpMIDEyNTkgNDIzOCANCkwgMjc3OCA0MjM4IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtM2UiIGQ9Ik0gMTkwMCAtMTI2OSANCkwgNTI1IC0xMjY5IA0KTCA1MjUgNDMzNCANCkwgMTkwMCA0MzM0IA0KTCAxOTAwIDQwODggDQpMIDk5NCA0MDg4IA0KTCA5OTQgLTEwMjUgDQpMIDE5MDAgLTEwMjUgDQpMIDE5MDAgLTEyNjkgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC00MCIgZD0iTSAyMzQgNDMzNCANCkwgMTYwOSA0MzM0IA0KTCAxNjA5IC0xMjcyIA0KTCAyMzQgLTEyNzIgDQpMIDIzNCAtMTAyNSANCkwgMTE0MSAtMTAyNSANCkwgMTE0MSA0MDg4IA0KTCAyMzQgNDA4OCANCkwgMjM0IDQzMzQgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtMzgiIGQ9Ik0gODM4IDQyMzggDQpMIDI2MjggNDIzOCANCkwgMjU5NCA0MTIyIA0KUSAyMzE5IDQxMTYgMjIwMSA0MDcwIA0KUSAyMDg0IDQwMjUgMjAwMSAzOTA5IA0KUSAxOTE5IDM3OTQgMTgyOCAzNDgxIA0KTCAxMzk0IDE5ODEgDQpRIDEyNzggMTU4MSAxMjU2IDE0NTYgDQpRIDEyMTkgMTI1NiAxMjE5IDEwODQgDQpRIDEyMTkgNzA2IDE1MDAgNDQ1IA0KUSAxNzgxIDE4NCAyMjUzIDE4NCANClEgMjU2MyAxODQgMjgwNiAzMDEgDQpRIDMwNTAgNDE5IDMyMjMgNjMwIA0KUSAzMzk3IDg0MSAzNTQwIDEyMDIgDQpRIDM2ODQgMTU2MyAzNzk0IDE5MzEgDQpMIDQxNjkgMzIyNSANClEgNDMwOSAzNzA5IDQzMDkgMzg1NiANClEgNDMwOSAzOTU5IDQyMTQgNDAzMiANClEgNDExOSA0MTA2IDM4MjggNDEyMiANCkwgMzg2MyA0MjM4IA0KTCA1MjA5IDQyMzggDQpMIDUxNzUgNDEyMiANClEgNDkzOCA0MTIyIDQ4MzMgNDA3NiANClEgNDcyOCA0MDMxIDQ2NTAgMzkxNCANClEgNDU3MiAzNzk3IDQ0NzUgMzQ1OSANCkwgNDAxNiAxODg4IA0KUSAzNzk0IDExMjIgMzU5NyA3NTggDQpRIDM0MDAgMzk0IDMwMjYgMTQ4IA0KUSAyNjUzIC05NyAyMTAwIC05NyANClEgMTQ2MyAtOTcgMTA3MiAyNjQgDQpRIDY4MSA2MjUgNjgxIDEwOTEgDQpRIDY4MSAxMjc1IDcxNiAxNDg0IA0KUSA3NDEgMTYyMiA4MzggMTk1OSANCkwgMTI1NiAzNDAwIA0KUSAxMzQ0IDM3MTMgMTM0NCAzODU5IA0KUSAxMzQ0IDM5NzUgMTI1NSA0MDM5IA0KUSAxMTY2IDQxMDMgODA2IDQxMjIgDQpMIDgzOCA0MjM4IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgIDwvZGVmcz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtMzkiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC00ZSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjEuMDgzOTg0IC0xMi42OTM3NSkgc2NhbGUoMC43KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWUiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEwMi4yNjM2NzIgLTEyLjY5Mzc1KSBzY2FsZSgwLjcpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE1MS44NTE1NjIgLTEyLjY5Mzc1KSBzY2FsZSgwLjcpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtYiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTg5LjA2MDkzNyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM2IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyMjIuMzYxNzE5IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1jIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyNzIuMzYxNzE5IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0yMCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMzIwLjEwNTg1OSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTMiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDM5MC45NDU3MDMgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTExIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg0NDAuOTQ1NzAzIDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xOCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNDY1Ljk0NTcwMyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtM2UiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDUxNS45NDU3MDMgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTEzIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg1NDkuMjQ2NDg0IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNTk5LjI0NjQ4NCAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTgiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDYyNC4yNDY0ODQgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC0zOSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjc0LjI0NjQ4NCAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTRlIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg3MzUuMzMwNDY5IC0xMi42OTM3NSkgc2NhbGUoMC43KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWIiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDc2OC42MDkxOCAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM2IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg4MDEuOTA5OTYxIDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1jIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg4NTEuOTA5OTYxIDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC00MCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODg1LjIxMDc0MiAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtZSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoOTMyLjk1NDg4MyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTMiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEwMDMuNzk0NzI3IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTA1My43OTQ3MjcgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTE4IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMDc4Ljc5NDcyNyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtM2UiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDExMjguNzk0NzI3IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xNCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTE2Mi4wOTU1MDggMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWUiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEyMjYuNTM4ODY3IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMyIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTI5Ny4zNzg3MTEgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTExIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMzQ3LjM3ODcxMSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTgiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEzNzIuMzc4NzExIDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtMzkiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE0MjIuMzc4NzExIDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtNGUiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE0ODMuNDYyNjk1IC0xMi42OTM3NSkgc2NhbGUoMC43KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWIiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE1MTYuNzQxNDA2IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtMzgiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE1NTAuMDQyMTg3IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1jIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxNjIyLjI1ODk4NCAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtNDAiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE2NTUuNTU5NzY2IDAuNTYyNSkiLz4NCiAgIDwvZz4NCiAgPC9nPg0KIDwvZz4NCjwvc3ZnPg0K)
![W_{k+1}(S)=\max[0.5W_k(S),1+0.5W_k(U)]](data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiIHN0YW5kYWxvbmU9Im5vIj8+DQo8IURPQ1RZUEUgc3ZnIFBVQkxJQyAiLS8vVzNDLy9EVEQgU1ZHIDEuMS8vRU4iDQogICJodHRwOi8vd3d3LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPg0KPHN2ZyB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgd2lkdGg9IjI2N3B0IiBoZWlnaHQ9IjE2cHQiIHZpZXdCb3g9IjAgMCAyNjcgMTYiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmVyc2lvbj0iMS4xIj4NCiA8bWV0YWRhdGE+DQogIDxyZGY6UkRGIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6Y2M9Imh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL25zIyIgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4NCiAgIDxjYzpXb3JrPg0KICAgIDxkYzp0eXBlIHJkZjpyZXNvdXJjZT0iaHR0cDovL3B1cmwub3JnL2RjL2RjbWl0eXBlL1N0aWxsSW1hZ2UiLz4NCiAgICA8ZGM6ZGF0ZT4yMDI2LTExLTEwVDIwOjMxOjQ2LjA1OTIwNDwvZGM6ZGF0ZT4NCiAgICA8ZGM6Zm9ybWF0PmltYWdlL3N2Zyt4bWw8L2RjOmZvcm1hdD4NCiAgICA8ZGM6Y3JlYXRvcj4NCiAgICAgPGNjOkFnZW50Pg0KICAgICAgPGRjOnRpdGxlPk1hdHBsb3RsaWIgdjMuMTEuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88L2RjOnRpdGxlPg0KICAgICA8L2NjOkFnZW50Pg0KICAgIDwvZGM6Y3JlYXRvcj4NCiAgIDwvY2M6V29yaz4NCiAgPC9yZGY6UkRGPg0KIDwvbWV0YWRhdGE+DQogPGRlZnM+DQogIDxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+KntzdHJva2UtbGluZWpvaW46IHJvdW5kOyBzdHJva2UtbGluZWNhcDogYnV0dH08L3N0eWxlPg0KIDwvZGVmcz4NCiA8ZyBpZD0iZmlndXJlXzEiPg0KICA8ZyBpZD0icGF0Y2hfMSI+DQogICA8cGF0aCBkPSJNIDAgMTYgDQpMIDI2NyAxNiANCkwgMjY3IDAgDQpMIDAgMCANCnoNCiIgc3R5bGU9ImZpbGw6ICNmZmZmZmYiLz4NCiAgPC9nPg0KICA8ZyBpZD0idGV4dF8xIj4NCiAgIDwhLS0gJFdfe2srMX0oUyk9XG1heFswLjVXX2soUyksMSswLjVXX2soVSldJCAtLT4NCiAgIDxnIHN0eWxlPSJmaWxsOiAjMTczZTQ2IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDEyKSBzY2FsZSgwLjE3IC0wLjE3KSI+DQogICAgPGRlZnM+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtM2EiIGQ9Ik0gODY2IC05NyANCkwgMTIzMSAzNDM0IA0KUSAxMjUzIDM2MzggMTI1MyAzNzQ3IA0KUSAxMjUzIDM5MjggMTE1MCA0MDIyIA0KUSAxMDQ3IDQxMTYgNzc4IDQxMjIgDQpMIDgxMyA0MjM4IA0KTCAyNDAwIDQyMzggDQpMIDIzNjYgNDEyMiANClEgMjA0NyA0MTI1IDE5MzAgNDAwOCANClEgMTgxMyAzODkxIDE3NjkgMzQzNCANCkwgMTUzNCAxMTQ3IA0KTCAzMTQ0IDM2MzEgDQpRIDMxNDcgMzY5NCAzMTQ3IDM3MzggDQpRIDMxNDcgMzkxMyAzMDQ0IDQwMDYgDQpRIDI5NDEgNDEwMCAyNjAwIDQxMjIgDQpMIDI2MjUgNDIzOCANCkwgNDM0MSA0MjM4IA0KTCA0MzA2IDQxMjIgDQpRIDQwMDkgNDExOSAzOTA2IDQwNzIgDQpRIDM4MzEgNDAzOCAzNzc5IDM5NjEgDQpRIDM3MjggMzg4NCAzNjk3IDM3MDAgDQpRIDM2ODggMzY0NyAzNjIwIDI5NzUgDQpRIDM1NTMgMjMwMyAzNDE5IDExNDcgDQpMIDQ3NTkgMzE4NCANClEgNDk3OCAzNTE5IDUwMzQgMzY1MyANClEgNTA5MSAzNzg4IDUwOTEgMzg4MSANClEgNTA5MSAzOTY5IDUwMTYgNDAzNiANClEgNDk0MSA0MTAzIDQ3NTkgNDEyMiANCkwgNDc4NCA0MjM4IA0KTCA2MDQxIDQyMzggDQpMIDYwMDkgNDEyMiANClEgNTg0NCA0MTA5IDU3MTYgNDA0MCANClEgNTU4OCAzOTcyIDU0MTMgMzc3OCANClEgNTMxMyAzNjY2IDUwMDAgMzE4NCANCkwgMjg3MiAtOTcgDQpMIDI3NTYgLTk3IA0KTCAzMTAwIDMxODQgDQpMIDk3MiAtOTcgDQpMIDg2NiAtOTcgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtNGUiIGQ9Ik0gMTgzMSA0NDQ0IA0KTCA5NDQgMTM3MiANCkwgMTMwMCAxNjc1IA0KUSAxODg0IDIxNjkgMjA1OSAyNDA2IA0KUSAyMTE2IDI0ODQgMjExNiAyNTI1IA0KUSAyMTE2IDI1NTMgMjA5NyAyNTc4IA0KUSAyMDc4IDI2MDMgMjAyMyAyNjE4IA0KUSAxOTY5IDI2MzQgMTgzMSAyNjM0IA0KTCAxNzIyIDI2MzQgDQpMIDE3NTYgMjc1MyANCkwgMzEyMiAyNzUzIA0KTCAzMDkxIDI2MzQgDQpRIDI4NTYgMjYyMiAyNjg2IDI1NjEgDQpRIDI1MTYgMjUwMCAyMzU2IDIzODEgDQpRIDIwNDEgMjE0NyAxOTQxIDIwNTYgDQpMIDE2NzIgMTgxOSANClEgMTc0MSAxNTk0IDE4NDEgMTE5MSANClEgMjAwOSA1MjIgMjA2OSAzOTcgDQpRIDIxMjggMjcyIDIxOTEgMjcyIA0KUSAyMjQ0IDI3MiAyMzE2IDMyNSANClEgMjQ3OCA0NDQgMjY2MyA3MDMgDQpMIDI3NzIgNjI1IA0KUSAyNDMxIDE5NyAyMjUwIDYxIA0KUSAyMDY5IC03NSAxOTE5IC03NSANClEgMTc4OCAtNzUgMTcxNiAyOCANClEgMTU5MSAyMTMgMTMwMCAxNDgxIA0KTCA4NzIgMTEwOSANCkwgNTUwIDAgDQpMIDYzIDAgDQpMIDExMDAgMzU2NiANCkwgMTE4OCAzODY5IA0KUSAxMjAzIDM5NDcgMTIwMyA0MDA2IA0KUSAxMjAzIDQwODEgMTE0NSA0MTMxIA0KUSAxMDg4IDQxODEgOTc4IDQxODEgDQpRIDkwNiA0MTgxIDc1MCA0MTUwIA0KTCA3NTAgNDI1OSANCkwgMTgzMSA0NDQ0IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtZSIgZD0iTSAxNjcyIDQ0MSANCkwgMTY3MiAyMDAwIA0KTCAxMTYgMjAwMCANCkwgMTE2IDIyNTYgDQpMIDE2NzIgMjI1NiANCkwgMTY3MiAzODA5IA0KTCAxOTIyIDM4MDkgDQpMIDE5MjIgMjI1NiANCkwgMzQ4NCAyMjU2IA0KTCAzNDg0IDIwMDAgDQpMIDE5MjIgMjAwMCANCkwgMTkyMiA0NDEgDQpMIDE2NzIgNDQxIA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtMTQiIGQ9Ik0gNzUwIDM4MjIgDQpMIDE3ODEgNDMyNSANCkwgMTg4NCA0MzI1IA0KTCAxODg0IDc0NyANClEgMTg4NCAzOTEgMTkxNCAzMDMgDQpRIDE5NDQgMjE2IDIwMzcgMTY5IA0KUSAyMTMxIDEyMiAyNDE5IDExNiANCkwgMjQxOSAwIA0KTCA4MjUgMCANCkwgODI1IDExNiANClEgMTEyNSAxMjIgMTIxMiAxNjcgDQpRIDEzMDAgMjEzIDEzMzQgMjg5IA0KUSAxMzY5IDM2NiAxMzY5IDc0NyANCkwgMTM2OSAzMDM0IA0KUSAxMzY5IDM0OTcgMTMzOCAzNjI4IA0KUSAxMzE2IDM3MjggMTI1OCAzNzc1IA0KUSAxMjAwIDM4MjIgMTExOSAzODIyIA0KUSAxMDAzIDM4MjIgNzk3IDM3MjUgDQpMIDc1MCAzODIyIA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtYiIgZD0iTSAxOTg4IC0xMjUzIA0KTCAxOTg4IC0xMzY5IA0KUSAxNTE2IC0xMTMxIDEyMDAgLTgxMyANClEgNzUwIC0zNTkgNTA2IDI1NiANClEgMjYzIDg3MiAyNjMgMTUzNCANClEgMjYzIDI1MDMgNzQxIDMzMDEgDQpRIDEyMTkgNDEwMCAxOTg4IDQ0NDQgDQpMIDE5ODggNDMxMyANClEgMTYwMyA0MTAwIDEzNTYgMzczMSANClEgMTEwOSAzMzYzIDk4NyAyNzk3IA0KUSA4NjYgMjIzMSA4NjYgMTYxNiANClEgODY2IDk0NyA5NjkgNDAwIA0KUSAxMDUwIC0zMSAxMTY1IC0yOTIgDQpRIDEyODEgLTU1MyAxNDc2IC03OTMgDQpRIDE2NzIgLTEwMzQgMTk4OCAtMTI1MyANCnoNCiIgdHJhbnNmb3JtPSJzY2FsZSgwLjAxNTYyNSkiLz4NCiAgICAgPHBhdGggaWQ9IlRpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC0zNiIgZD0iTSA0NCAtOTcgDQpMIDM5NCAxNDc1IA0KTCA1MDAgMTQ3NSANClEgNDc4IDEyNDcgNDc4IDEwOTcgDQpRIDQ3OCA2NjkgNzUxIDQwMCANClEgMTAyNSAxMzEgMTQ2NiAxMzEgDQpRIDE4NzUgMTMxIDIwODcgMzgyIA0KUSAyMzAwIDYzNCAyMzAwIDk2MyANClEgMjMwMCAxMTc1IDIyMDMgMTM1MCANClEgMjA1NiAxNjA5IDE0MTkgMjI2MyANClEgMTEwOSAyNTc1IDEwMjIgMjcyOCANClEgODc4IDI5ODEgODc4IDMyNTkgDQpRIDg3OCAzNzAzIDEyMDkgNDAxOCANClEgMTU0MSA0MzM0IDIwNTkgNDMzNCANClEgMjIzNCA0MzM0IDIzOTEgNDMwMCANClEgMjQ4OCA0MjgxIDI3NDQgNDE3NSANClEgMjkyNSA0MTAzIDI5NDQgNDA5NyANClEgMjk4OCA0MDg4IDMwNDEgNDA4OCANClEgMzEzMSA0MDg4IDMxOTcgNDEzNCANClEgMzI2MyA0MTgxIDMzNTAgNDMzNCANCkwgMzQ2OSA0MzM0IA0KTCAzMTQ0IDI5MjggDQpMIDMwMzggMjkyOCANClEgMzA1MCAzMTE2IDMwNTAgMzIzMSANClEgMzA1MCAzNjA5IDI4MDAgMzg1MCANClEgMjU1MCA0MDkxIDIxNDEgNDA5MSANClEgMTgxNiA0MDkxIDE2MTIgMzkwMCANClEgMTQwOSAzNzA5IDE0MDkgMzQ1OSANClEgMTQwOSAzMjQxIDE1MzkgMzA0MiANClEgMTY2OSAyODQ0IDIxMzYgMjM5NSANClEgMjYwMyAxOTQ3IDI3NDAgMTY5NSANClEgMjg3OCAxNDQ0IDI4NzggMTE1OSANClEgMjg3OCA4MzggMjcwOCA1MzYgDQpRIDI1MzggMjM0IDIyMTkgNjggDQpRIDE5MDAgLTk3IDE1MjIgLTk3IA0KUSAxMzM0IC05NyAxMTcxIC02MiANClEgMTAwOSAtMjggNjUzIDEwNiANClEgNTMxIDE1MyA0NTAgMTUzIA0KUSAyNjYgMTUzIDE2MyAtOTcgDQpMIDQ0IC05NyANCnoNCiIgdHJhbnNmb3JtPSJzY2FsZSgwLjAxNTYyNSkiLz4NCiAgICAgPHBhdGggaWQ9IlRpbWVzTmV3Um9tYW5QU01ULWMiIGQ9Ik0gMTQ0IDQzMTMgDQpMIDE0NCA0NDQ0IA0KUSA2MTkgNDIwOSA5MzQgMzg5MSANClEgMTM4MSAzNDM0IDE2MjUgMjgyMCANClEgMTg2OSAyMjA2IDE4NjkgMTU0MSANClEgMTg2OSA1NzIgMTM5MiAtMjI2IA0KUSA5MTYgLTEwMjUgMTQ0IC0xMzY5IA0KTCAxNDQgLTEyNTMgDQpRIDUyOCAtMTAzOCA3NzYgLTY3MCANClEgMTAyNSAtMzAzIDExNDUgMjY0IA0KUSAxMjY2IDgzMSAxMjY2IDE0NDcgDQpRIDEyNjYgMjExMyAxMTYzIDI2NjMgDQpRIDEwODQgMzA5NCA5NjcgMzM1MyANClEgODUwIDM2MTMgNjU2IDM4NTMgDQpRIDQ2MyA0MDk0IDE0NCA0MzEzIA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtMjAiIGQ9Ik0gMTE2IDI3NTkgDQpMIDM0ODggMjc1OSANCkwgMzQ4OCAyNTAzIA0KTCAxMTYgMjUwMyANCkwgMTE2IDI3NTkgDQp6DQpNIDExNiAxNzQxIA0KTCAzNDg4IDE3NDEgDQpMIDM0ODggMTQ4NCANCkwgMTE2IDE0ODQgDQpMIDExNiAxNzQxIA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtNTAiIGQ9Ik0gMTA1MCAyMzM4IA0KUSAxMzYzIDI2NTAgMTQxOSAyNjk3IA0KUSAxNTU5IDI4MTYgMTcyMSAyODgxIA0KUSAxODg0IDI5NDcgMjA0NCAyOTQ3IA0KUSAyMzEzIDI5NDcgMjUwNiAyNzkwIA0KUSAyNzAwIDI2MzQgMjc2NiAyMzM4IA0KUSAzMDg4IDI3MTMgMzMwOSAyODMwIA0KUSAzNTMxIDI5NDcgMzc2NiAyOTQ3IA0KUSAzOTk0IDI5NDcgNDE3MCAyODMwIA0KUSA0MzQ3IDI3MTMgNDQ1MCAyNDQ3IA0KUSA0NTE5IDIyNjYgNDUxOSAxODc4IA0KTCA0NTE5IDY0NyANClEgNDUxOSAzNzggNDU1OSAyNzggDQpRIDQ1OTEgMjA5IDQ2NzUgMTYxIA0KUSA0NzU5IDExMyA0OTUwIDExMyANCkwgNDk1MCAwIA0KTCAzNTM4IDAgDQpMIDM1MzggMTEzIA0KTCAzNTk3IDExMyANClEgMzc4MSAxMTMgMzg4NCAxODQgDQpRIDM5NTYgMjM0IDM5ODggMzQ0IA0KUSA0MDAwIDM5NyA0MDAwIDY0NyANCkwgNDAwMCAxODc4IA0KUSA0MDAwIDIyMjggMzkxNiAyMzcyIA0KUSAzNzk0IDI1NzIgMzUyNSAyNTcyIA0KUSAzMzU5IDI1NzIgMzE5MiAyNDg5IA0KUSAzMDI1IDI0MDYgMjc4OCAyMTgxIA0KTCAyNzgxIDIxNDcgDQpMIDI3ODggMjAxMyANCkwgMjc4OCA2NDcgDQpRIDI3ODggMzUzIDI4MjAgMjgxIA0KUSAyODUzIDIwOSAyOTQzIDE2MSANClEgMzAzNCAxMTMgMzI1MyAxMTMgDQpMIDMyNTMgMCANCkwgMTgwNiAwIA0KTCAxODA2IDExMyANClEgMjA0NCAxMTMgMjEzMyAxNjkgDQpRIDIyMjIgMjI1IDIyNTYgMzM4IA0KUSAyMjcyIDM5MSAyMjcyIDY0NyANCkwgMjI3MiAxODc4IA0KUSAyMjcyIDIyMjggMjE2OSAyMzgxIA0KUSAyMDMxIDI1ODEgMTc4NCAyNTgxIA0KUSAxNjE2IDI1ODEgMTQ1MCAyNDkxIA0KUSAxMTkxIDIzNTMgMTA1MCAyMTgxIA0KTCAxMDUwIDY0NyANClEgMTA1MCAzNjYgMTA4OSAyODEgDQpRIDExMjggMTk3IDEyMDQgMTU1IA0KUSAxMjgxIDExMyAxNTE2IDExMyANCkwgMTUxNiAwIA0KTCAxMDAgMCANCkwgMTAwIDExMyANClEgMjk3IDExMyAzNzUgMTU1IA0KUSA0NTMgMTk3IDQ5MyAyODkgDQpRIDUzNCAzODEgNTM0IDY0NyANCkwgNTM0IDE3NDEgDQpRIDUzNCAyMjEzIDUwNiAyMzUwIA0KUSA0ODQgMjQ1MyA0MzcgMjQ5MiANClEgMzkxIDI1MzEgMzA5IDI1MzEgDQpRIDIyMiAyNTMxIDEwMCAyNDg0IA0KTCA1MyAyNTk3IA0KTCA5MTYgMjk0NyANCkwgMTA1MCAyOTQ3IA0KTCAxMDUwIDIzMzggDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC00NCIgZD0iTSAxODIyIDQxMyANClEgMTM4MSA3MiAxMjY5IDE5IA0KUSAxMTAwIC01OSA5MDkgLTU5IA0KUSA2MTMgLTU5IDQyMCAxNDQgDQpRIDIyOCAzNDcgMjI4IDY3OCANClEgMjI4IDg4OCAzMjIgMTA0MSANClEgNDUwIDEyNTMgNzY3IDE0NDAgDQpRIDEwODQgMTYyOCAxODIyIDE4OTcgDQpMIDE4MjIgMjAwOSANClEgMTgyMiAyNDM4IDE2ODYgMjU5NyANClEgMTU1MCAyNzU2IDEyOTEgMjc1NiANClEgMTA5NCAyNzU2IDk3OCAyNjUwIA0KUSA4NTkgMjU0NCA4NTkgMjQwNiANCkwgODY2IDIyMjUgDQpRIDg2NiAyMDgxIDc5MiAyMDAzIA0KUSA3MTkgMTkyNSA2MDAgMTkyNSANClEgNDg0IDE5MjUgNDExIDIwMDYgDQpRIDMzOCAyMDg4IDMzOCAyMjI4IA0KUSAzMzggMjQ5NyA2MTMgMjcyMiANClEgODg4IDI5NDcgMTM4NCAyOTQ3IA0KUSAxNzY2IDI5NDcgMjAwOSAyODE5IA0KUSAyMTk0IDI3MjIgMjI4MSAyNTE2IA0KUSAyMzM4IDIzODEgMjMzOCAxOTY2IA0KTCAyMzM4IDk5NCANClEgMjMzOCA1ODQgMjM1MyA0OTIgDQpRIDIzNjkgNDAwIDI0MDUgMzY5IA0KUSAyNDQxIDMzOCAyNDg4IDMzOCANClEgMjUzOCAzMzggMjU3NSAzNTkgDQpRIDI2NDEgNDAwIDI4MjggNTg4IA0KTCAyODI4IDQxMyANClEgMjQ3OCAtNTYgMjE1OSAtNTYgDQpRIDIwMDYgLTU2IDE5MTUgNTAgDQpRIDE4MjUgMTU2IDE4MjIgNDEzIA0Keg0KTSAxODIyIDYxNiANCkwgMTgyMiAxNzA2IA0KUSAxMzUwIDE1MTkgMTIxMyAxNDQxIA0KUSA5NjYgMTMwMyA4NTkgMTE1MyANClEgNzUzIDEwMDMgNzUzIDgyNSANClEgNzUzIDYwMCA4ODcgNDUxIA0KUSAxMDIyIDMwMyAxMTk3IDMwMyANClEgMTQzNCAzMDMgMTgyMiA2MTYgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC01YiIgZD0iTSA4NCAyODYzIA0KTCAxNDMxIDI4NjMgDQpMIDE0MzEgMjc0NyANClEgMTMwMyAyNzQ3IDEyNTEgMjcwMyANClEgMTIwMCAyNjU5IDEyMDAgMjU4OCANClEgMTIwMCAyNTEzIDEzMDkgMjM1NiANClEgMTM0NCAyMzA2IDE0MTMgMjIwMCANCkwgMTYxNiAxODc1IA0KTCAxODUwIDIyMDAgDQpRIDIwNzUgMjUwOSAyMDc1IDI1OTEgDQpRIDIwNzUgMjY1NiAyMDIyIDI3MDEgDQpRIDE5NjkgMjc0NyAxODUwIDI3NDcgDQpMIDE4NTAgMjg2MyANCkwgMjgxOSAyODYzIA0KTCAyODE5IDI3NDcgDQpRIDI2NjYgMjczOCAyNTUzIDI2NjMgDQpRIDI0MDAgMjU1NiAyMTM0IDIyMDAgDQpMIDE3NDQgMTY3OCANCkwgMjQ1NiA2NTMgDQpRIDI3MTkgMjc1IDI4MzEgMTk4IA0KUSAyOTQ0IDEyMiAzMTIyIDExMyANCkwgMzEyMiAwIA0KTCAxNzcyIDAgDQpMIDE3NzIgMTEzIA0KUSAxOTEzIDExMyAxOTkxIDE3NSANClEgMjA1MCAyMTkgMjA1MCAyOTEgDQpRIDIwNTAgMzYzIDE4NTAgNjUzIA0KTCAxNDMxIDEyNjYgDQpMIDk3MiA2NTMgDQpRIDc1OSAzNjkgNzU5IDMxNiANClEgNzU5IDI0MSA4MjkgMTgwIA0KUSA5MDAgMTE5IDEwNDEgMTEzIA0KTCAxMDQxIDAgDQpMIDEwNiAwIA0KTCAxMDYgMTEzIA0KUSAyMTkgMTI4IDMwMyAxOTEgDQpRIDQyMiAyODEgNzAzIDY1MyANCkwgMTMwMyAxNDUwIA0KTCA3NTkgMjIzOCANClEgNTI4IDI1NzUgNDAxIDI2NjEgDQpRIDI3NSAyNzQ3IDg0IDI3NDcgDQpMIDg0IDI4NjMgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC0zZSIgZD0iTSAxOTAwIC0xMjY5IA0KTCA1MjUgLTEyNjkgDQpMIDUyNSA0MzM0IA0KTCAxOTAwIDQzMzQgDQpMIDE5MDAgNDA4OCANCkwgOTk0IDQwODggDQpMIDk5NCAtMTAyNSANCkwgMTkwMCAtMTAyNSANCkwgMTkwMCAtMTI2OSANCnoNCiIgdHJhbnNmb3JtPSJzY2FsZSgwLjAxNTYyNSkiLz4NCiAgICAgPHBhdGggaWQ9IlRpbWVzTmV3Um9tYW5QU01ULTEzIiBkPSJNIDIzMSAyMDk0IA0KUSAyMzEgMjgxOSA0NTAgMzM0MiANClEgNjY5IDM4NjYgMTAzMSA0MTIyIA0KUSAxMzEzIDQzMjUgMTYxMyA0MzI1IA0KUSAyMTAwIDQzMjUgMjQ4OCAzODI4IA0KUSAyOTcyIDMyMTMgMjk3MiAyMTU5IA0KUSAyOTcyIDE0MjIgMjc1OSA5MDYgDQpRIDI1NDcgMzkxIDIyMTcgMTU4IA0KUSAxODg4IC03NSAxNTgxIC03NSANClEgOTc1IC03NSA1NzIgNjQxIA0KUSAyMzEgMTI0NCAyMzEgMjA5NCANCnoNCk0gODQ0IDIwMTYgDQpRIDg0NCAxMTQxIDEwNTkgNTg4IA0KUSAxMjM4IDEyMiAxNTkxIDEyMiANClEgMTc1OSAxMjIgMTk0MCAyNzMgDQpRIDIxMjIgNDI1IDIyMTYgNzgxIA0KUSAyMzU5IDEzMTkgMjM1OSAyMjk3IA0KUSAyMzU5IDMwMjIgMjIwOSAzNTA2IA0KUSAyMDk3IDM4NjYgMTkxOSA0MDE2IA0KUSAxNzkxIDQxMTkgMTYwOSA0MTE5IA0KUSAxMzk3IDQxMTkgMTIzMSAzOTI4IA0KUSAxMDA2IDM2NjkgOTI1IDMxMTIgDQpRIDg0NCAyNTU2IDg0NCAyMDE2IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtMTEiIGQ9Ik0gODAwIDYwNiANClEgOTQ3IDYwNiAxMDQ3IDUwNCANClEgMTE0NyA0MDMgMTE0NyAyNTkgDQpRIDExNDcgMTE2IDEwNDUgMTQgDQpRIDk0NCAtODggODAwIC04OCANClEgNjU2IC04OCA1NTQgMTQgDQpRIDQ1MyAxMTYgNDUzIDI1OSANClEgNDUzIDQwNiA1NTQgNTA2IA0KUSA2NTYgNjA2IDgwMCA2MDYgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC0xOCIgZD0iTSAyNzc4IDQyMzggDQpMIDI1MzQgMzcwNiANCkwgMTI1OSAzNzA2IA0KTCA5ODEgMzEzOCANClEgMTgwOSAzMDE2IDIyOTQgMjUyMiANClEgMjcwOSAyMDk3IDI3MDkgMTUyMiANClEgMjcwOSAxMTg4IDI1NzMgOTAzIA0KUSAyNDM4IDYxOSAyMjMxIDQxOSANClEgMjAyNSAyMTkgMTc3MiA5NyANClEgMTQxMyAtNzUgMTAzNCAtNzUgDQpRIDY1MyAtNzUgNDc5IDU0IA0KUSAzMDYgMTg0IDMwNiAzNDEgDQpRIDMwNiA0MjggMzc4IDQ5NSANClEgNDUwIDU2MyA1NTkgNTYzIA0KUSA2NDEgNTYzIDcwMiA1MzggDQpRIDc2MyA1MTMgOTA5IDQwOSANClEgMTE0NCAyNDcgMTM4NCAyNDcgDQpRIDE3NTAgMjQ3IDIwMjYgNTIzIA0KUSAyMzAzIDgwMCAyMzAzIDExOTcgDQpRIDIzMDMgMTU4MSAyMDU2IDE5MTQgDQpRIDE4MDkgMjI0NyAxMzc1IDI0MjggDQpRIDEwMzQgMjU2OSA0NDcgMjU5MSANCkwgMTI1OSA0MjM4IA0KTCAyNzc4IDQyMzggDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFNNVC1mIiBkPSJNIDM0NCAtMTA2NiANCkwgMzQ0IC05MjggDQpRIDY2NiAtODIyIDg0MiAtNTk4IA0KUSAxMDE5IC0zNzUgMTAxOSAtMTI1IA0KUSAxMDE5IC02NiA5OTEgLTI1IA0KUSA5NjkgMyA5NDcgMyANClEgOTEzIDMgNzk3IC01OSANClEgNzQxIC04OCA2NzggLTg4IA0KUSA1MjUgLTg4IDQzNCAzIA0KUSAzNDQgOTQgMzQ0IDI1MyANClEgMzQ0IDQwNiA0NjEgNTE1IA0KUSA1NzggNjI1IDc0NyA2MjUgDQpRIDk1MyA2MjUgMTExNCA0NDUgDQpRIDEyNzUgMjY2IDEyNzUgLTMxIA0KUSAxMjc1IC0zNTMgMTA1MSAtNjI5IA0KUSA4MjggLTkwNiAzNDQgLTEwNjYgDQp6DQoiIHRyYW5zZm9ybT0ic2NhbGUoMC4wMTU2MjUpIi8+DQogICAgIDxwYXRoIGlkPSJUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtMzgiIGQ9Ik0gODM4IDQyMzggDQpMIDI2MjggNDIzOCANCkwgMjU5NCA0MTIyIA0KUSAyMzE5IDQxMTYgMjIwMSA0MDcwIA0KUSAyMDg0IDQwMjUgMjAwMSAzOTA5IA0KUSAxOTE5IDM3OTQgMTgyOCAzNDgxIA0KTCAxMzk0IDE5ODEgDQpRIDEyNzggMTU4MSAxMjU2IDE0NTYgDQpRIDEyMTkgMTI1NiAxMjE5IDEwODQgDQpRIDEyMTkgNzA2IDE1MDAgNDQ1IA0KUSAxNzgxIDE4NCAyMjUzIDE4NCANClEgMjU2MyAxODQgMjgwNiAzMDEgDQpRIDMwNTAgNDE5IDMyMjMgNjMwIA0KUSAzMzk3IDg0MSAzNTQwIDEyMDIgDQpRIDM2ODQgMTU2MyAzNzk0IDE5MzEgDQpMIDQxNjkgMzIyNSANClEgNDMwOSAzNzA5IDQzMDkgMzg1NiANClEgNDMwOSAzOTU5IDQyMTQgNDAzMiANClEgNDExOSA0MTA2IDM4MjggNDEyMiANCkwgMzg2MyA0MjM4IA0KTCA1MjA5IDQyMzggDQpMIDUxNzUgNDEyMiANClEgNDkzOCA0MTIyIDQ4MzMgNDA3NiANClEgNDcyOCA0MDMxIDQ2NTAgMzkxNCANClEgNDU3MiAzNzk3IDQ0NzUgMzQ1OSANCkwgNDAxNiAxODg4IA0KUSAzNzk0IDExMjIgMzU5NyA3NTggDQpRIDM0MDAgMzk0IDMwMjYgMTQ4IA0KUSAyNjUzIC05NyAyMTAwIC05NyANClEgMTQ2MyAtOTcgMTA3MiAyNjQgDQpRIDY4MSA2MjUgNjgxIDEwOTEgDQpRIDY4MSAxMjc1IDcxNiAxNDg0IA0KUSA3NDEgMTYyMiA4MzggMTk1OSANCkwgMTI1NiAzNDAwIA0KUSAxMzQ0IDM3MTMgMTM0NCAzODU5IA0KUSAxMzQ0IDM5NzUgMTI1NSA0MDM5IA0KUSAxMTY2IDQxMDMgODA2IDQxMjIgDQpMIDgzOCA0MjM4IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgICA8cGF0aCBpZD0iVGltZXNOZXdSb21hblBTTVQtNDAiIGQ9Ik0gMjM0IDQzMzQgDQpMIDE2MDkgNDMzNCANCkwgMTYwOSAtMTI3MiANCkwgMjM0IC0xMjcyIA0KTCAyMzQgLTEwMjUgDQpMIDExNDEgLTEwMjUgDQpMIDExNDEgNDA4OCANCkwgMjM0IDQwODggDQpMIDIzNCA0MzM0IA0Keg0KIiB0cmFuc2Zvcm09InNjYWxlKDAuMDE1NjI1KSIvPg0KICAgIDwvZGVmcz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtM2EiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC00ZSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODMuMzAwNzgxIC0xMi42OTM3NSkgc2NhbGUoMC43KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWUiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEyNC40ODA0NjkgLTEyLjY5Mzc1KSBzY2FsZSgwLjcpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE3NC4wNjgzNTkgLTEyLjY5Mzc1KSBzY2FsZSgwLjcpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtYiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjExLjI3NzczNCAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM2IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyNDQuNTc4NTE2IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1jIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyOTQuNTc4NTE2IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0yMCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMzQyLjMyMjY1NiAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtNTAiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDQxMy4xNjI1IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC00NCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNDkwLjk0NTcwMyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtNWIiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDUzNS4zMzA0NjkgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTNlIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg1ODUuMzMwNDY5IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMyIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjE4LjYzMTI1IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjY4LjYzMTI1IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xOCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjkzLjYzMTI1IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFMtSXRhbGljTVQtM2EiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDc0My42MzEyNSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTRlIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg4MjYuOTMyMDMxIC0xMi42OTM3NSkgc2NhbGUoMC43KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULWIiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDg2MC4yMTA3NDIgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVC0zNiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODkzLjUxMTUyMyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtYyIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoOTQzLjUxMTUyMyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtZiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoOTc2LjgxMjMwNSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDEwMTYuMjU1NjY0IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1lIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMDgwLjY5OTAyMyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtMTMiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDExNTEuNTM4ODY3IDAuNTYyNSkiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC0xMSIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTIwMS41Mzg4NjcgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTE4IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjI2LjUzODg2NyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTNhIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjc2LjUzODg2NyAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTRlIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMzU5LjgzOTY0OCAtMTIuNjkzNzUpIHNjYWxlKDAuNykiLz4NCiAgICA8dXNlIHhsaW5rOmhyZWY9IiNUaW1lc05ld1JvbWFuUFNNVC1iIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMzkzLjExODM1OSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTLUl0YWxpY01ULTM4IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxNDI2LjQxOTE0MSAwLjU2MjUpIi8+DQogICAgPHVzZSB4bGluazpocmVmPSIjVGltZXNOZXdSb21hblBTTVQtYyIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTQ5OC42MzU5MzcgMC41NjI1KSIvPg0KICAgIDx1c2UgeGxpbms6aHJlZj0iI1RpbWVzTmV3Um9tYW5QU01ULTQwIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxNTMxLjkzNjcxOSAwLjU2MjUpIi8+DQogICA8L2c+DQogIDwvZz4NCiA8L2c+DQo8L3N2Zz4NCg==)
V₁(S)=0.5，V₁(U)=0.5×2+0.5×(-1)=0.5。第二轮 S=0.5×0.25+0.5×1.25=0.75；U=1+0.5(-1+0.25)=0.625。
W₁=(max(0,1),max(2,-1))=(1,2)。W₂=(max(0.5,2),max(2,-0.5))=(2,2)。
Q(S,a)=0.5×0.75=0.375；Q(S,b)=1+0.5×0.625=1.3125；Q(U,c)=2；Q(U,d)=-1+0.5×0.75=-0.625。选 b、c。
V 的每轮都在问“随机选动作的策略平均如何”；W 在问“每一步选择最佳动作能如何”。同一个环境可以有很多策略，价值也因此不同。V 在评估 π，W 在逼近 v*，不能把两者当成同一算法的两种记号。
轨迹A得1→A得2→B得3→T；γ=0.5、α=0.5、初值A=2/B=1。比较首次常数步长MC、按时间顺序每次常数步长MC、在线TD，并解释为何不是回报算术平均。
先为三次访问算完整回报。每种方法重新取初值；每次访问MC按题给顺序连续更新A两次；TD第一条是A→A，目标读取本次更新前的A。
An episode is A --1→ A --2→ B --3→ T. γ=0.5. Initialize V(A)=2,V(B)=1 and use constant α=0.5. Reset to these initial values for each method. Every-visit MC updates are applied in chronological visit order after the episode has ended.
(a) Compute G₀,G₁,G₂. [2]
(b) Apply first-visit constant-step MC. [2]
(c) Apply every-visit constant-step MC, showing both A updates. [2]
(d) Apply online TD(0), showing all three updates. [3]
(e) Why is the every-visit result not simply the arithmetic mean of A’s two returns? [1]
和其他卷的 fresh sample mean 不同，这里 α 固定为 0.5，还保留了初始值的影响。读题时先圈出这些约定。
G₂=3，G₁=2+0.5×3=3.5，G₀=1+0.5×3.5=2.75。
A=2+0.5(2.75-2)=2.375，B=1+0.5(3-1)=2。
A 第一次到 2.375；第二次用回报 3.5，A=2.375+0.5(3.5-2.375)=2.9375。B 仍为 2。
第一转移 A→A：目标 1+0.5×2=2，所以 A=2 不变。读取的是更新前的当前 A，不是把新值当未知量联立求解。
第二转移 A→B：目标 2+0.5×1=2.5，A=2+0.5(2.5-2)=2.25。第三转移 B→T：B=1+0.5(3-1)=2。
最终 A=0.25×初值2+0.25×2.75+0.5×3.5=2.9375；两个回报的权重不相同，而且初值仍占 0.25。算术平均 (2.75+3.5)/2=3.125 是另一种估计方法。
旧Q=3、奖励0、γ=0.9、α=0.1；在线(2,6)、目标(7,4)、实际a₁、ε=0.2。计算行为概率、四种目标和终止版本更新。
把当前值3与两张下一状态表分开，避免把“目标网络值”误当最终训练目标。每次先找该算法读取哪个数，乘γ加奖励，再在要求时做α更新。
Q(S,A)=3, r=0, γ=0.9, α=0.1. The transition is nonterminal. Online next-state values for (a₁,a₂) are (2,6), target-network values are (7,4), and the actual next action is a₁. Behaviour is ε-greedy with ε=0.2. Run each algorithm independently.
(a) Compute behaviour action probabilities. [2]
(b) Compute the SARSA target and tabular update. [2]
(c) Compute the Q-learning target and tabular update. [2]
(d) Compute DQN and Double DQN targets; state which network selects and which evaluates the next action. [3]
(e) If this transition is terminal, compute its target and tabular update. [1]
a₂ 唯一最大，概率 1-0.2+0.2/2=0.9；a₁ 为 0.1。实际选到小概率 a₁ 完全可能，不是题目矛盾。
用实际 a₁ 的在线值 2：y=0+0.9×2=1.8；新 Q=3+0.1(1.8-3)=2.88。
用在线最大 6：y=5.4；新 Q=3+0.1(5.4-3)=3.24。
DQN 用目标网络选 a₁，且用它评价为 7，所以 y=6.3。Double 用在线网络选 a₂，再读目标网络 a₂=4，所以 y=3.6。它们都不直接采用“实际下一动作 a₁”作为 SARSA 式约束。
y=0，新 Q=3+0.1(0-3)=2.7。不是把 Q 立刻设为零，因为学习率仅 0.1。
S有short得4结束、long得0去U、wait得1回S；U有collect得6结束、back得0回S。γ=0.5，旧策略wait/back。要评估、比较全部五个动作、验证改进策略最优并做两轮价值迭代。
动作数量并不相同：S三个、U两个。先列完整候选清单避免漏算wait；每次改策略后再用新的未来价值重新检查所有候选。
γ=0.5. At S, short gives reward 4 and terminates; long gives reward 0 and moves to U; wait gives reward 1 and returns to S. At U, collect gives reward 6 and terminates; back gives reward 0 and moves to S. Start with π₀=(wait,back).
(a) Evaluate π₀. [2]
(b) Compute all five qπ₀ values and obtain π₁ by greedy improvement. [3]
(c) Evaluate π₁ and prove it is optimal by checking every action. [3]
(d) From V₀=(0,0), compute V₁,V₂ with synchronous value iteration. [2]
S 有三个候选动作，U 有两个。每个都算一遍，再比较最大值，不能漏掉 wait。
vS=1+0.5vS，得 2；vU=0.5×2=1。
S: short=4，long=0.5×1=0.5，wait=1+0.5×2=2；U: collect=6，back=0.5×2=1。因此 π₁=(short,collect)。
新策略一步终止，vπ₁=(4,6)。再检查 S 的 short=4、long=0.5×6=3、wait=1+0.5×4=3；U 的 collect=6、back=0.5×4=2。选择仍为 short、collect，满足最优方程。
V₁(S)=max(4,0,1)=4，V₁(U)=max(6,0)=6。V₂(S)=max(4,3,3)=4，V₂(U)=max(6,2)=6。此例第一轮数值恰好就等于固定点，但不能据此说所有价值迭代都一轮收敛。