A robot must collect a key before opening a door. Which state representation most directly fixes the missing information if position alone is not Markov?
A. Position only, with a smaller learning rate.
B. Position together with whether the key is held.
In REINFORCE, why can a state-dependent, action-independent baseline be subtracted from the return without changing the expected policy-gradient estimator?
A. The baseline must equal zero.
B. The expected score function at a fixed state is zero.
C. Every sampled gradient becomes zero.
D. The rewards are replaced by the baseline.
下面跟着题目逐步算
正确答案:B
固定状态 s 后,对动作分布求和:Σₐπ(a|s)∇logπ(a|s)=Σₐ∇π(a|s)=∇1=0。基线 b(s) 与动作无关,可以从求和中提出,因此被减掉的那一项期望为零。
A 不要求基线为零;C 说的是每个样本,实际上只有附加基线项的期望为零;D 不是替换环境奖励,而是调整梯度权重。
Two states have similar feature vectors in a linear value approximator. Updating the weights using one state may affect the estimate of the other state. This illustrates:
Explain prediction and control. Describe the two stages of policy iteration, and explain why a greedy policy improvement uses argmax rather than max. [5 points]
下面跟着题目逐步算
题目在问什么
先分清“评价已有做法”和“寻找更好做法”,再说明策略迭代怎样交替完成它们。
五个得分点(各 1 分)
① Prediction 在固定 π 下估计 vπ 或 qπ;它不负责主动换成另一策略。② Control 寻找最优策略或最优价值。
State both GLIE conditions. Is GLIE itself a learning update or an action-selection formula? Discuss fixed ε = 0.1 and explain why merely requiring εₜ → 0 is insufficient. [5 points]
下面跟着题目逐步算
先说结论
GLIE 是一组关于行为策略序列的长期要求,不是 MC 的别名,也不是独立的 Q 更新公式。
评分与推理
① Infinite exploration(1 分):在收敛定理所要求的状态动作范围内,各状态动作对被无限次访问。② Greedy in the limit(1 分):行为策略在极限中把概率集中到当前价值估计的贪心动作上。
③ 分类(1 分):MC、SARSA 是学习方式;ε-greedy 是选动作的方法;GLIE 是探索与极限行为的性质。
Give the standard DQN and Double DQN nonterminal targets. Explain which network selects and evaluates the next action in each case, and why Double DQN can reduce overestimation. [5 points]
Describe proportional prioritized experience replay. State a priority, a sampling probability and an importance-sampling weight. Explain the roles of the small positive priority offset and the correction exponent β. [5 points]
For an undiscounted episodic task, write a REINFORCE update with a state baseline. Then describe the actor and critic, and give a one-step TD-error signal commonly used by the actor. [5 points]
下面跟着题目逐步算
把两个角色分开
Actor 决定动作概率;critic 评价状态或动作有多好。REINFORCE 用完整回报作反馈,actor–critic 常用 critic 产生的自举信号。
There are nonterminal states A, B and terminal state T. γ=0.5. From A, the immediate reward is always 2; the next state is A or B with probability 0.5 each. From B, the reward is always -1; the next state is A or T with probability 0.5 each. T is absorbing with zero reward.
(a) Write the transition matrix in the order (A,B,T) and the expected reward vector. [2]
(b) For the observed episode A → B → T, compute G₀ and G₁. [2]
(c) Write and solve the Bellman equations for v(A),v(B). [2]
(d) From V₀=(0,0), calculate the synchronous vectors V₁ and V₂. [2]
γ=0.5. In S, action stay gives reward 1 and returns to S; move gives reward 0 and goes to U. In U, cash gives reward 4 and terminates; reset gives reward 0 and goes to S. Policy π selects each available action with probability 0.5.
(a) Write the Bellman expectation equations under π. [2]
(b) Solve for vπ(S) and vπ(U). [2]
(c) Compute qπ for all four state-action pairs, and give a greedy improved policy π′. [2]
(d) Evaluate π′. Is its action at S uniquely greedy with respect to vπ′? Explain. [2]
The grid has two rows and three columns. (1,1) and (2,3) are terminal. Four actions (up, down, left, right) are equiprobable. A move outside the grid leaves the state unchanged. Every move FROM a nonterminal state gives reward -1, including a move entering a terminal. γ=1; terminal values are fixed at zero.
(a) Write the synchronous random-policy update equation. [1]
(b) Starting from V₀=0, give the whole V₁ grid. [2]
(c) Give the whole V₂ grid. [2]
(d) Calculate V₃(1,2) and V₃(2,1). [2]
(e) Using V₂ for successor values, compute all four one-step action values at (1,2) and identify every greedy action. [2]
An episode is A --2→ B --(-1)→ A --4→ T. γ=0.5. For constant-step-size algorithms initialize V(A)=0, V(B)=2 and use α=0.5. Run each named algorithm independently from these initial values.
(a) Compute G₀,G₁,G₂. [2]
(b) Apply first-visit MC with the stated constant α once per visited state. [2]
(c) Using ONLY this episode and fresh zero counts, compute every-visit sample-mean MC estimates. [2]
(d) Apply online TD(0) in chronological order; show the value after each update. [3]
(e) Explain why the TD update at the second transition uses a value that has already changed. [1]
A nonterminal transition has current Q(S,A)=1, reward r=-1, γ=0.8 and learning rate α=0.25. Online next-state values for (a₁,a₂,a₃) are (5,2,0). The actual next action is a₂. The behaviour is ε-greedy with ε=0.3. Target-network next-state values are (1,4,3). Compute each algorithm independently.
(a) Give the behaviour probabilities of a₁,a₂,a₃. [2]
(b) Give the SARSA target and updated Q(S,A). [2]
(c) Give the Q-learning target and updated Q(S,A). [2]
(d) Give the standard DQN and Double DQN targets. Show which next action each selects. [3]
(e) If the transition instead terminates, give its target and tabular update. [1]
下面跟着题目逐步算
通用更新先写在纸上
只有目标 y 随算法改变。不要把 ε 当学习率,也不要先做 SARSA 再把结果送进 Q-learning。
γ=0.5. In X, safe gives reward 2 and goes to T; travel gives reward 0 and goes to Y. In Y, collect gives reward 4 and goes to T; cycle gives reward 1 and goes to X. Start policy iteration with π₀=(travel,cycle).
(a) Evaluate π₀ exactly. [2]
(b) Compute all four qπ₀ values and make a greedy improvement π₁. [3]
(c) Evaluate π₁, verify Bellman optimality, and list ALL deterministic optimal policies. [3]
(d) From V₀=0, give the synchronous value-iteration vectors V₁ and V₂. [2]