A1. The meaning of k [2 points]
A. k counts completed synchronous sweeps.
Detailed Chinese explanation →Answer Key & Model Responses
Concise English answers for checking and exam phrasing. See the Chinese worked solutions for all intermediate steps and common mistakes.
10 questions × 2 points
I. Single-choice questions
20 pointsA2. A sufficient state [2 points]
B. Include key possession in the state so that it contains the information needed to predict the next outcome.
Detailed Chinese explanation →A3. A DQN target [2 points]
A. y = 5.
Detailed Chinese explanation →A4. Bellman expectation [2 points]
D. vπ(s) = 5.
Detailed Chinese explanation →A5. Bootstrapping [2 points]
C. The one-step TD target bootstraps from V(Sₜ₊₁).
Detailed Chinese explanation →A6. An action-independent baseline [2 points]
B. E[∇ log π(A|s) b(s) | s] = 0.
Detailed Chinese explanation →A7. Discounted return [2 points]
C. G₀ = 2.
Detailed Chinese explanation →A8. Function approximation [2 points]
D. Shared parameters allow updates to generalize across states.
Detailed Chinese explanation →A9. Ties in ε-greedy [2 points]
B. Each greedy action has probability 0.45.
Detailed Chinese explanation →A10. Dueling aggregation [2 points]
C. Q = (5, 1).
Detailed Chinese explanation →5 questions × 5 points
II. Short-answer questions
25 pointsB1. Prediction versus control [5 points]
Prediction evaluates a fixed policy; control searches for an optimal policy. Policy iteration alternates evaluation and greedy improvement. max returns a value, whereas argmax returns an action attaining that value.
Detailed Chinese explanation →B2. GLIE is a property [5 points]
GLIE requires infinite exploration and a policy that becomes greedy in the limit. It is a property of a policy sequence. Fixed positive ε generally violates the second condition; εₜ → 0 alone does not ensure the first.
Detailed Chinese explanation →B3. DQN and Double DQN [5 points]
DQN selects and evaluates with the target network. Double DQN selects with the online network and evaluates with the target network, reducing the coupling that contributes to maximization bias.
Detailed Chinese explanation →B4. Prioritized replay [5 points]
PER samples experiences according to priorities such as absolute TD errors. Positive offsets retain sampling support; importance weights compensate for nonuniform sampling, with β controlling the correction strength.
Detailed Chinese explanation →B5. REINFORCE and Actor–critic [5 points]
REINFORCE weights the score-function gradient by Gₜ−b(Sₜ). The actor learns the policy; the critic estimates values. A one-step TD error can serve as an approximate advantage signal.
Detailed Chinese explanation →3 questions: 8 + 8 + 9 points
III. Calculation — MDP
25 pointsC1. A two-state Markov reward process [8 points]
P rows: (1/2,1/2,0), (1/2,0,1/2), (0,0,1). r=(2,-1,0). Returns: (1.5,-1). v=(28/11,-4/11). V₁=(2,-1), V₂=(2.25,-0.5).
Detailed Chinese explanation →C2. Policy evaluation and one improvement [8 points]
vπ=(16/11,26/11); qπ=(19/11,13/11;4,8/11). π′=(stay,cash), vπ′=(2,4). At S both actions are greedy with respect to vπ′.
Detailed Chinese explanation →C3. A 2 × 3 grid [9 points]
V₁=[0,-1,-1; -1,-1,0]. V₂=[0,-1.75,-1.75; -1.75,-1.75,0]. Both requested V₃ values equal -2.3125. Action values at (1,2): (-2.75,-2.75,-1,-2.75); greedy action: left.
Detailed Chinese explanation →2 questions × 10 points
IV. Calculation — Learning algorithms
20 pointsD1. Repeated states: MC and online TD [10 points]
Returns: (2.5,1,4). First-visit MC: (1.25,1.5). Every-visit sample means: (3.25,1). Online TD: A→1.5, B→0.875, A→2.75.
Detailed Chinese explanation →D2. Four targets from one transition [10 points]
Probabilities: (0.8,0.1,0.1). SARSA: y=0.6, Qnew=0.9. Q-learning: y=3, Qnew=1.5. DQN: 2.2 (a₂); Double DQN: -0.2 (a₁). Terminal: y=-1, Qnew=0.5.
Detailed Chinese explanation →1 question, 4 subquestions
V. Calculation — Optimal policy
10 pointsE1. An optimal policy with a tie [10 points]
vπ₀=(2/3,4/3). qπ₀=(2,2/3;4,4/3). π₁=(safe,collect), v*=(2,4). Both (safe,collect) and (travel,collect) are optimal. V₁=V₂=(2,4).
Detailed Chinese explanation →