A1. max and argmax [2 points]
A. The maximum value is -2, attained by right.
Detailed Chinese explanation →Answer Key & Model Responses
Concise English answers for checking and exam phrasing. See the Chinese worked solutions for all intermediate steps and common mistakes.
10 questions × 2 points
I. Single-choice questions
20 pointsA2. The target network [2 points]
A. A delayed target network stabilizes learning by slowing changes in the bootstrap targets.
Detailed Chinese explanation →A3. An MDP model [2 points]
C. A transition model and a reward model support expected Bellman backups.
Detailed Chinese explanation →A4. Planning in Dyna [2 points]
D. Dyna performs extra value updates using model-generated transitions.
Detailed Chinese explanation →A5. Fixed exploration [2 points]
C. A fixed positive exploration rate prevents the policy from becoming greedy in the limit.
Detailed Chinese explanation →A6. Immediate reward and value [2 points]
D. The return is 7; only rewards after the first transition would be discounted.
Detailed Chinese explanation →A7. A uniform replay special case [2 points]
B. With αₚ=0, P(i)=1/N.
Detailed Chinese explanation →A8. Policy evaluation versus value iteration [2 points]
B. Value iteration maximizes over actions while still averaging over stochastic outcomes.
Detailed Chinese explanation →A9. Stochastic policies [2 points]
C. Randomization can prevent exploitation of a predictable deterministic choice.
Detailed Chinese explanation →A10. On-policy control [2 points]
D. SARSA uses the value of the actually selected next action.
Detailed Chinese explanation →5 questions × 5 points
II. Short-answer questions
25 pointsB1. DP, MC and TD comparison [5 points]
DP uses model-based expected bootstrapped backups. MC uses sampled complete returns without bootstrapping. TD uses sampled bootstrapped targets and can update before termination. MC commonly has higher variance; TD can have bias from its value estimates.
Detailed Chinese explanation →B2. Policy improvement theorem [5 points]
A greedy action attains at least the policy-weighted average qπ value. Repeated Bellman comparison yields vπ′≥vπ. Stability after exact evaluation implies optimality; an unchanged approximate sweep alone does not.
Detailed Chinese explanation →B3. Why Q-learning is off-policy [5 points]
Q-learning learns a greedy target policy from data generated by a potentially exploratory behaviour policy. SARSA evaluates the actual behaviour, including exploration risk, while Q-learning bootstraps from a greedy next action.
Detailed Chinese explanation →B4. Dueling is not Double [5 points]
Dueling separates a shared state-value stream from action advantages and combines them with mean subtraction. Double DQN instead separates action selection from evaluation in the target; the two ideas are compatible.
Detailed Chinese explanation →B5. Function approximation risks [5 points]
Function approximation compresses a large value table and generalizes through shared features. A linear approximator uses x(s)ᵀw; a semi-gradient update holds the target fixed while differentiating the current prediction. Shared parameters can cause approximation error or interference.
Detailed Chinese explanation →3 questions: 8 + 8 + 9 points
III. Calculation — MDP
25 pointsC1. A branching episodic MRP [8 points]
P rows: (0,.5,.5,0),(0,0,1,0),(0,0,0,1),(0,0,0,1). r=(1,0,4,0). G=(2,2,4). v=(2.5,2,4). V₁=(1,0,4), V₂=(2,2,4), V₃=(2.5,2,4).
Detailed Chinese explanation →C2. Risky action and policy averaging [8 points]
vπ=(108/31,-4/31), qπ=(2,154/31;0,-8/31). First improvement: (bold,quit), v′=(5,0). It is not optimal: recover then has value 0.5>0.
Detailed Chinese explanation →C3. Value iteration with stochastic outcomes [9 points]
V₁=(1,2), V₂=(1.75,2), V₃=(1.9375,2), actions (a,c). Exact optimal values are (2,2).
Detailed Chinese explanation →2 questions × 10 points
IV. Calculation — Learning algorithms
20 pointsD1. MC, online TD and one frozen batch sweep [10 points]
Returns: A:[3,2], B:[2]. MC=(2.5,2). Online TD: A→0.5, B→1, A→1.25. One frozen batch sweep gives (0.75,1).
Detailed Chinese explanation →D2. Negative action values [10 points]
Probabilities=(1/15,1/15,13/15). SARSA: y=-3,Qnew=-3. Q-learning: y=-2.5,Qnew=-2.9. DQN=-2.5 (a₂); Double=-3 (a₃). Terminal: y=-2,Qnew=-2.8.
Detailed Chinese explanation →1 question, 4 subquestions
V. Calculation — Optimal policy
10 pointsE1. An optimal continuing action [10 points]
vπ₀=(3,2), qπ₀=(3,3.4;2,2.4). π₁=(wait,join), v*=(5,4). V₁=(3,2), V₂=(3.4,2.4).
Detailed Chinese explanation →