A1. Double DQN action selection [2 points]
A. Select action 2 online and evaluate it as 2 with the target network.
Detailed Chinese explanation →Answer Key & Model Responses
Concise English answers for checking and exam phrasing. See the Chinese worked solutions for all intermediate steps and common mistakes.
10 questions × 2 points
I. Single-choice questions
20 pointsA2. An optimality equation [2 points]
C. Average stochastic outcomes within each action, then maximize over actions.
Detailed Chinese explanation →A3. What GLIE does not imply [2 points]
B. GLIE provides no finite-time guarantee of knowing an exact optimal policy.
Detailed Chinese explanation →A4. Discount factor zero [2 points]
A. Gₜ=Rₜ₊₁.
Detailed Chinese explanation →A5. DQN output [2 points]
D. The outputs are estimated action values, not necessarily action probabilities.
Detailed Chinese explanation →A6. ε-greedy probabilities [2 points]
A. Total nongreedy probability is 4×0.1/5=0.08.
Detailed Chinese explanation →A7. The critic [2 points]
B. The critic estimates values or related signals to guide the actor.
Detailed Chinese explanation →A8. First-visit MC [2 points]
D. First-visit MC uses the return 4 for this episode.
Detailed Chinese explanation →A9. Which state needs a velocity? [2 points]
B. Including velocity can make the state representation sufficient for prediction.
Detailed Chinese explanation →A10. Linear value approximation [2 points]
C. v̂=1.
Detailed Chinese explanation →5 questions × 5 points
II. Short-answer questions
25 pointsB1. Reading the notation [5 points]
Reward is immediate; return accumulates future rewards. V averages over the policy’s first action; Q fixes it. max returns a value, argmax an optimizer. k indexes sweeps, t indexes transitions. α controls update size; ε controls exploration.
Detailed Chinese explanation →B2. MC and TD with repeated visits [5 points]
Visit conventions determine which returns are used; step-size conventions determine how those returns are averaged. Online TD reads the current table after earlier updates, so transition order matters.
Detailed Chinese explanation →B3. Choose the correct DQN improvement [5 points]
Replay: reuse and decorrelation. Target network: slower targets. Double: separate selection and evaluation. PER: nonuniform experience sampling with correction. Dueling: value/advantage decomposition.
Detailed Chinese explanation →B4. A small REINFORCE calculation [5 points]
π(a₁)=0.5, score gradient=0.5, θnew=0.1. The probability of a₁ increases to about 0.525. The action-independent baseline centers the return without changing the expected gradient.
Detailed Chinese explanation →B5. Learning, planning and model errors [5 points]
Model-free updates use real experience directly; planning uses a model. Dyna learns from real transitions, updates its model, and performs additional model-generated updates. Model bias can be amplified by excessive planning.
Detailed Chinese explanation →3 questions: 8 + 8 + 9 points
III. Calculation — MDP
25 pointsC1. Undiscounted absorbing MRP [8 points]
v=(-4,-2), sample returns=(-3,-2,-1). Expected remaining steps are 4 and 2. Finite expected termination time makes the undiscounted values finite.
Detailed Chinese explanation →C2. A deterministic initial policy [8 points]
vπ=(8/3,4/3). qπ=(8/3,1;4/3,3). Improved policy=(a,d), with optimal values (3.5,3).
Detailed Chinese explanation →C3. Expectation backups versus optimality backups [9 points]
V₁=(0.5,0.5), V₂=(0.75,0.625); W₁=(1,2),W₂=(2,2). Action values from V₂: (0.375,1.3125;2,-0.625), so choose (b,c).
Detailed Chinese explanation →2 questions × 10 points
IV. Calculation — Learning algorithms
20 pointsD1. Every-visit constant-step MC [10 points]
G=(2.75,3.5,3). First-visit MC=(2.375,2). Every-visit constant-step MC=(2.9375,2). TD updates: A→2,A→2.25,B→2. Constant-step averaging retains unequal weights and an initial-value contribution.
Detailed Chinese explanation →D2. Separate behaviour, learning and target networks [10 points]
Probabilities=(0.1,0.9). SARSA: y=1.8,Qnew=2.88. Q-learning: y=5.4,Qnew=3.24. DQN=6.3; Double=3.6. Terminal: y=0,Qnew=2.7.
Detailed Chinese explanation →1 question, 4 subquestions
V. Calculation — Optimal policy
10 pointsE1. Three actions at one state [10 points]
vπ₀=(2,1), qπ₀=(4,0.5,2;6,1). π₁=(short,collect), v*=(4,6). V₁=V₂=(4,6).
Detailed Chinese explanation →