A1. Tabular convergence conditions [2 points]
B. GLIE and appropriate diminishing learning rates are part of standard tabular convergence assumptions.
Detailed Chinese explanation →Answer Key & Model Responses
Concise English answers for checking and exam phrasing. See the Chinese worked solutions for all intermediate steps and common mistakes.
10 questions × 2 points
I. Single-choice questions
20 pointsA2. Expectation over next states [2 points]
C. The action value is 2.
Detailed Chinese explanation →A3. Experience replay [2 points]
D. Replay enables data reuse and reduces sequential correlation in training batches.
Detailed Chinese explanation →A4. An advantage signal [2 points]
B. The advantage is 2.
Detailed Chinese explanation →A5. An infinite discounted stream [2 points]
B. The discounted return is 4.
Detailed Chinese explanation →A6. A model versus a value function [2 points]
C. A model predicts state transitions and rewards.
Detailed Chinese explanation →A7. Value iteration [2 points]
A. Correct action rankings can arise before exact value convergence.
Detailed Chinese explanation →A8. TD error [2 points]
C. δ=-1.
Detailed Chinese explanation →A9. Importance weights [2 points]
A. The unnormalized weight is 0.5.
Detailed Chinese explanation →A10. A wall in gridworld [2 points]
D. A wall transition has target -1+γV(s).
Detailed Chinese explanation →5 questions × 5 points
II. Short-answer questions
25 pointsB1. Three DP procedures [5 points]
Policy evaluation estimates vπ for a fixed policy. Policy iteration alternates evaluation and greedy improvement. Value iteration repeatedly applies optimality backups. k counts computational sweeps.
Detailed Chinese explanation →B2. Exploration and tie handling [5 points]
Each greedy action gets (1−ε)/g+ε/m; each nongreedy action gets ε/m. GLIE also requires infinite state-action exploration, which ε decay alone does not guarantee.
Detailed Chinese explanation →B3. Double DQN does not average networks [5 points]
Double DQN separates online action selection from target-network evaluation to reduce maximization bias. It does not average the networks and does not guarantee unbiased values or convergence.
Detailed Chinese explanation →B4. Numerical PER probabilities [5 points]
p=(1,2,4), P=(1/7,2/7,4/7), w=(7/3,7/6,7/12). The positive offset keeps the zero-error transition sampleable.
Detailed Chinese explanation →B5. Baseline and actor update direction [5 points]
The score-function gradient locally increases the selected action’s log probability. Its weight is −2, so the update moves in the opposite direction. An action-independent baseline has zero expected score-function contribution.
Detailed Chinese explanation →3 questions: 8 + 8 + 9 points
III. Calculation — MDP
25 pointsC1. Rewards correlated with transitions [8 points]
Expected reward=4; v=4+0.25v=16/3. Sample returns=(5,6). V₁=4,V₂=5,V₃=5.25.
Detailed Chinese explanation →C2. Nonuniform policy probabilities [8 points]
vπ=(5/4,8/3); qπ=(1,4/3;3,7/3). π′=(support,finish), vπ′=(1.5,3), and it is optimal.
Detailed Chinese explanation →C3. A random walk with two terminals [9 points]
V₁=(-1,-1,-1), V₂=(-1.5,-2,-1.5), V₃=(-2,-2.5,-2). vπ=(-3,-4,-3). Greedy actions: L→left, M→either, R→right.
Detailed Chinese explanation →2 questions × 10 points
IV. Calculation — Learning algorithms
20 pointsD1. Targets of different horizons [10 points]
Returns: A:[2,2], B:[2]. MC=(2,2). Online TD: A→0.5,B→1,A→1.25. The separate two-step target is 2 and updates A to 1.5; it spans the remainder of the episode.
Detailed Chinese explanation →D2. Tie-breaking changes Double DQN [10 points]
Probabilities=(0.45,0.45,0.1). SARSA: y=3,Qnew=1.5. Q-learning: y=5,Qnew=2.5. DQN=7 (a₃); Double=6 (a₁). Terminal: y=2,Qnew=1.
Detailed Chinese explanation →1 question, 4 subquestions
V. Calculation — Optimal policy
10 pointsE1. Discounting changes the optimal choice [10 points]
vπ₀=(2,1.5); qπ₀=(2,2.3;1.5,1.6). π₁=(work,join), v*=(2.5,2). At S, work is strictly better for γ>0.5; a tie occurs at γ=0.5.
Detailed Chinese explanation →