D 卷 · 中文详细解析

Which combination is relevant to standard tabular SARSA convergence to optimal action values under suitable finite-MDP assumptions?

A. Fixed positive ε and any constant learning rate guarantee convergence.

B. GLIE exploration together with suitable diminishing step-size conditions.

C. Never visiting nongreedy actions.

D. Using a nonlinear neural network alone.

An action gives reward 1 and moves to X or Y with probabilities 0.25 and 0.75. V(X)=8, V(Y)=0 and γ=0.5. What is its one-step action value?

A. 5

B. 3

C. 2

D. 1

What is a main purpose of sampling minibatches from a replay buffer?

A. To make every stored transition terminal.

B. To avoid storing rewards.

C. To force the policy to be deterministic.

D. To reuse data and reduce the effect of sequential correlations.

At a fixed state, qπ(s,a)=6 and vπ(s)=4. What is Aπ(s,a)?

A. 10

B. 2

C. 4

D. 6

A continuing task gives reward 2 on every step. If γ=0.5, what is the return?

A. 2

B. 4

C. 1

D. Infinity

Which output is most clearly an environment-model prediction rather than a value prediction?

A. The expected discounted return from a state.

B. The probability of choosing an action.

C. The next-state distribution and expected immediate reward after an action.

D. The advantage of one action.

Which statement about intermediate value-iteration estimates is generally correct?

A. A greedy policy may already be optimal before the numerical values have fully converged.

B. One sweep always gives v*.

C. The same fixed policy must be used forever.

D. Increasing k means taking another real environment action.

V(S)=3, r=-1, γ=0.5 and V(S′)=6 for a nonterminal transition. What is the TD error?

A. 2

B. 0

C. -1

D. -4

In PER with N=4, P(i)=0.5 and β=1, what is the UNNORMALIZED importance weight (N P(i))^(−β)?

A. 0.5

B. 2

C. 4

D. 0.125

In the stated gridworld convention, attempting to move outside the grid keeps the state unchanged and costs -1. Which target is correct for this action?

A. 0

B. V(s)

C. -1

D. -1 + γV(s)

Distinguish iterative policy evaluation, policy iteration and value iteration. Explain what is fixed during evaluation, what changes during improvement and what k counts. [5 points]

Give the ε-greedy probabilities for m actions with g tied greedy actions, using uniform exploration over all m actions. Explain why greedy actions also receive exploration probability and why GLIE needs more than a decaying ε. [5 points]

Explain maximization bias using noisy Q estimates. Give the Double DQN target, identify the roles of the online and target networks, and state one limitation of the method. [5 points]

Three transitions have absolute TD errors (0,1,3). Set εₚ=1, αₚ=1 and β=1. Find priorities, sampling probabilities and unnormalized importance weights. Explain why transition 1 can still be sampled. [5 points]

For an undiscounted REINFORCE update, explain the role of ∇θ log πθ(Aₜ|Sₜ). If Gₜ=3 and b(Sₜ)=5, what is the weight on this gradient, and what is the usual local effect on the selected action probability? Explain why the baseline must not depend on the sampled action in the simple unbiased-baseline argument. [5 points]

There is one nonterminal state S and one terminal state T. From S, with probability 0.5 the process returns to S and receives reward 2; with probability 0.5 it enters T and receives reward 6. γ=0.5.

(a) Find the expected immediate reward at S and write the Bellman equation. [2]

(b) Solve for v(S). [2]

(c) For S --2→ S --6→ T, compute both visit returns. [2]

(d) From V₀(S)=0, compute V₁,V₂,V₃ synchronously. [2]

γ=0.5. At A, fast gives reward 1 and terminates; support gives reward 0 and moves to B. At B, finish gives reward 3 and terminates; repeat gives reward 1 and returns to B. π(fast|A)=0.25, π(support|A)=0.75; π chooses each B action with probability 0.5.

(a) Write Bellman expectation equations. [2]

(b) Solve for vπ(A),vπ(B). [2]

(c) Find all four qπ values and a greedy policy π′. [2]

(d) Evaluate π′ and verify whether another improvement is possible. [2]

Five locations lie on a line: Tₗ — L — M — R — Tᵣ. Both end locations are terminal. At each nonterminal state, left and right are chosen with probability 0.5 and move to the adjacent location. Every move costs -1, including entry into a terminal. γ=1.

(a) From V₀=0, give V₁ for (L,M,R). [1]

(b) Compute V₂ synchronously. [2]

(c) Compute V₃ synchronously. [2]

(d) Solve for the exact random-policy values. [2]

(e) Using those exact values, list all greedy actions in each state. [2]

Observed episodes are (1) A --0→ B --2→ T; (2) A --2→ T. γ=1. For constant-step updates initialize V(A)=1,V(B)=0 and α=0.5. Treat methods independently.

(a) List all visit returns. [2]

(b) Compute first-visit sample-mean MC estimates from these episodes with fresh counts. [2]

(c) Process both episodes with online TD(0), showing all three updates. [3]

(d) Starting from the original values, perform ONLY the two-step update for the first visit to A in episode 1. [2]

(e) Why does this two-step target equal the MC return for that visit? [1]

Q(S,A)=0, r=2, γ=1, α=0.5 and the transition is nonterminal. Online next-state values are (3,3,1); the actual next action is a₃. Behaviour uses ε=0.3, splitting exploitation equally among tied maxima. Target-network values are (4,1,5). For Double DQN only, break online argmax ties by choosing the lowest action index.

(a) Compute the behaviour probabilities. [2]

(b) Compute SARSA's target and update. [2]

(c) Compute Q-learning's target and update. [2]

(d) Compute DQN and Double DQN targets, applying the stated tie rule. [3]

(e) Give the target and tabular update for a terminal version of this transition. [1]

At S, exit gives reward 2 and terminates; work gives reward 1.5, then moves to S or T with probability 0.5 each. At U, stop gives reward 1.5 and terminates; join gives reward 0 and goes to S. For (a)–(c), γ=0.8 and π₀=(exit,stop).

(a) Evaluate π₀. [2]

(b) Compute all qπ₀ values and improve the policy. [3]

(c) Evaluate the improved policy and verify optimality. [3]

(d) Let γ vary in [0,1). At S, for which γ is work strictly better than exit when followed optimally? State the tie value. [2]