3.II.28I

Optimization and Control
Part II, 2006

A discrete-time controlled Markov process evolves according to

Xt+1=λXt+ut+εt,t=0,1,,X_{t+1}=\lambda X_{t}+u_{t}+\varepsilon_{t}, \quad t=0,1, \ldots,

where the ε\varepsilon are independent zero-mean random variables with common variance σ2\sigma^{2}, and λ\lambda is a known constant.

Consider the problem of minimizing

Ft,T(x)=E[j=tT1βjtC(Xj,uj)+βTtR(XT)],F_{t, T}(x)=\mathbb{E}\left[\sum_{j=t}^{T-1} \beta^{j-t} C\left(X_{j}, u_{j}\right)+\beta^{T-t} R\left(X_{T}\right)\right],

where C(x,u)=12(u2+ax2),β(0,1)C(x, u)=\frac{1}{2}\left(u^{2}+a x^{2}\right), \beta \in(0,1) and R(x)=12a0x2+b0R(x)=\frac{1}{2} a_{0} x^{2}+b_{0}. Show that the optimal control at time jj takes the form uj=kTjXju_{j}=k_{T-j} X_{j} for certain constants kik_{i}. Show also that the minimized value for Ft,T(x)F_{t, T}(x) is of the form

12aTtx2+bTt\frac{1}{2} a_{T-t} x^{2}+b_{T-t}

for certain constants aj,bja_{j}, b_{j}. Explain how these constants are to be calculated. Prove that the equation

f(z)a+λ2βz1+βz=zf(z) \equiv a+\frac{\lambda^{2} \beta z}{1+\beta z}=z

has a unique positive solution z=az=a_{*}, and that the sequence (aj)j0\left(a_{j}\right)_{j} \geqslant 0 converges monotonically to aa_{*}.

Prove that the sequence (bj)j0\left(b_{j}\right)_{j \geqslant 0} converges, to the limit

bβσ2a2(1β).b_{*} \equiv \frac{\beta \sigma^{2} a_{*}}{2(1-\beta)} .

Finally, prove that kjkβaλ/(1+βa)k_{j} \rightarrow k_{*} \equiv-\beta a_{*} \lambda /\left(1+\beta a_{*}\right).