Jorge Nocedal.
Projected Hessian updating algorithms for nonlinearly constrained optimization online
. (page 3 of 4)
Online Library → Jorge Nocedal → Projected Hessian updating algorithms for nonlinearly constrained optimization → online text (page 3 of 4)
Font size
-44-
Theorem 4.3: Suppose "pdate Rule 1 Is user! in Aisorithm 4.1.
Choose a value v "> in condition (4.3). Then
\/re(0,n,3e>0, 5>0,ri>0, such that if He II _< e ,
II B -H^ll ,^ < 5, and n is used to define condition (4.3), then
n e^+^W < nie^.^ll , k ? 1 , (4.21)
i.e. , xj^ > x^ at least at a two-step 0-linear rate.
Proof: We prove the result for the PSB and OFP cases, leaving the
BFGS case as an exercise. Let a,a,,a2 be the constants from Lemma ^.3,
_ o
and define y^, M as in Corollary 4.2. Let a^ = II Mil ^a^. Let 3 be such
that
XII < B 11X11^
for any n X n mtrix X. Let ic = 2liH;'^ll , 5 = 1/ (211 H^^^!!) , so that, bv the
Banach lemma,
X-H,^ll < 5 => II X ^11 < K . (4.22)
Let E,,C, be as in Theorem 4.1, using this definition of 0, 5 > 0, n > to be
chosen small enough that
-45-
e < e ^/C^
e < £9/0^^
6 < 5/(23)
C^(e + 2B6 ) < r
Mil - {209(1 + n)C, e + niiG^ii } < 1
'1
•• ■4C9a + n)C,e ,
(2a, 6 +09) ( : ^ + n(i +-] IIG^Il) < 5 .
(4.23)
Such a choice of e ,6 ,n is clearly possible. Now let 11 e^!l < e,
II B -H^ll X, < 5. We will establish by induction that
(i) IIB-H*II^ < 26 , .1 > .
(ii) II eyi < C^lle^.^ll , 1^1.
(iii) lle^^ll < rllej_-Jl , j > 1 • . (4.24)
(i V ) If_ the update formula is used at iteration j ,
then IIB^^^-H^IIm < [(1 - aO ^)^^- + a ^y jl" B .-H^ll ^ + a9Y ^
else B :j+i = B . , i > .
Note that (iii) is the desired result.
For j = 0, (i) is true by assumption and (ii), (iii) are vacuous.
For (iv), we need only verify the inequality assuming the update
formila is used to obtain B^. Because II e^H < E < t -^ and H B^ II < < ,
Theorem 4.1 applies (with k = 1), giving
-46-
e^ll
Note that (1 -ct6^ )'â– '-< 1 - a ^. Now from (4.24) (iv), we have
\.+r"*"M* [(1 -ote2_)i/2 +ci^Y,,.]llB^_-H^il +ajy^^_ ,
where, proceeding as before to apply Corollary 4.2, and usin as i + " .
S, II
'^i
Thus, using either (4. la) or (4.1b) for the definition of Sj^, and with
U|^ defined by (4.12), we obtain
(jj^ •»■as i + » .
Now from Theorem 4.1(iii), for i large enough,
â– 52-
X 1 1 1 u
and hence bv ( 4. 2 4) (i 1) , (ill') ,
Thus
^k = li r"^ n as i > » ,
i "^k.-l" ,
I.e. , {Ci wJ NJ —
> >
tsj —
c
wi wi
ui ui \yi Ln
Ui Ui Ui Ui
O* OS CT 0^
ON
i> X>
^ ^
-s
- -
C^ ^ O^ o^
ON o^ o^ a^
^ ^ ^ ^
â– ^
NJ 00
3 —
U) LJ UJ UJ
H- O^ NJ NJ
J> £^ X> i>
VD — — UJ
Ul U1 Wl UI
N) U) W —
UI +
l'
o
— rsj
U b
P! PI
1 1
3
ISJ
3
3
C
N3 00
1 1
00 O vTi —
c=: p: m r=3
1 t 1 1
O — CT^ C
p] p: m p:
1 1 1 1
00 O -o £-
PI p] n m
1 1 1 1
CC
.- — O r^J
NJ < fSJ
o o o o
^
i— ho
Ui LP
^ ^J -^ -^
-J -J ^ ^
i^ ui i-n £>
ON
X> £-
^ ^
^
CO 00 00 03
Ui k^ ui m
00 03 OC 00
Ui ui i»n ui
On On O^ Ui
www
J> *^ i> UJ
J> UJ U) -^
(SJ 00
O O
L.J
II
rsj
0=
r-
p:
2
rO N3 PO LJ
n p: m n
yi ri :^ T=i
O NO NC Ul
P3 PI m p:
o
ON J>
P] PI
ISJ
VJ5 «
— ,- — —
— — — —
— — — ^
J> ^ £-. ^-'
O
X> f>
^
w w w w
WW w^
Ui J>
Ui Ui ^ IJ1
i-n ui UI Ln
ON CT* ON ON
Ln vO
I> £>
'^
. -^
."'^ â– ''^.
— vo
c> c^ cr o^
0^ O^ O O^
•^ -^ -J -J
OS O â–
A-n "— „
::.
-'
J> oJ
(^ L-i Lo ro
U> UJ LJ K)
i-n LD Ui Ln
£> 00
o o
3
II
U«
3
II
C
i
-a
wJ
— 'J
!> J> ^ Ln
Ui m Un 03
00 00 ^ —
^ 00
[SJ *»J
O -J
1 1
- vO
\0 ^O vO ^
n rr: pj n
1 1 1 1
— — ^ o
p; p: p: P!
1 1 1 1
-J X> O UJ
— PI p: PI
1 1 1 1
NJ N> SJ O
PI p]
1 1
n£> no
£- SO
PI P5
: 1
N3 J>
—
z
^ ^
CO 00 00 OS
03 X 00 -^
ON OS ON OS
CO - J
U> Lf>
^O ^£1
O '^ ^ sO
uo o ^ o
^ '^ -^ -O
NO 00
\0 i>
^
-3
2:
00 00
'^ *J ^J -vl
â– ^ - J -^ 00
Ui Ui ui Ln
kJ
tsJ L-J
w
\0 SJ 00 ^
P3 pj M rn
i' !T> O ISJ
PI PI P3 PI
(jJ W ^- tsj
PI P] P5 P3
ro o
m PI
vC —
VO H-
\0 ^ vO ^o
i£3 ^ sC —
•^D ■■^ \D ■-0
so NO
55
w ^
Sj NJ NJ —
ro ro NJ H-
^- — tsj NJ
ISJ
Ui Ui
uj UJ uj n:
uj uj uj ro
N3 |NO W UJ
— +
V
sC
UJ ISJ
— OS
c o
ON VO
PI PI
1 1
D
1
3
II
(SJ
T3
C
2
i
c
o
*^ vO "^ vO
— — — o
— l^ »— Ul
o o <
vO 00 NJ Ln
ro ON
PI W
1 1
O —
O Ln UJ Ui
PI m w rr
1 1 t 1
o o o o
Ln cjv i* J>
p: PI PI PI
t 1 t 1
^ ^ ^ ^
1^ -^ *» Ln
P! PI m p:
1 1 1 1
^j2 \C ^ "—
\o o
^ V.O ^C O
O v£> ^ ^C
ro NJ ro u)
O Ui
Ul Ul
-3
p:
— Isj
o o o o
C C O O
1^ UJ UJ x>
— 0^
«; uj
II
O
00 \o
00 09 00 00
00 00 03 03
— " ^ I^
vO i^
O O
^
^
5i
fJN i-
— _ ^ ^
— — O —
— ON NJ Ui
— —
OS £^
Ln CT^ Ui UJ
P5 PI PI PI
•^ UJ ON -C*
PI PJ P5 PI
O \0 C3N x>
m p: p: p:
O NO
m p:
Ul UJ
p: p:
^O vO
>X) \0 — vC
w w o ^
vO vO ►— v£
w W O w
00 *0 \Ci \£>
NO NO
CC
^ -o
00
3
II
-3
PI
EH
o
o o ^O o
o — o —
1^ Ui 1J1 i_n
— —
— Z 5 ^
— ISJ — ro
ON On O* ON
N> NJ
NJ —
o o
\C 00 00 \0
\0 i'
O O
o o
ON £*
O^ ^ CO 00
^- Ln -^J **
NO LP -^ n£)
— U1
vO 00
00 ^
vO -C* vO LO
PI PI PI p»
■— UJ \0 —
O ^ w O
V£| w- \0 ,—
w O w o
*£ ^O nO sO
NO NO
w^
-63-
Let us now comment on the resuJ.ts. ''^irst of aJJ., the results for
DNl and ^N2 do not show anv sl?inif leant arivanta>^e Co either fom of the
nuitlplier estlTTate. This is an option onlv for Newton's method or a
method such as Al ,A? which approximates a full Hessian matrix. Methods
'^l through r)2 do not have the aptlon of using (2.7e^ since thev do not
approximate ''^. â– .-
The results for Method. Al on Problems "4 and HSlll could he
improved hv reducing p, therebv improving the conditioning of the
matrices i\K '''e -chose — p ^ 10 because It is difficult to know an
oDtimai value of p _a priori, z '5ven then It was necessarv to Increase p
for Problem HS104, .where we used p = 100. The larger p is made, the
worse the results bfecoine", "because' of increased iil-conditionlng. Thus
the sensitivity of -Method Al bo -the choice of p is a definite drawback
of the method, Powell' s. Method A2. seems, from these results, to be
preferable. However we note^ithat no proof of local convergence has yet
been given for Method A2. ' '
The partial Rroyden methods R1-B4 are quite successful, but they
do seem inferior to Method A2 on the larger problems. The fact that
thev are one-step 0-superllnearly convergent does not seem to be a
significant advantage, compared with the disadvantage that thev do not
use the ^'PG9, update. '''e should mention, however, that Method A2
theoretically has a one-step o-superllnear rate of convergence en
Problems HSl^n and HSlll, since ^J^ is positive definite for these
problems (and only these). The partial '^royden methods have the
advantage that they maintain an approximation matrix with smaller
dimension than that used by 'fethod A2.
The two-sided proiected Hessian methods C1-C8 compare favorably
with Method A2. Thev have the advantage that the di-nenslon of ( "^i, } is
further reduced. Methods ni-r)2 clearlv suffer from the extra i»radient
evaluations required. ^gthods Cl-C^ ail have ahout the same
performance as each other. The same is true of Methods '^l-'^'i.
The parameter values n = 1 and v = O.ni for Methods Ci-CS were
successful for these test prohlems.. There vrouJ.d prohahiy never he a
need for a different choice of v, since the oniv purpose of this
Theorem 4.3: Suppose "pdate Rule 1 Is user! in Aisorithm 4.1.
Choose a value v "> in condition (4.3). Then
\/re(0,n,3e>0, 5>0,ri>0, such that if He II _< e ,
II B -H^ll ,^ < 5, and n is used to define condition (4.3), then
n e^+^W < nie^.^ll , k ? 1 , (4.21)
i.e. , xj^ > x^ at least at a two-step 0-linear rate.
Proof: We prove the result for the PSB and OFP cases, leaving the
BFGS case as an exercise. Let a,a,,a2 be the constants from Lemma ^.3,
_ o
and define y^, M as in Corollary 4.2. Let a^ = II Mil ^a^. Let 3 be such
that
XII < B 11X11^
for any n X n mtrix X. Let ic = 2liH;'^ll , 5 = 1/ (211 H^^^!!) , so that, bv the
Banach lemma,
X-H,^ll < 5 => II X ^11 < K . (4.22)
Let E,,C, be as in Theorem 4.1, using this definition of 0, 5 > 0, n > to be
chosen small enough that
-45-
e < e ^/C^
e < £9/0^^
6 < 5/(23)
C^(e + 2B6 ) < r
Mil - {209(1 + n)C, e + niiG^ii } < 1
'1
•• ■4C9a + n)C,e ,
(2a, 6 +09) ( : ^ + n(i +-] IIG^Il) < 5 .
(4.23)
Such a choice of e ,6 ,n is clearly possible. Now let 11 e^!l < e,
II B -H^ll X, < 5. We will establish by induction that
(i) IIB-H*II^ < 26 , .1 > .
(ii) II eyi < C^lle^.^ll , 1^1.
(iii) lle^^ll < rllej_-Jl , j > 1 • . (4.24)
(i V ) If_ the update formula is used at iteration j ,
then IIB^^^-H^IIm < [(1 - aO ^)^^- + a ^y jl" B .-H^ll ^ + a9Y ^
else B :j+i = B . , i > .
Note that (iii) is the desired result.
For j = 0, (i) is true by assumption and (ii), (iii) are vacuous.
For (iv), we need only verify the inequality assuming the update
formila is used to obtain B^. Because II e^H < E < t -^ and H B^ II < < ,
Theorem 4.1 applies (with k = 1), giving
-46-
e^ll
Note that (1 -ct6^ )'â– '-< 1 - a ^. Now from (4.24) (iv), we have
\.+r"*"M* [(1 -ote2_)i/2 +ci^Y,,.]llB^_-H^il +ajy^^_ ,
where, proceeding as before to apply Corollary 4.2, and usin as i + " .
S, II
'^i
Thus, using either (4. la) or (4.1b) for the definition of Sj^, and with
U|^ defined by (4.12), we obtain
(jj^ •»■as i + » .
Now from Theorem 4.1(iii), for i large enough,
â– 52-
X 1 1 1 u
and hence bv ( 4. 2 4) (i 1) , (ill') ,
Thus
^k = li r"^ n as i > » ,
i "^k.-l" ,
I.e. , {Ci wJ NJ —
> >
tsj —
c
wi wi
ui ui \yi Ln
Ui Ui Ui Ui
O* OS CT 0^
ON
i> X>
^ ^
-s
- -
C^ ^ O^ o^
ON o^ o^ a^
^ ^ ^ ^
â– ^
NJ 00
3 —
U) LJ UJ UJ
H- O^ NJ NJ
J> £^ X> i>
VD — — UJ
Ul U1 Wl UI
N) U) W —
UI +
l'
o
— rsj
U b
P! PI
1 1
3
ISJ
3
3
C
N3 00
1 1
00 O vTi —
c=: p: m r=3
1 t 1 1
O — CT^ C
p] p: m p:
1 1 1 1
00 O -o £-
PI p] n m
1 1 1 1
CC
.- — O r^J
NJ < fSJ
o o o o
^
i— ho
Ui LP
^ ^J -^ -^
-J -J ^ ^
i^ ui i-n £>
ON
X> £-
^ ^
^
CO 00 00 03
Ui k^ ui m
00 03 OC 00
Ui ui i»n ui
On On O^ Ui
www
J> *^ i> UJ
J> UJ U) -^
(SJ 00
O O
L.J
II
rsj
0=
r-
p:
2
rO N3 PO LJ
n p: m n
yi ri :^ T=i
O NO NC Ul
P3 PI m p:
o
ON J>
P] PI
ISJ
VJ5 «
— ,- — —
— — — —
— — — ^
J> ^ £-. ^-'
O
X> f>
^
w w w w
WW w^
Ui J>
Ui Ui ^ IJ1
i-n ui UI Ln
ON CT* ON ON
Ln vO
I> £>
'^
. -^
."'^ â– ''^.
— vo
c> c^ cr o^
0^ O^ O O^
•^ -^ -J -J
OS O â–
A-n "— „
::.
-'
J> oJ
(^ L-i Lo ro
U> UJ LJ K)
i-n LD Ui Ln
£> 00
o o
3
II
U«
3
II
C
i
-a
wJ
— 'J
!> J> ^ Ln
Ui m Un 03
00 00 ^ —
^ 00
[SJ *»J
O -J
1 1
- vO
\0 ^O vO ^
n rr: pj n
1 1 1 1
— — ^ o
p; p: p: P!
1 1 1 1
-J X> O UJ
— PI p: PI
1 1 1 1
NJ N> SJ O
PI p]
1 1
n£> no
£- SO
PI P5
: 1
N3 J>
—
z
^ ^
CO 00 00 OS
03 X 00 -^
ON OS ON OS
CO - J
U> Lf>
^O ^£1
O '^ ^ sO
uo o ^ o
^ '^ -^ -O
NO 00
\0 i>
^
-3
2:
00 00
'^ *J ^J -vl
â– ^ - J -^ 00
Ui Ui ui Ln
kJ
tsJ L-J
w
\0 SJ 00 ^
P3 pj M rn
i' !T> O ISJ
PI PI P3 PI
(jJ W ^- tsj
PI P] P5 P3
ro o
m PI
vC —
VO H-
\0 ^ vO ^o
i£3 ^ sC —
•^D ■■^ \D ■-0
so NO
55
w ^
Sj NJ NJ —
ro ro NJ H-
^- — tsj NJ
ISJ
Ui Ui
uj UJ uj n:
uj uj uj ro
N3 |NO W UJ
— +
V
sC
UJ ISJ
— OS
c o
ON VO
PI PI
1 1
D
1
3
II
(SJ
T3
C
2
i
c
o
*^ vO "^ vO
— — — o
— l^ »— Ul
o o <
vO 00 NJ Ln
ro ON
PI W
1 1
O —
O Ln UJ Ui
PI m w rr
1 1 t 1
o o o o
Ln cjv i* J>
p: PI PI PI
t 1 t 1
^ ^ ^ ^
1^ -^ *» Ln
P! PI m p:
1 1 1 1
^j2 \C ^ "—
\o o
^ V.O ^C O
O v£> ^ ^C
ro NJ ro u)
O Ui
Ul Ul
-3
p:
— Isj
o o o o
C C O O
1^ UJ UJ x>
— 0^
«; uj
II
O
00 \o
00 09 00 00
00 00 03 03
— " ^ I^
vO i^
O O
^
^
5i
fJN i-
— _ ^ ^
— — O —
— ON NJ Ui
— —
OS £^
Ln CT^ Ui UJ
P5 PI PI PI
•^ UJ ON -C*
PI PJ P5 PI
O \0 C3N x>
m p: p: p:
O NO
m p:
Ul UJ
p: p:
^O vO
>X) \0 — vC
w w o ^
vO vO ►— v£
w W O w
00 *0 \Ci \£>
NO NO
CC
^ -o
00
3
II
-3
PI
EH
o
o o ^O o
o — o —
1^ Ui 1J1 i_n
— —
— Z 5 ^
— ISJ — ro
ON On O* ON
N> NJ
NJ —
o o
\C 00 00 \0
\0 i'
O O
o o
ON £*
O^ ^ CO 00
^- Ln -^J **
NO LP -^ n£)
— U1
vO 00
00 ^
vO -C* vO LO
PI PI PI p»
■— UJ \0 —
O ^ w O
V£| w- \0 ,—
w O w o
*£ ^O nO sO
NO NO
w^
-63-
Let us now comment on the resuJ.ts. ''^irst of aJJ., the results for
DNl and ^N2 do not show anv sl?inif leant arivanta>^e Co either fom of the
nuitlplier estlTTate. This is an option onlv for Newton's method or a
method such as Al ,A? which approximates a full Hessian matrix. Methods
'^l through r)2 do not have the aptlon of using (2.7e^ since thev do not
approximate ''^. â– .-
The results for Method. Al on Problems "4 and HSlll could he
improved hv reducing p, therebv improving the conditioning of the
matrices i\K '''e -chose — p ^ 10 because It is difficult to know an
oDtimai value of p _a priori, z '5ven then It was necessarv to Increase p
for Problem HS104, .where we used p = 100. The larger p is made, the
worse the results bfecoine", "because' of increased iil-conditionlng. Thus
the sensitivity of -Method Al bo -the choice of p is a definite drawback
of the method, Powell' s. Method A2. seems, from these results, to be
preferable. However we note^ithat no proof of local convergence has yet
been given for Method A2. ' '
The partial Rroyden methods R1-B4 are quite successful, but they
do seem inferior to Method A2 on the larger problems. The fact that
thev are one-step 0-superllnearly convergent does not seem to be a
significant advantage, compared with the disadvantage that thev do not
use the ^'PG9, update. '''e should mention, however, that Method A2
theoretically has a one-step o-superllnear rate of convergence en
Problems HSl^n and HSlll, since ^J^ is positive definite for these
problems (and only these). The partial '^royden methods have the
advantage that they maintain an approximation matrix with smaller
dimension than that used by 'fethod A2.
The two-sided proiected Hessian methods C1-C8 compare favorably
with Method A2. Thev have the advantage that the di-nenslon of ( "^i, } is
further reduced. Methods ni-r)2 clearlv suffer from the extra i»radient
evaluations required. ^gthods Cl-C^ ail have ahout the same
performance as each other. The same is true of Methods '^l-'^'i.
The parameter values n = 1 and v = O.ni for Methods Ci-CS were
successful for these test prohlems.. There vrouJ.d prohahiy never he a
need for a different choice of v, since the oniv purpose of this
Online Library → Jorge Nocedal → Projected Hessian updating algorithms for nonlinearly constrained optimization → online text (page 3 of 4)
Using the text of ebook Projected Hessian updating algorithms for nonlinearly constrained optimization by Jorge Nocedal active link like:
read the ebook Projected Hessian updating algorithms for nonlinearly constrained optimization
read the ebook Projected Hessian updating algorithms for nonlinearly constrained optimization
