7.Large-Scale Unconstrained Optimization.pdfVIP

This is pag Printer: O CHAP T E R7 Large-Scale Unconstrained Optimization Many applications give rise to unconstrained optimization problems with thousands or millions of variables. Problems of this size can be solved efficiently only if the storage and computational costs of the optimization algorithm can be kept at a tolerable level. A diverse collection of large-scale optimization methods has been developed to achieve this goal, each being particularly effective for certain problem types. Some of these methods are straightforward adaptations of the methods described in Chapters 3, 4, and 6. Other approaches are modifications of these basic methods that allow approximate steps to be calculated at lower cost in computation and storage. One set of approaches that we have alreadydiscussed—thenonlinear conjugate gradientmethodsof Section5.2—canbe applied 7 . 1 . I N E X A C T N E W T O N M E T H O D S 165 to large problems without modification, because of its minimal storage demands and its reliance on only first-order derivative information. The line search and trust-regionNewton algorithmsofChapters 3 and4 requirematrix factorizations of the Hessian matrices ?2 fk . In the large-scale case, these factorizations can be carried out using sparse elimination techniques. Such algorithms have received much attention, and high quality software implementations are available. If the computational cost and memory requirements of these sparse factorization methods are affordable for a given application, and if the Hessian matrix can be formed explicitly, Newton methods based on sparse factorizations constitute an effective approach for solving such problems. Often, however, the cost of factoring the Hessian is prohibitive, and it is preferable to compute approximations to the Newton step using iterative linear algebra techniques. Section 7.1 discusses inexact Newton methods that use these techniques, in both line search and trust-region frameworks. The resulting algorithms hav