The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Le dépliage profond est une technique d’apprentissage profond prometteuse, dont l’architecture réseau repose sur l’expansion de la structure récursive des algorithmes itératifs existants. Bien que le déploiement profond réalise une accélération de la convergence, ses aspects théoriques n’ont pas encore été révélés. Cette étude détaille l'analyse théorique de l'accélération de convergence dans la descente de gradient profondément dépliée (DUGD) dont les paramètres entraînables sont des tailles de pas. Nous proposons une interprétation plausible des paramètres de taille de pas appris dans DUGD en introduisant le principe des étapes de Chebyshev dérivées des polynômes de Chebyshev. L'utilisation des étapes de Chebyshev dans la descente de gradient (GD) nous permet de délimiter le rayon spectral d'une matrice régissant la vitesse de convergence de GD, conduisant à une limite supérieure stricte du taux de convergence. Les résultats numériques montrent que les étapes de Chebyshev expliquent bien numériquement les paramètres de taille de pas appris dans DUGD.
Satoshi TAKABE
Tokyo Institute of Technology
Tadashi WADAYAMA
Nagoya Institute of Technology
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Copier
Satoshi TAKABE, Tadashi WADAYAMA, "Convergence Acceleration via Chebyshev Step: Plausible Interpretation of Deep-Unfolded Gradient Descent" in IEICE TRANSACTIONS on Fundamentals,
vol. E105-A, no. 8, pp. 1110-1120, August 2022, doi: 10.1587/transfun.2021EAP1139.
Abstract: Deep unfolding is a promising deep-learning technique, whose network architecture is based on expanding the recursive structure of existing iterative algorithms. Although deep unfolding realizes convergence acceleration, its theoretical aspects have not been revealed yet. This study details the theoretical analysis of the convergence acceleration in deep-unfolded gradient descent (DUGD) whose trainable parameters are step sizes. We propose a plausible interpretation of the learned step-size parameters in DUGD by introducing the principle of Chebyshev steps derived from Chebyshev polynomials. The use of Chebyshev steps in gradient descent (GD) enables us to bound the spectral radius of a matrix governing the convergence speed of GD, leading to a tight upper bound on the convergence rate. Numerical results show that Chebyshev steps numerically explain the learned step-size parameters in DUGD well.
URL: https://global.ieice.org/en_transactions/fundamentals/10.1587/transfun.2021EAP1139/_p
Copier
@ARTICLE{e105-a_8_1110,
author={Satoshi TAKABE, Tadashi WADAYAMA, },
journal={IEICE TRANSACTIONS on Fundamentals},
title={Convergence Acceleration via Chebyshev Step: Plausible Interpretation of Deep-Unfolded Gradient Descent},
year={2022},
volume={E105-A},
number={8},
pages={1110-1120},
abstract={Deep unfolding is a promising deep-learning technique, whose network architecture is based on expanding the recursive structure of existing iterative algorithms. Although deep unfolding realizes convergence acceleration, its theoretical aspects have not been revealed yet. This study details the theoretical analysis of the convergence acceleration in deep-unfolded gradient descent (DUGD) whose trainable parameters are step sizes. We propose a plausible interpretation of the learned step-size parameters in DUGD by introducing the principle of Chebyshev steps derived from Chebyshev polynomials. The use of Chebyshev steps in gradient descent (GD) enables us to bound the spectral radius of a matrix governing the convergence speed of GD, leading to a tight upper bound on the convergence rate. Numerical results show that Chebyshev steps numerically explain the learned step-size parameters in DUGD well.},
keywords={},
doi={10.1587/transfun.2021EAP1139},
ISSN={1745-1337},
month={August},}
Copier
TY - JOUR
TI - Convergence Acceleration via Chebyshev Step: Plausible Interpretation of Deep-Unfolded Gradient Descent
T2 - IEICE TRANSACTIONS on Fundamentals
SP - 1110
EP - 1120
AU - Satoshi TAKABE
AU - Tadashi WADAYAMA
PY - 2022
DO - 10.1587/transfun.2021EAP1139
JO - IEICE TRANSACTIONS on Fundamentals
SN - 1745-1337
VL - E105-A
IS - 8
JA - IEICE TRANSACTIONS on Fundamentals
Y1 - August 2022
AB - Deep unfolding is a promising deep-learning technique, whose network architecture is based on expanding the recursive structure of existing iterative algorithms. Although deep unfolding realizes convergence acceleration, its theoretical aspects have not been revealed yet. This study details the theoretical analysis of the convergence acceleration in deep-unfolded gradient descent (DUGD) whose trainable parameters are step sizes. We propose a plausible interpretation of the learned step-size parameters in DUGD by introducing the principle of Chebyshev steps derived from Chebyshev polynomials. The use of Chebyshev steps in gradient descent (GD) enables us to bound the spectral radius of a matrix governing the convergence speed of GD, leading to a tight upper bound on the convergence rate. Numerical results show that Chebyshev steps numerically explain the learned step-size parameters in DUGD well.
ER -