The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Les protocoles de routage simples existants (par exemple, OSPF, RIP) présentent certains inconvénients : ils sont rigides et sujets à la congestion en raison de la concentration des paquets sur des routeurs particuliers. Pour résoudre ces problèmes, des méthodes de routage de paquets utilisant l’apprentissage automatique ont été proposées récemment. Par rapport à ces algorithmes, les méthodes basées sur l’apprentissage automatique peuvent choisir intelligemment un chemin de routage en apprenant des itinéraires efficaces. Cependant, les méthodes basées sur l’apprentissage automatique présentent un inconvénient en termes de temps de formation. Nous nous concentrons ainsi sur un algorithme de machine learning léger, OS-ELM (Online Sequential Extreme Learning Machine), pour réduire le temps de formation. Bien que des travaux antérieurs sur l’apprentissage par renforcement utilisant OS-ELM existent, ils présentent un problème de faible précision d’apprentissage. Dans cet article, nous proposons OS-ELM QN (Q-Network) avec un tampon de relecture d'expérience priorisé pour améliorer les performances d'apprentissage. Elle est comparée à une méthode de routage de paquets basée sur un apprentissage par renforcement profond utilisant un simulateur de réseau. Les résultats expérimentaux montrent que l’introduction du tampon de relecture d’expérience améliore les performances d’apprentissage. OS-ELM QN atteint une accélération 2.33 fois supérieure à celle d'un DQN (Deep Q-Network) en termes de vitesse d'apprentissage. Concernant la latence de transfert de paquets, les OS-ELM QN sont comparables ou légèrement inférieurs aux DQN alors qu'ils sont meilleurs que les OSPF dans la plupart des cas puisqu'ils peuvent répartir les congestions.
Kenji NEMOTO
Keio University
Hiroki MATSUTANI
Keio University
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Copier
Kenji NEMOTO, Hiroki MATSUTANI, "A Lightweight Reinforcement Learning Based Packet Routing Method Using Online Sequential Learning" in IEICE TRANSACTIONS on Information,
vol. E106-D, no. 11, pp. 1796-1807, November 2023, doi: 10.1587/transinf.2022EDP7231.
Abstract: Existing simple routing protocols (e.g., OSPF, RIP) have some disadvantages of being inflexible and prone to congestion due to the concentration of packets on particular routers. To address these issues, packet routing methods using machine learning have been proposed recently. Compared to these algorithms, machine learning based methods can choose a routing path intelligently by learning efficient routes. However, machine learning based methods have a disadvantage of training time overhead. We thus focus on a lightweight machine learning algorithm, OS-ELM (Online Sequential Extreme Learning Machine), to reduce the training time. Although previous work on reinforcement learning using OS-ELM exists, it has a problem of low learning accuracy. In this paper, we propose OS-ELM QN (Q-Network) with a prioritized experience replay buffer to improve the learning performance. It is compared to a deep reinforcement learning based packet routing method using a network simulator. Experimental results show that introducing the experience replay buffer improves the learning performance. OS-ELM QN achieves a 2.33 times speedup than a DQN (Deep Q-Network) in terms of learning speed. Regarding the packet transfer latency, OS-ELM QN is comparable or slightly inferior to the DQN while they are better than OSPF in most cases since they can distribute congestions.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2022EDP7231/_p
Copier
@ARTICLE{e106-d_11_1796,
author={Kenji NEMOTO, Hiroki MATSUTANI, },
journal={IEICE TRANSACTIONS on Information},
title={A Lightweight Reinforcement Learning Based Packet Routing Method Using Online Sequential Learning},
year={2023},
volume={E106-D},
number={11},
pages={1796-1807},
abstract={Existing simple routing protocols (e.g., OSPF, RIP) have some disadvantages of being inflexible and prone to congestion due to the concentration of packets on particular routers. To address these issues, packet routing methods using machine learning have been proposed recently. Compared to these algorithms, machine learning based methods can choose a routing path intelligently by learning efficient routes. However, machine learning based methods have a disadvantage of training time overhead. We thus focus on a lightweight machine learning algorithm, OS-ELM (Online Sequential Extreme Learning Machine), to reduce the training time. Although previous work on reinforcement learning using OS-ELM exists, it has a problem of low learning accuracy. In this paper, we propose OS-ELM QN (Q-Network) with a prioritized experience replay buffer to improve the learning performance. It is compared to a deep reinforcement learning based packet routing method using a network simulator. Experimental results show that introducing the experience replay buffer improves the learning performance. OS-ELM QN achieves a 2.33 times speedup than a DQN (Deep Q-Network) in terms of learning speed. Regarding the packet transfer latency, OS-ELM QN is comparable or slightly inferior to the DQN while they are better than OSPF in most cases since they can distribute congestions.},
keywords={},
doi={10.1587/transinf.2022EDP7231},
ISSN={1745-1361},
month={November},}
Copier
TY - JOUR
TI - A Lightweight Reinforcement Learning Based Packet Routing Method Using Online Sequential Learning
T2 - IEICE TRANSACTIONS on Information
SP - 1796
EP - 1807
AU - Kenji NEMOTO
AU - Hiroki MATSUTANI
PY - 2023
DO - 10.1587/transinf.2022EDP7231
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E106-D
IS - 11
JA - IEICE TRANSACTIONS on Information
Y1 - November 2023
AB - Existing simple routing protocols (e.g., OSPF, RIP) have some disadvantages of being inflexible and prone to congestion due to the concentration of packets on particular routers. To address these issues, packet routing methods using machine learning have been proposed recently. Compared to these algorithms, machine learning based methods can choose a routing path intelligently by learning efficient routes. However, machine learning based methods have a disadvantage of training time overhead. We thus focus on a lightweight machine learning algorithm, OS-ELM (Online Sequential Extreme Learning Machine), to reduce the training time. Although previous work on reinforcement learning using OS-ELM exists, it has a problem of low learning accuracy. In this paper, we propose OS-ELM QN (Q-Network) with a prioritized experience replay buffer to improve the learning performance. It is compared to a deep reinforcement learning based packet routing method using a network simulator. Experimental results show that introducing the experience replay buffer improves the learning performance. OS-ELM QN achieves a 2.33 times speedup than a DQN (Deep Q-Network) in terms of learning speed. Regarding the packet transfer latency, OS-ELM QN is comparable or slightly inferior to the DQN while they are better than OSPF in most cases since they can distribute congestions.
ER -