The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Les rapports de vraisemblance (LR), qui sont couramment utilisés pour le traitement des données probabilistes, sont souvent estimés sur la base des décomptes de fréquence d'éléments individuels obtenus à partir d'échantillons. Dans le traitement du langage naturel, un élément peut être une séquence continue de N articles, appelés un N-gramme, dans lequel chaque élément est un mot, une lettre, etc. Dans cet article, nous tentons d'estimer les LR en fonction de N-informations sur la fréquence des grammes. Une approche d’estimation naïve qui utilise uniquement N-les fréquences gramme sont sensibles aux basses fréquences (rares) N-grammes et non applicable à la fréquence nulle (non observé) N-grammes; ceux-ci sont respectivement connus sous le nom de problèmes de basse fréquence et de fréquence nulle. Pour résoudre ces problèmes, nous proposons une méthode de décomposition N-grammes en unités d'articles, puis en appliquant leurs fréquences avec l'original N-fréquences de gramme. Notre méthode peut obtenir les estimations des N-grammes en utilisant les fréquences unitaires. Bien que l'utilisation uniquement des fréquences unitaires ignore les dépendances entre les éléments, notre méthode tire parti du fait que certains éléments coexistent souvent dans la pratique et maintient donc leurs dépendances en utilisant les éléments pertinents. N-fréquences de gramme. Nous introduisons également une régularisation pour obtenir une estimation robuste pour les cas rares. N-grammes. Nos résultats expérimentaux démontrent que notre méthode est efficace pour résoudre les deux problèmes et peut contrôler efficacement les dépendances.
Masato KIKUCHI
Nagoya Institute of Technology
Kento KAWAKAMI
LINE Corporation
Kazuho WATANABE
Toyohashi University of Technology
Mitsuo YOSHIDA
Toyohashi University of Technology
Kyoji UMEMURA
Toyohashi University of Technology
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Copier
Masato KIKUCHI, Kento KAWAKAMI, Kazuho WATANABE, Mitsuo YOSHIDA, Kyoji UMEMURA, "Unified Likelihood Ratio Estimation for High- to Zero-Frequency N-Grams" in IEICE TRANSACTIONS on Fundamentals,
vol. E104-A, no. 8, pp. 1059-1074, August 2021, doi: 10.1587/transfun.2020EAP1088.
Abstract: Likelihood ratios (LRs), which are commonly used for probabilistic data processing, are often estimated based on the frequency counts of individual elements obtained from samples. In natural language processing, an element can be a continuous sequence of N items, called an N-gram, in which each item is a word, letter, etc. In this paper, we attempt to estimate LRs based on N-gram frequency information. A naive estimation approach that uses only N-gram frequencies is sensitive to low-frequency (rare) N-grams and not applicable to zero-frequency (unobserved) N-grams; these are known as the low- and zero-frequency problems, respectively. To address these problems, we propose a method for decomposing N-grams into item units and then applying their frequencies along with the original N-gram frequencies. Our method can obtain the estimates of unobserved N-grams by using the unit frequencies. Although using only unit frequencies ignores dependencies between items, our method takes advantage of the fact that certain items often co-occur in practice and therefore maintains their dependencies by using the relevant N-gram frequencies. We also introduce a regularization to achieve robust estimation for rare N-grams. Our experimental results demonstrate that our method is effective at solving both problems and can effectively control dependencies.
URL: https://global.ieice.org/en_transactions/fundamentals/10.1587/transfun.2020EAP1088/_p
Copier
@ARTICLE{e104-a_8_1059,
author={Masato KIKUCHI, Kento KAWAKAMI, Kazuho WATANABE, Mitsuo YOSHIDA, Kyoji UMEMURA, },
journal={IEICE TRANSACTIONS on Fundamentals},
title={Unified Likelihood Ratio Estimation for High- to Zero-Frequency N-Grams},
year={2021},
volume={E104-A},
number={8},
pages={1059-1074},
abstract={Likelihood ratios (LRs), which are commonly used for probabilistic data processing, are often estimated based on the frequency counts of individual elements obtained from samples. In natural language processing, an element can be a continuous sequence of N items, called an N-gram, in which each item is a word, letter, etc. In this paper, we attempt to estimate LRs based on N-gram frequency information. A naive estimation approach that uses only N-gram frequencies is sensitive to low-frequency (rare) N-grams and not applicable to zero-frequency (unobserved) N-grams; these are known as the low- and zero-frequency problems, respectively. To address these problems, we propose a method for decomposing N-grams into item units and then applying their frequencies along with the original N-gram frequencies. Our method can obtain the estimates of unobserved N-grams by using the unit frequencies. Although using only unit frequencies ignores dependencies between items, our method takes advantage of the fact that certain items often co-occur in practice and therefore maintains their dependencies by using the relevant N-gram frequencies. We also introduce a regularization to achieve robust estimation for rare N-grams. Our experimental results demonstrate that our method is effective at solving both problems and can effectively control dependencies.},
keywords={},
doi={10.1587/transfun.2020EAP1088},
ISSN={1745-1337},
month={August},}
Copier
TY - JOUR
TI - Unified Likelihood Ratio Estimation for High- to Zero-Frequency N-Grams
T2 - IEICE TRANSACTIONS on Fundamentals
SP - 1059
EP - 1074
AU - Masato KIKUCHI
AU - Kento KAWAKAMI
AU - Kazuho WATANABE
AU - Mitsuo YOSHIDA
AU - Kyoji UMEMURA
PY - 2021
DO - 10.1587/transfun.2020EAP1088
JO - IEICE TRANSACTIONS on Fundamentals
SN - 1745-1337
VL - E104-A
IS - 8
JA - IEICE TRANSACTIONS on Fundamentals
Y1 - August 2021
AB - Likelihood ratios (LRs), which are commonly used for probabilistic data processing, are often estimated based on the frequency counts of individual elements obtained from samples. In natural language processing, an element can be a continuous sequence of N items, called an N-gram, in which each item is a word, letter, etc. In this paper, we attempt to estimate LRs based on N-gram frequency information. A naive estimation approach that uses only N-gram frequencies is sensitive to low-frequency (rare) N-grams and not applicable to zero-frequency (unobserved) N-grams; these are known as the low- and zero-frequency problems, respectively. To address these problems, we propose a method for decomposing N-grams into item units and then applying their frequencies along with the original N-gram frequencies. Our method can obtain the estimates of unobserved N-grams by using the unit frequencies. Although using only unit frequencies ignores dependencies between items, our method takes advantage of the fact that certain items often co-occur in practice and therefore maintains their dependencies by using the relevant N-gram frequencies. We also introduce a regularization to achieve robust estimation for rare N-grams. Our experimental results demonstrate that our method is effective at solving both problems and can effectively control dependencies.
ER -