Deep Sentence Denoising Beyond Grammatical Error Correction
Open AccessMisunderstanding happens all the time, especially when a non-native speaker is involved. It is a bare-minimum prerequisite for clear and efficient communication to make sure that sentences be grammatically correct, and usually more (than grammar) is needed to ensure correctness and fluency, especially for non-native speakers. Thus, we propose a new task – Sentence Denoising, to go beyond Grammatical Error Correction (GEC). This research consists of four parts: As the basis for everything, we motivate the sentence denoising problem and explain how it differs from GEC. We develop a two-layer noise taxonomy and definitions that draw intuition from a linguistic perspective, with the goal being sentence denoising. It is useful and realistic because it is derived from the literature on second language acquisition. Deep learning has shown to shed light on similar problems like GEC, so we choose to explore it in sentence denoising, which is more difficult because of inadequate public data. We overcome it by developing techniques to inject single noise into sentences from clean and monolingual corpora using the building blocks from Natural Language Processing, which can then be chained to generate mixed types of noise. We mainly deal with individual noise in our first attempt in sentence denoising and pick 3 categories of noise in an English sentence, namely, synonym noise, misordering (i.e., words-out-of-order) noise, and declarative-question noise. Neural network architectures are benchmarked on the two simple tasks – noise classification and denoising (with end-to-end models). Good accuracy for both tasks implies that the sentence denoising problem is solvable with deep learning approaches. A system comprised of two parts is designed for sentence denoising – one being fine-tuned BERT models for noise classification, and the other being Transformer models for “translating” noisy sentences into correct ones. We obtain qualitative results from it on correcting multiple types of noise from sentences. We then raise our work to a higher level of mixed noise. In addition to the 3 types of noise, one more category of noise, verbosity noise, is added to the noise collection. We focus on eliminating the inefficiency of our previous approach while evaluating the systems in a quantitative way. We define different denoising objectives and derive a synthetic dataset that can serve as a surrogate for real-world noisy data. We then propose several pipelines as alternative to end-to-end models to draw insights on how to decompose a problem for which the required training data increase exponentially. Better results for denoising both individual and compound noise compared to our baseline work are obtained with an efficient training approach that only utilizes linear amount of data. Lastly, we examine and summarize the advantages and disadvantages of all the denoising models so far, and draw insights on further improving our denoising system. As a result, we derive a unified end-to-end model based on Transformer. An efficient algorithm for constructing the training data is given, together with a separate fine-tuning step to get the ideal model. Our new model outperforms previous ones and keeps good accuracy even when the noise composition gets more complicated.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.