Geometric Properties of the Gradient of Loss Functions in Discriminant Deep Neural Networks
Open AccessClassification is an important and challenging problem in the field of machine learning (ML). In recent years, deep neural networks (DNNs) are the rising stars for solving classification problems. DNNs marginally outperform the last generation ML approaches and even humans in certain applications due to the capacity of learning representations of the input data automatically.As a supervised learning approach, DNNs need to be trained to learn the parameters that map the input data to representations. Learning good representations that can facilitate the classification is the goal of training a discriminant DNN. The backpropagation establishes the learning rule in state-of-the-art DNNs, which updates the DNN parameters by a proportion of the negative gradients of the loss function with respect to the parameters. Therefore, the loss function plays a key role in the training of DNNs; it determines the direction of updating the parameters and the form of representations. This thesis analyzes the geometric properties of the gradient of loss functions for discriminant DNNs. Based on the properties, a set of new loss functions is proposed to obtain better representations for classification. By analyzing the properties of the cross-entropy loss function, which is the most popular loss function for the discriminant DNNs, the approximations of its gradient are proposed that overcome the vanishing gradient problem and accelerate the training of DNNs.Adversarial examples are the instances with small, intentional feature perturbations that cause a machine learning model to make false predictions. The vulnerability to the adversarial examples is one of the issues that limit the application of DNNs. By analyzing the geometric properties of the decision boundaries in the representation space, a new approach of generating the adversarial examples is proposed that aims to monotonically move the representation close to the decision boundaries. By using the approximation of the gradient of the cross-entropy loss function, a new form of the representations that are far from the decision boundaries can be obtained, which robustifies DNNs against adversarial examples.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.