Comparative Study of Deep Learning Parameter Selection for Multi-Output Regression on Head Pose Estimation

This paper presents a systemic analysis on the implications of the loss function, batch size and optimiser function applied within the multi-output regression problem domain of Head Pose estimation. To the best of our knowledge, prior comparative analysis of this calibre has primarily focused on the image classification problem domain, with limited analysis on the optimisation of hyperparameters influencing the performance of a Convolutional Neural Network (CNN) for regression. This motivates the current study which proposes a vanilla regression model based on the EfficientNet models of varying depths between EfficientNet BO and B5, and experiments on multiple protocols incorporating the benchmark datasets of BIWI, 300W-LP and AFLW2000. Therefore, the main contributions of the paper include a study on the impact of: loss functions for models of varying depths; batch sizes for significantly less and more complex models; and different optimiser functions on the model performance. The investigation of the effect of Mean Squared Error (MSE), Mean Absolute Error (MAE) and Huber loss functions on the performance of the models suggested that MAE and Huber loss (d=1.0) yield optimal performance regardless of the model complexity. The comparative analysis based on batch sizes ranging between 8 and 128 evidenced that moderate batch sizes of 16 and 32 yield an optimal performance on less complex models, while large batch training complements more complex models across all tested loss functions, an observation not evident in previous work. Finally, it was proven that the Adam optimiser function generates a minimal loss and is void of overfitting, outperforming other optimiser functions of Stochastic Gradient Descent, RMSProp, AdaGrad and Adam.