  Generative probability models are widely used for speaker verification (SV).
However, the generative models are lack of discriminative feature selection
ability. As a hypothesis test, the SV can be regarded as a binary
classification task which can be designed as a Siamese neural network (SiamNN)
with discriminative training. However, in most of the discriminative training
for SiamNN, only the distribution of pair-wised sample distances is considered,
and the additional discriminative information in joint distribution of samples
is ignored. In this paper, we propose a novel SiamNN with consideration of the
joint distribution of samples. The joint distribution of samples is first
formulated based on a joint Bayesian (JB) based generative model, then a SiamNN
is designed with dense layers to approximate the factorized affine transforms
as used in the JB model. By initializing the SiamNN with the learned model
parameters of the JB model, we further train the model parameters with the
pair-wised samples as a binary discrimination task for SV. We carried out SV
experiments on data corpus of speakers in the wild (SITW) and VoxCeleb.
Experimental results showed that our proposed model improved the performance
with a large margin compared with state of the art models for SV.
