It seems that you used different criteria during training and testing as the code below shows:
IN TEST:
scores = np.dot(vecs.T, qvecs)
IN TRAIN:
dif = x1 - x2
D = torch.pow(dif+eps, 2).sum(dim=0).sqrt()

   y = 0.5*lbl*torch.pow(D,2) + 0.5*(1-lbl)*torch.pow(torch.clamp(margin-D, min=0),2)
   y = torch.sum(y)

I did not get it why you do so?