Historically, I measure the NEXT and Return Loss of a differential twisted pair using a balun. 1 port calibration is performed using an open, short and 100R load using 100thou header pins, with the thru measurement with the baluns back-to-back.
I dont have access to an LCR meter, and I dont know how to characterise the calibration artifacts so I havent selected a cal kit. I'm using the E5071B ENA. Frequency range is from 1MHz to 1GHz. Baluns used are the BH Electronics 040-0192. Test leads are RG402 50R Semi-rigid Coax approx 150mm long. Twisted pair is 1km long, nominal 100R characteristic impedance. Far end is terminated with 50R from each wire down to ground.
I dont know what other information to give, and I have run out of ideas.