The trouble here is you are using electrical delay to do a "port extension" function, which is the cause of your problems. You can use electrical delay to remove delay from a measurement, and it will have 2:1 phase response for reflection measurments and 1:1 for transmission. But that is not the proper function to use. If you use port extension to remove delay from your measurement then it automatically accounts for relfection vs. transmission measurment. Stop using electrical delay and try port extensions for this purpose. I think you will find your troubles go away.
I tried to measure the DUT without normalization with using port extension. By input 36 ps or 18 ps in Extension input A, I still do not have correct phase result for S11. (in the link http://www.vnahelp.com/tip9.html, it says I should use 18 ps to achieve 36 ps ED for S11)
Then, I measured the DUT with normalization. By input 18 ps in Extension input A, I can have correct phase result for S11 now.
But I am still confused by the normalization, electrical delay and port extension even I have the correct phase result now. Is the last method by which I can have good result is correct?
Still not doing it right. Don't use extension A or B. Use PORT 1 extension or Port 2 extension, as you are extending the test port, not the receiver.
But when I input 18ps (half electrical length of single connector) or 36 ps (electrical length of single connector) in Port 1 extensions, I can not have measured result similar as the simulation (10 degree difference) with or without the normalization.
If you want to measure the delay of a DUT with an SMA connector, using S11, then you must do a calibration without a 3.5 mm calibration kit, don't do any normalization.
Measure S11, put up a "delay" trace and use trace statisitics to measure the mean value.
Remember that your DUT can have radiation and fringing that can add phase rotation, increasing the apparent delay.
Don't use port extension on A, use port 1 (extension on A is intended to be used when cables are added behind the directional coupler). Just because it gives an answer you like, doesn't mean it is the correct method.
Remember that your simulation is not the real thing. No reason the simulation should match the DUT response if the simulation is not correct.