When I increase the input level from -40 dBm to -20 dBm, why does the second harmonic jump up on the spectrum analyzer?
The excessive second harmonic at -20 dBm is caused by the input mixer of the RF Analyzer being overdriven.
With an input of -40 dBm and Input Atten at 0 dB, the mixer is not overdriven and the second harmonic is essentially in the noise at >65 dB down (with Span set to 10 MHz and Ref Level set to -30 dBm). When you set the input to -20 dBm with Input Atten at 0 dB, the level is too high for the input mixer and the second harmonic jumps up to about 30 dB.
You would think that Input Atten would autorange and reduce the excessive input at -20 dBm. It does for the RF Analyzer, but not for the Spectrum Analyzer. Auto Input Atten does not function in the Spectrum Analyzer. Notice that when you switch from -40 to -20 dBm that Input Atten autoranges to 20 dB on the RF Analyzer but not the Spectrum Analyzer. (To observe that the Input Atten on the Spectrum Analyzer does not autorange, on the Spectrum Analyzer screen set Controls to Auxiliary.)
To get the better harmonics performance at -20 dBm, set Input Atten to Hold at 20 dB and increase the Ref Level to -10 dBm. That way the signal level into the input mixer of the RF Analyzer will be the same as it was for -40 dBm.