I have a HP8591A spectrum analyzer.When i put a 1Ghz signal in to input the signal jump from 800Mhz to 980 Mhz , very unstable .The calibration failed with "Cal signal not found" The CAL Output works on a other spectrumanalyzer HP8591 E.
you mention that your 300MHz output works on another spectrum analyzer, (SA), but what do you get on the faulty SA?
Is it only 800-980MHz that is unstable?
I would guess, but this is just a guess, that your input attenuator is broken since you mention that you get "Cal not found". When you have done a calibration with this broken attenuator then the SA will not be able to fullfill the calibration and you might see very strange behavior.
If you connect the CAL to the input, do you see anything then? Is there any difference if you change the attenuator? If you change the attenuator in 10dB steps will the level follow?
The attenuators can be purchased either as surplus on Ebay or the attenuator component from Agilent. It's tricky to exchange and often a lot of parts must be taken out from the SA to make this repair. Also there might be more errors on the SA as the input mixer since there is always a reason for an attenuator to break. If it's a high power signal that got into the SA then it can have destroyed both the attenuator and the input mixer, but both can be repaired.
If it's a faulty attenuator you'll be able to make the calibration again when it's fixed.
Try pressing the Frequency key and enter -2001 Hz, then Cal, More, Cal Freq & Amplitude. I may not have the key sequence correct because I do not have an 8591A in front of me. By entering this pass code, the analyzer bypasses an initial frequency accuracy test that the instrument firmware performs. Let this run and see if the internal calibration operates.
If not, does the frequency jump around if you inject a 1.2 GHz or 1.5 GHz signal? When the signal is jumping at 1 GHz, what happens if you change the span to 10.01 MHz versus 10 MHz? I believe the first I.F. is 2121.4 MHz, so you could tune the analyzer to 1 GHz, go into zero span and then measure the 1st L.O. on a separate spectrum analyzer. The L.O. should be at 2121.4 MHz + 1000 MHz = 3121.4 MHz and stable, is it?
You could also use the user level service passcode (to skip the 300MHz Cal Signal validity check) of -37 Hz, then CAL, More, Cal Freq & Amptd. This too can be used when the 300MHz signal is way off frequency due to bad cal data.