I've followed this thread with interest and now have an update to my case, and hopefully someone can suggest something. It seems different people have had different problems and solutions. I have a 2017, bought new, with noticeable static/slow speed wobble, no high speed wobble; and suicidal off road tendencies. I would like to correct the problem, but have begun to shop for another bike.
I have a KLR650 with 10K miles, with a Cogent Moab on the rear and cartridge emulators on the front. I just took a 1600 mile trip, carrying about 60-70 pounds, with about 13 lb in each side bag and the bulk of the rest right behind me in a duffle. Side racks keep the bags off the plastic. They mount from one of the subframe points in front to the rear point where the OEM rack attaches. On top of the OEM rack I have a Nelson Rigg soft tail pack, bolted to the KLR rack. In it I carried a few pounds of miscellaneous stuff. Not heavy. Bike has proper sag. Up front I have a Happy Trails skid plate and nerf bars, which mount to the downtube. I have a steering damper. Bike was perfectly stable on pavement--straights and turns--up to 75 mph, but horribly unstable on dirt. (With 14-43 gearing I got 50 mpg at 60-70 mph.).....
My bike was also fine on the road and death in the dirt. My short answer to you is get the CG low, don't bolt the bag to the rack, use bags that sit on the rear seat and rack simultaneously, and strap them tightly with no stacked bags.
The long version:
FWIW, there are a couple things going on on my bike that may be applicable to your:
1. The heavy Tusk panniers (30# each loaded including pannier and lid weight) cause the bike to oscillate. This oscillation is not fatal....it is a pain in tough slow going, but doable. I am culling out my equipment and will drop the weight down to minimize the oscillation.
2. Even 10# in a topcase bolted on my rack (with panniers on...I did not test the rack weight without panniers, as I am a camper) turns my bike offroad into the devil's death drop of wobbly oscillating horror, with cyclic steering inputs to counteract the weight in the topcase on the rack. I know well the white knuckle, broken frame feeling, oscillation you describe offroad.
I took my bike to the local OHV park to more rigorously test it without the topcase, with forks at 15mm raised in clamps, and stock rear suspension at 5. I headed straight to the single track with gnarly rocks and holes and what not. This is what I found:
1. With SW drybag 260 on top of drybag 350 (both full) and at right angles to bike on the rear seat and rack, and full panniers: a bit of an oscillation at slow speeds offroad. Nothing like 10# in the topcase but noticeable enough to require concentration and correction. Not happy times.
2. No drybags on the bike, with full panniers: perfectly rideable with no oscillations except if I did u-turns in the rocks, and then very minor and not really impacting riding. I just noticed it. No drybags would make camping tough.
3. I put all of the 260 items into the 350, so retained all the weight, and rode with the 350 only across the rear seat and rack, with full panniers: This made a substantial difference. My KLR does not like the 260 being up on top of the 350, whether because of higher CG or the instability inherent in the strapping system between the two bags.
In this configuration I was up on the pegs in 2nd gear (it was a tiny area) bouncing off the rocks in the path as I zoomed along. This was previously impossible with the bolted topcase with 10#, as you know.
4. While 15mm raised forks are great for testing slow speed handling in the dirt, they are death on the highway. Don't raise them this much.
5. The kids on the JR50s and CRF50s thought I was crazy beboping around their single tracks on my leviathon.
My opinion, and you know what that is worth, after all the work and tests I have done, is 1. the KLR needs to have weight down low (major). 2. The rack is really only mounted with the two front bolts and the two 6mm bolts on top, and the mounting of the tusk panniers changes the mounting system. 3. The stock rack is flexible (minor) and I will no longer bolt anything to it on the top. 4. The tusk panniers full make the bike frame oscillate from their inertia (60# combined has leverage when it is 8 inches from the side of the MC) but is not fatal. 5. The high up weight bolted on the rack is above the CG and the flexible rack is mounted weakly and it oscillates the bike side to side in the vertical dimension (metronome movement) of the frame whereas the panniers down low oscillate in the horizontal dimension of the frame (fish movement).vertical movement is bad and horizonal...not so bad.
I just installed a precision rack and hucked the stock rack. I also lowered the fork tubes back to stock, put the 10W back in the forks, and put on a top gun spring on the rear solely to get the proper rear sag with camping load (before I drop cogent cash). I am going camping at the end of September with the full panniers, 350 drybag, tank bag, and 8L drybags on the crash bars. I will cull out some of the weight of the camping equipment and put some heavier items in the crash bar bags.
All this assumes you have setup the entire rest of your bike properly, of course. stem tight proper, bearings good, etc., etc., etc.
I'll let you know how it goes on my NV ride, but based on my OHV testing, it should be fine.