When I bought my used 08 in 09 I paid 3900 and the bike had 2900 miles (came only with KLR saddlebags). The dealer had new (never serviced) 07s that I talked down to 3K before reconsidering. If I was going to buy the bike for mostly dirt I'd buy the KLX 250 or Honda XL650 (because their both more dirty than flirty - made that up myself). The 06 should be negotiable below 3K (in my opinion) and that would leave you some money for doohickys and such. I've actually looked at an 05 as a base platform for a tracker project. Just can't figure the tank/front end cosmetics out in my head.
"My KLR belongs to God, when he wants it clean he washes it."