Did you spend very much time with the Rav 4 ? I would borrow if for a full weekend before you make any decisions.
IMO.. I had an 2008 Rental Rav4 for 2 weeks with the V6 and FWD. It was really nice. and the single button glove box just blows my mind

.... I was thinking about trading my 01 HL in for one right at first because their very comparable in size/power/MPG..ect. But after about 10 days i missed the comfort of my HL. The Rav4 is just not the same and kinda left me wanting more. Its hard to expain other than when i had a honda accord, it nice and new but was just simply boring to drive.
There is just something about the HL that is super smooth and makes me FEEL very relaxed when driving. Needless to say i'm keeping my HL until it dies.