I have been going over this debate myself. I don't know much about the Xterra but was considering it. I have decided on the FJ because it is just me and I don't have to worry about hauling kids. So it is a perfect vehicle for me and some groceries
Now if your wife will be doing a lot of hauling the kids around I would agree that it MAY not be the best choice. Still a great SUV though. Good luck in your decision.
Also, as far as pricing goes I am shopping right now for a left over 07 and the prices I am finding are $1600 - $1900 below invoice.