Heh heh heh. If you "keep track of it in some manner", then its not off the record

Is what the customer wants reasonable? Does he have it coming? Can you work out a compromise? Does his version of what was discussed sound like something you might have agreed to and, in fact, you are in error? Will you be called a liar and damage the companies reputation if you deny whatever it is that you are talking about?
If you know for a fact that the customer is wrong in his version of the story then you may wanna stick to your guns.....if the damage from doing so won't be too great. Remember that pride cometh before a fall.
I avoid saying anything that I wouldn't put in writing. I offer to put things in writing all the time.....then there's no misunderstanding and it helps me stay in control of the situation. Never wait for a customer to ask/demand that something be put it writing. If you agree, then THEY are controlling the situation. In fact, I *prefer* settling complaints/problems by letter or email. A) written words are hard to dispute or misunderstand and B) sometimes I don't make the best decisions via the phone or face-to-face. A written offer/solution/response can be more carefully thought out.
For the record, I also often write apology letters to unhappy customers when we've goofed things up. I think it helps.
Good luck.
Doug