With small children, I think it's extremely difficult to share Christmas.  They can't understand why they have to go and leave most of their new stuff behind. It gets easier when they get older, but as my 15 year old just told me, "it still sucks".

My ex was very difficult.  In the end, we wound up with a very specific custody order, with times and dates and behavior restrictions for dad (no drinking, no drugs, no spanking, no smoking), which makes it very easy to deal with him.  We can change things on our own if we agree, but if he starts playing games, I just say "I can get out the order and we can just go with what that says" and he calms down and gets more reasonable.  And time and AA have curbed a lot of his worst behavior.

For Christmas, we go with the same schedule every year so that both parents can develop some traditions and the kids know what to expect every year (they're 12 and 15 now, but we split up when they were 2 and 4 1/2).  I took Christmas eve (we celebrate with my family then) to noon on Christmas day (in part because of his tendency to try to make everyone feel bad when they left, not something I think should be part of anyone's holiday), and he took Christmas day overnight to the 26th.  It's been the best solution for all of us. He has the extra time to get ready, the kids are fresh for getting together with my family (he has none in the area), and of course, I like the relaxing afternoon on Christmas day after everyone leaves.

ps- if your ex stresses out your holiday, don't bother inviting him to spend it with you.  It's your holiday too, you should enjoy it, and you don't need to be a martyr.