I always start questioning going back. Its like I can feel myself retreat a little into myself (self-protection mode)- and its a little bit of a relief. I hate needing her, I hate wanting her- all of that. So when it cant happen for a while and I start getting used to that, it seems a little absurd to want to go back to that agony. I start to think -I have enough "real" problems as it it- I dont need to add agonizing over my relationship with her to that mess. I get hooked on thinking how the relationship isnt even "real".
In addition, both on vacations or not, I (very often) struggle with the idea that there will be an end to this relationship- I mean thats not a maybe... one day she will retire and I will have to stop going. That just about paralyzes me with sickening fear. I start thinking about this alot when she's on vacation and I start to think- this is my chance to escape- I could not go back... I'm kind of walled up here to deal with her absence... why in hell am I going to go back and start feeling all that need and want, and then start to worry again about when it wont be in my control to quit?
When I do go back it can take weeks for me to lower my guard... that is getting a little better (it takes less time)... but its still a thing for me. Loss is thing, trust is a thing- there are a lot of things that get riled up in this for me.
So- I'm sorry, I dont know that I've said anything helpful here- I really dont have any ideas on how to make this better- but for what its worth, you are definitely not alone in this.