I'm just agreeing with so many others here - get TF out. He's a parasite.
I understand sometimes it's just not possible to pack up and move to a new home right away. You'll need some time to get your stuff together, save some money, etc. In the meantime, starting RIGHT NOW, stop doing anything for him. Don't make his appointments, don't buy his gifts, don't pick up his socks. He's a grown-ass man.
And don't leave your baby with him - that kind of neglect is actually considered abuse. Document it (nanny cam, maybe?) so he doesn't try the typical a-hole move of now that you're leaving, suddenly he wants custody of the kid just to spite you.
Best of luck. You deserve better.