I asked him to be in a ... what do I do now? Guys POV appreciated?

I don't think you scared him off. He had to go back to work. Let him absorb what you have proposed, give it a few days. If he still agrees, have a "meeting" with the three of you to lay the rules down as to what is/isn't expected. Just discussing it together may get the three of you rolling around before you know it.