Kill my boyfriends social life, or continue to feel left out?

My live in boyfriend of over two years decided a few months ago that he was going to start smoking weed regularly to try to help his anxiety. I dont have anything against weed or any kind of mind altering substances for that matter but I just can't stand how weed in particular makes me feel so socially awkward. So fast forward to now and it's turned into a nighty social event at our house. He invites people from work and random new people that he meets that smoke weed over. I'm really feeling left out because I can't join in and look like the lame gf that always goes to sleep. I'm admittedly jealous of his suddenly booming social life. When I invite people over for drinks and monopoly nobody ever shows up. What should I do? We love each other and live together, I want it to work but asking him to stop and give up his social life seems unfair just as much as it is unfair for me right now.