Part of a successful career trajectory in physics is building and maintaining a professional network of peers, collaborators, and mentors that supports one’s professional growth and advancement. Building these networks can be challenging for minoritized groups, such as women and LGBT people, and may be one of the reasons leading to the challenges they face in physics. This study applies qualitative Social Network Analysis (SNA) to better understand how these groups build their social networks and the impact of these networks on their careers. In this presentation we focus on experiences contributing to the permanence of women and LGBT physicists in the field of physics, discussing how people in different sectors talk about their trajectories, challenges and ways how their institutions supported (or not) their identities and professional advancement.