The next option was of course creating a view with various levels of linked entities and set the IsDefault of the team entity to No, so that not all team members will be retrieved.
Since this was a comments system and the application had a lot of teams for different Roles and Categories, so setting the IsDefault attribute to No would not solve our problem because we would still get the Role based users.
What we ended up doing is excluding from the view all of the Teams that are "Role based", which included Users spanning across different categories. by doing so, we were able to retrieve the expected results.
When looking at the view, we will be able to see all the records that are assigned to other members of your team.