By default, all project-related tasks are visible to all project members.
In order to show the tasks only to staff members who are assigned/added as Follower, you need to turn off Allow all staff to see all tasks related to projects

To achieve this go to Setup -> Settings -> Tasks and => set Allow all staff to see all tasks related to projects to NO