It is really easy. If you want to find them, and remove them. First, go to the profile of the inactive ghost follower that you want to force to unfollow you. Next, tap on the three dots in the top right corner. Then, tap Remove Follower. And finally, tap Remove.