Sirens cannot confuse you. You have to be doing something else to get confused.
The loss of stamina I feel is just okay, because if it was weaker, sirens would have no chance to actually make a visible dent into your stamina reserves (just look how long it takes for you to run out of stamina later in the game where sirens spawn more often). And it's still very easily counterable with water if you cannot kill the siren before becoming Exhausted.
I actually think the only nasty effect is the charming of pets. Sirens won't even run away when they steal something, and are fragile enough to kill even if they steal your weapon.