The overuse of "per se" has got to stop

When people use the expression "per se" a lot...it irks me. Why do they use it, per se, a lot? I don't know. It doesn't make you sound smart, because a lot of people use "per se", but they think it's spelled out at "per say". They don't even know how to spell their "exclusive" vocabulary terms. Dumb.