This is to impart the need for a higher hash rate than the rest of the (honest) miners. Mining success is probabilistic, however, so this 50% or 51% is an indication of the expected behaviour given an infinite number of attempts.
Using the scenario where an attacker tries to build their own chain to eventually replace the original one:
If the attacker has 50% of the network hash rate, the expected result is that the two chains will grow at the same pace, and the attacker's chain will never outpace the honest chain.
If the attacker has slightly over 50% of the network hash rate, the attacker's chain will outpace the honest chain, at a rate commensurate with how much over 50% the attacker controls.
If the attacker has less, then the attack chain will be left in the dust.
This is why it's called a 51% attack. 50% is the limit, but is not enough in this probabilistic model. Given the number is an integer, the next step up is 51. Calling it a 50.1% attack would be as wrong, and there's no quantum in real numbers that'd allow a 50.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001% attack to be used as a canonical term (you'd them get people asking why not a 50.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001% attack). Turns out humans have better things to do.
Now, the above is based on a model of the expected results given an infinite number of attempts. In practice, we never reach infinity, and so an attacker with 49% of the network hash rate may well succeed in getting ahead of the honest chain enough to replace theirs. So 49% is technically enough to get a short enough chain up some of the time. For similar reasons (after all, 48.5% is also enough some of the time), this number is also ignored in favour of the 51% memorable number. Humans remember them easier.