Legacies are actually treated mostly fairly.
If you're admitted to a school but then drop out, then we know tautologically that whatever caused you to do so was not measured by the admissions process, or else you wouldn't have been admitted in the first place. Actually graduating is an additional major barrier to entry. Whatever the unknown factors are (maybe drive for independent learning in an environment with more freedom? who knows, probably many different things), parents and children should correlate in them, and so legacy status should give real information about merit that grades and test scores do not.
Indeed, legacy students get the same college grades and drop out at the same rates as admitted non-legacies:
https://sci-hub.ru/https://doi.org/10.1016/S0165-1765(99)00207-4
^ n=12k paper looking at prestigious schools. Despite the admissions preference they get, legacies achieve roughly equivalent overall college grades and superior within-major grades than admitted non-legacies (see figs 1 & 2):
In the regression controlling for race and major and parental education however, legacies do worse by a very small amount. The major control is fair since some majors are easier than others, but the race and education controls are spurious; they only make sense if you believe that whites and asians and the children of highly-educated people receive good college grades for reasons other than the merit of these groups. Given the trends of american racial demographics over time (and of the strength of racial affirmative action policies), legacy admits will be a bit whiter than non-legacies and have more educated parents.
In any case, even with the spurious regression, this is only enough that the effect size becomes small (so small that p>.01 despite n=12k and outright p>.1 in the within-major model). That the effect size for college performance differences is so small in spite of the large admissions bonus means that even taking these (wrong) numbers at face value, legacy admits are treated almost entirely appropriately, and so the removal of the legacy bonus would increase the absolute value of the difference in college performance between legacies and non-legacies.
Note also that admitted non-legacies are a different population than the general applicant pool. Legacies generally have superior high school grades + test scores to the general applicant pool but are generally inferior to admitted non-legacies in high school grades + test scores; that this inferiority in high school performance results in equivalent college performance means that dropping the legacy bonus would result in legacies who have superior college performance. That legacies are admitted more than what would be otherwise merited by grades + SAT + etc alone is true and deserves to be called a bonus, but this does not contradict any of my arguments.
Now, is grade inflation one of the unknown factors legacies share with their kids? No. Any grade inflation from wealthy, aggressive helicopter parents should apply to highschool grades all the same, if not more, as kids have more independence in college. Moreover, most admitted students are not legacies, and the parents of most legacies were not themselves legacies when they were admitted.
Finally, if I could add my own two cents, it's actually a bad thing that the legacy bonus is only high enough to put legacies on even footing with other admitted students. A nation should put its ultra-competent in contact with its ultra-rich from a young age so that the nation's resources end up being marshaled most effectively.