The bards and scribes who recorded events for posterity received their income essentially in the form of tips from the rulers and the rich, so had an incentive to flatter, describing their patrons as more attractive, virtuous, brave, etc than they really were.
In addition to the exaggeration of their actual attractiveness in reports that have reached us, the children of the wealthy probably really were more beautiful than the poor. Richer youth were better fed and cared for, thus had fewer developmental abnormalities (e.g. bent legs from calcium deficiency) and diseases. The poor were malnourished, lived in dirty conditions and were subject to violence, therefore were more likely stunted, stank and had skin diseases, missing teeth and scars. The latter two distinctions in looks have to a lesser extent lasted to the present day, for the same reason.
Attractiveness consists of the visual, audible and olfactory signals of a fit mate (healthy, fertile conspecific), because organisms evolved to consider fit mates attractive. In times when most people were malnourished and diseased, a well fed and healthy rich person would have been much fitter than most, thus a preferred sexual partner for others.
On the other hand, conditional on surviving to adulthood, the poor likely had better immune-related genes, because they were under stronger selection pressure. Poorer people also experienced more infections, thus acquired stronger immunity to more diseases if they survived. Then conditional on equal looks, a person from a poorer background would have been a fitter mate. Also, the ruling class intermarried to keep wealth in the family, so were inbred (hereditary diseases among European royalty are an example consequence). For these two reasons, it is not surprising that the rulers and the rich found some poor people sexually attractive, specifically the outwardly healthiest-appearing among those who reached maturity.