The poor don’t have much wealth or income, so not much to tax. The rich find it worthwhile to hire accountants and lawyers to find loopholes useful for avoiding taxes. Optimizing one’s taxes is almost a fixed cost (does not depend much on the wealth or income), but the benefit is proportional to what is taxed. In other words, there are economies of scale in tax evasion. The middle class is not rich enough for professional loophole-seeking to be profitable, but has enough to tax.
This is only theoretical reasoning. Empirically, the average tax rate increases in income, as far as I know. Whether the rate of increase should be higher or lower than at present is a matter of political preference.
To avoid the unequal treatment based on the ability to hire tax advisors, the tax system should be simple. Changing income from one form to another (salary to dividends for self-employed for example), or changing the source (foreign corporation vs its domestic subsidiary) should not change the tax rate. Minimizing loopholes means no special deductions for belonging to select groups (farmers, pensioners, veterans, parents).
The existence of a profession for providing tax advice to ordinary people is a sign that the tax code is too complex. Those in a common salaried job should be able to understand and file their taxes themselves.