In fiscal year 2022, the IRS employed 79,070 full-time employee equivalents, and had $14.3 billion in expenditures. That’s a lot of people and a lot of money, but the job is a big one: $5.03 trillion was collected by the federal government last year.
If the IRS were a business, they’d be the scaling world champs, with an average revenue per employee of over $63 million. However, the IRS does not collect all taxes that are due by law. The actual shortfall in tax collections takes years to calculate, but the IRS estimates that the gap for the years 2017 – 2019, after enforcement and collections, will be 87%, or $470 billion. Even for a behemoth (the U.S. Government spent $6.27 trillion in 2022), that is a spectacular amount of money.
Given all the money that goes uncollected, it’s not a big surprise that the Inflation Reduction Act earmarked $45.6 billion of new money over the next 10 years for tax collection enforcement. The IRS already makes significant effort to determine whom to audit, yet only about one in 300 returns are examined each year.
157,500 returns were filed in 2020, and about 452,000 examinations were closed that year. “Examination” is another word for audit, and the average additional tax recommendation was $12,230. However, 3.6% of the examinations closed resulted in an average refund to the taxpayer of $49,862.
The total of those refunds was over $800 million, so it’s reasonable to assume that many American taxpayers overpay significantly. After all, it’s not in the interests of the IRS to search for taxpayer overpayments. Presumably, the IRS is looking for those returns most likely to yield additional collections.
The billions of dollars that American taxpayers overpay their taxes is known as the Reverse Tax Gap. The precise number is difficult to come by, but it is undoubtedly very significant. Our concern is that you may be needlessly overpaying tax due to inefficiencies in the way your financial and investment accounts are structured and managed.
Taxes First, Then Math: The Way of the Family Office
Much of our tax compliance work is backward-looking. Clients provide their 1099s for interest earned and capital gains, along with other information, and we do our best to prepare returns as diligently and accurately as we can in an effort to make sure you pay only what you are legally obligated to pay.
Professional assistance with tax preparation can help you avoid mistakes, both in the application of tax laws and in calculations. This can give you comfort that you are not overpaying what you should pay, based on the inputs you provide, but it may not do much for what you could have paid, if the structure of your financial accounts were specifically designed to minimize your total tax bill.
It’s ironic that the wealthiest Americans—those who can most easily afford to pay taxes—tend to be the best at tax avoidance. We are not suggesting that they evade taxes, which is illegal. But we can all take heart from the words of Supreme Court Justice George Sutherland, who said, “The legal right of a taxpayer to decrease the amount of what otherwise would be his taxes, or altogether to avoid them, by means which the law permits, cannot be doubted.”
Many billionaire families have their affairs managed by a family office, not by a bank, a brokerage house, or an insurance company. The family office operates as a fiduciary, meaning they are obligated to act only in the best interests of the family they serve. In this way, transparency is maximized, and potential conflicts are minimized.
Family offices typically (if not always) employ strategic tax professionals, as well as asset and risk managers. They work together to solve problems holistically, and given that their clients live at the high end of the tax brackets and that the top federal estate tax rate is 40%, taxes are of major concern.
In fact, taxes are the first concern in the family office process, and it’s easy to understand why. If you were facing a multi-billion dollar tax hit, wouldn’t you seek help avoiding it?
We believe that virtually everyone wants to minimize their tax bill, and mass-affluent families, those who enjoy a comfortable lifestyle, will feel the impact of overpaid taxes more acutely than a billionaire would. Even people who are “set for life” often have preferences regarding their legacy and the disposition of their assets after death.
Family offices are concerned with more than taxes, of course. They will work to protect their client’s assets from creditors, for example, and may also want to put certain restrictions on beneficiaries. Putting the structure upstream from investment policy can also have the benefit of making investment selection more well-defined.
Wealth strategists in family offices will concern themselves with asset location. This means, essentially, that it matters which types of assets go into which types of accounts. For example, you would likely not want to put muni bonds into a qualified account, like an IRA. Munis generally work better in taxable accounts, for obvious reasons.
On the other hand, dividend payers may work well in a qualified account, particularly if the dividends are being automatically reinvested. As a general rule, you want bonds, preferreds, and dividend payers in your retirement accounts, and growth stocks in your taxable account.
The reasoning behind putting longer-term investments in taxable accounts is simple: if you hold those assets for a year or more, any gains will be subject to the lower capital gains tax rate, which, for people still working, is typically lower than ordinary income rates.
Putting growth stocks in your taxable accounts also allows for the practice of tax loss harvesting. No one picks winners 100% of the time, even if they have every reason to believe in the stock they’re buying. Stuff happens, markets move fast, and sometimes a stock you love will drop in price. When that happens, it’s not all bad news; you can realize that loss, and use it to offset gains on your winners. If you still love the stock, you can buy it again in 31 days.
There are many kinds of assets, and many types of accounts. Straightening that out before making investment decisions is a wise policy and a common practice among the very wealthy. There is more on the table than just effective tax management. Diversification among asset classes is the art and skill of asset managers and financial planners whose goal is to create structures that dampen volatility and thereby increase compounding.
Many investors simply don’t know if their portfolio, and its management, is as tax efficient as it could or should be. Quite often, that can be fixed without taking on additional risk. By focusing on net returns, your money can work harder for you.
If you’re concerned about the tax efficiency of your portfolio, we’d be happy to provide you with a second opinion called our Taxes First, Then Math analysis. This report is free of cost or obligation, and it could help you enjoy the benefits enjoyed by the super-rich. If you have any questions about this topic or the report, use this link to book some time with me. If you’re ready to order your report, click here and follow the instructions.
