Over the last few weeks, as the deadline for the congressionally mandated budget cuts known as the “sequester” came and went, we got a taste of how difficult cutting federal spending actually turns out to be. The news is disconcerting: thousands fewer food safety inspections, some 70,000 fewer kids in early education programs, people with mental illness losing access to treatment, civilian employees of the military furloughed, ships and aircraft going without maintenance… It’s a long and dispiriting list.

Yet as painful as the sequester might be, most policy-makers know that it is not the main event when it comes to our fiscal challenges. Discretionary spending, the kind getting cut in the sequester, amounts to less than a third of federal spending.

That’s not what many people believe, of course. Whenever I give talks about the federal budget I’m taken aback by where my listeners think most of our money gets spent. At two meetings recently, members of the audience stood up to complain that if we just cut what we give away to other countries in foreign aid, we could resolve our budget issues. This isn’t even close to the truth: altogether, we spend well less than 1 percent of the federal budget on foreign aid.

If you think of federal spending as a pie, by far the biggest slices go to Social Security and unemployment support, Medicare, Medicaid, and other health programs, which altogether make up well over half. Military spending accounts for about another quarter, while the next biggest slice, about 7 percent, is for interest on the federal debt — a figure that will explode in upcoming years. Everything else we think of as the federal government — spending on highways and the aviation system, money for student loans and other education programs, housing, food stamps, medical research and, yes, foreign aid — comes in at under one-fifth of the total.

The biggest driver of growth in federal spending, as Nate Silver of The New York Times pointed out in a thoughtful analysis in January, is entitlements: Medicare, Medicaid, Social Security and other social insurance programs. This is especially true of health-care, which accounted for about half of the increase in federal spending relative to the economy over the last 40 years. We cannot get control of federal spending without reining in health-care spending — and though its rate of increase has slackened over the last few years, no one knows whether it’s a permanent or temporary change.

So if Congress and the White House are serious about tackling federal spending, then the piece they left out of the sequester —entitlement reform — must be on the table. But it’s been hard to tell from their actions that they’re really serious. Members of Congress have been taking to the airwaves for weeks to decry the sequester’s meat-cleaver approach to budget-cutting, yet most of them voted for it. That’s because it’s simpler to impose across-the-board cuts than to make discriminating judgments about individual programs. Members find it easy to demand cuts in federal spending in the abstract, but painfully difficult to cut specific programs.

Americans as a whole do, too: a recent Pew Center poll found they approve cutting government spending in general, but when asked about specific programs, they want to boost funding or keep it the same. Americans are demanding that government cut spending without cutting actual programs.

This is why it takes extraordinary leadership to address our fiscal issues. Americans may bear some responsibility, but our leaders have not leveled with us about what it takes to get a sensible budget and put the economy on a path to recovery. I am hard-pressed to think of an example of government failure to match our political leaders’ inability to lead us to a solution.

Their prolonged fighting is causing businesses to hesitate, workers to remain in limbo, and an economy that needs a boost to continue to stutter. They are denying us the ability to invest in our future, promote economic growth, and deal with the many other challenges our nation faces. Let’s stop the blame game and get to work.

Lee Hamilton is Director of the Center on Congress at Indiana University. He was a member of the U.S. House of Representatives for 34 years.