NEW YORK—College endowments and state pension funds plowed billions of dollars into hedge funds and private-equity investments as a way to balance their stock holdings, and for a time they got supercharged returns.

Those days are over. From Harvard University to the state pension fund of California, officials are watching the value of their alternative investments shrink.

So far, the losses are mostly on paper, but analysts say they could eventually lead to reduced payouts to retirees, higher taxes so state governments can fulfill their promises, or less cash available for colleges to give out financial aid.

“Everyone was in a desperate search to find returns,” said Colin Blaydon, head of the Center for Private Equity and Entrepreneurship at Dartmouth’s Tuck School of Business. “Now they have to face the dirty little secret that those investments aren’t in great shape.”

In recent years, endowments and pensions heaped cash into hedge funds—private investment funds that often use unconventional and risky trading strategies. And they also bought into private equity funds, which make direct investments into private companies or buy them out.

It’s too soon to know the depth of the damage from these investments. Annual results won’t be in until January for pensions and July for endowments.

But there are already indications that the value of these alternative investments is falling, including disclosures from some publicly traded private-equity firms that some of their holdings are worth less than they were just months ago. The weakness in those holdings has delivered another punch to colleges and pension funds, which are bracing for the worst.

Harvard University announced this week that its endowment tumbled since July 1 by about $8 billion, or 22 percent, to about $29 billion, and said that “sobering figure” doesn’t fully capture its losses because it doesn’t reflect declines in its private-equity and real estate investments. It forecast total losses for its fiscal year ending in June 2009 could be as much as 30 percent, its worst performance on record.