Tens of thousands of homeowners in New Jersey have been pushed to the point of no return, and their days of owning a home are gone for good, according to a new study from the National Association of Realtors.

Tim Boyle, Getty Images

Of the 120,000 property owners in New Jersey who lost their home to foreclosure or short sale between 2006 and 2014, the report said, just 38,000 have returned or will return to the real estate market by the year 2023.

It's a similar picture nationwide. Nearly 9.3 million borrowers were affected during the same timeframe, but only about 3 million are expected to return.

Ken Fears, NAR's director of Housing Finance and Regional Economics, said there are plenty of factors keeping former owners away from the homebuying scene.

For starters, about a quarter of the affected owners were only investors and may simply not be interested in returning to the real estate business.

However, for those looking to bounce back, they're dealing with a potentially major hit to their credit score after losing a home.

"It could be as large as 200 points for a prime borrower and take as long as 10 years to recover," Fears said.

In some cases, the very event that led to delinquency - loss of income or a family emergency - may still be impacting the potential borrower.

In other instances, Fears said, former owners may just no longer have the desire to own a home after the trouble they've already experienced. Instead, they will rent or move in with family.

In any event, the report said the number of return buyers coming to the market over the next decade or so will be a welcome addition to the market of first-time and trade-up buyers.