Imagine having your home wrecked by a natural disaster, receiving recovery funds from the federal government, spending those funds, and then having the government ask that the money be returned. That's the harsh reality for Superstorm Sandy survivors in New Jersey and four other states.

Spencer Platt, Getty Images

The Federal Emergency Management Agency (FEMA) is in the process of recouping more than $23 million from recipients "who are not eligible" in Connecticut, Maryland, New Jersey, New York and Rhode Island. As of the end of October, according to FEMA, 2 percent of survivors who obtained assistance have also received a recoupment letter.

A statement from FEMA to Townsquare Media explained that some post-disaster payments are mistakenly made due to "fraud, human or accounting errors, or other reasons."

Bridget Holmes of the Ocean County Long Term Recovery Group, which provides case management to individuals impacted by the storm, said there has been an increase recently in the number of recoupment letters among the group's clients.

However, in almost every case, the money has already been spent.

"There are very few people who are still holding on to FEMA money," Holmes said. "In lots of cases, it was rental assistance money, and so it was money that went directly to a landlord while an individual or household was displaced from their primary home."

Holmes urged affected storm survivors to go through the proper appeal process.

"There is reason to believe that maybe there was a clerical or human error this time around, and not the first time around," she said.

FEMA said people have the option to establish a payment plan or request a "compromise based on inability to pay."

According to Holmes, each recipient of FEMA assistance signs an agreement that requires the money be used in a certain way, and if the recipient goes a different route, FEMA has the right to come back for a reimbursement.