U.S Rep. Scott Garrett was the only congressman from New Jersey to vote against banning Confederate flags in veterans cemeteries.

The bills, approved with a 265-159 vote, would block descendants and others seeking to commemorate veterans of the Confederate States of America from flying the Confederate Battle Flag over mass graves on the two days a year that flag displays are permitted.

California Democrat Jared Huffman drafted the prohibition, saying the flag represents "racism, slavery and division."

Garrett, one of the most conservative congressmen from New Jersey who represents the state's 5th congressional district, said in a statement  that while he strongly disagrees with what the Confederate flag represents, “we must protect the First Amendment — even when it’s unpopular."

Peter Vallorosi, one of Garrett's opponents in the upcoming June 7 primary, agrees with Garrett's vote. "It's a part of history and not something we should be running away from,"  the Newton Republican said and blames the media for making the rebel flag a symbol of hate. "When you watched the 'Dukes of Hazzard' did you think they were full of hate?" asked Vallorisi referring to the CBS program featuring a car nicknamed the General Lee with a Confederate flag painted on it.

"I'm from the north so it means nothing to me but to people from the south it means a lot to them," Vallorisi said. He believes that flying an ISIS or Nazi come under the First Amendment. "Would I like it? Absolutely not because to me ISIS represents hate. But if someone wants to fly it who am I to tell them 'no, you can't fly it?'"

Democrat Josh Gottheimer in an email said, “Garrett and the Confederate Flag are both relics of a different era – one of widespread bigotry, discrimination, and hatred that most of the country has moved on from. As he has demonstrated time and again in his 25 years in politics, Garrett is clearly stuck in the past and should be ashamed of his votes.”

Republican Michael Cino did not yet return a call seeking comment.

Huffman's amendment is mostly symbolic and applies only to instances in which Confederate flags are flown on flagpoles over mass graves. The amendment would not ban the display of small Confederate flags placed at individual graves. Such displays are generally permitted on Memorial Day and Confederate Memorial Day in the states that observe it.

The Associated Press contributed to this report