The members of Yad L'achim, an organization that describes its aim as rescuing "Jewish souls", snatched the girl during a night-time raid on a house in Tulkarm, in the north of the Israeli-occupied West Bank, Israeli newspaper Ma'ariv said.
"We turned to the police and other services, but none of them was able to help us. So we used one of our undercover units," Mati Dushenski, a spokesman for Yad L'achim, told Reuters, confirming Wednesday's operation.
"We have done many similar operations in the past," he said.
Ma'ariv said the Yad L'achim unit, armed with assault rifles, had forced the father and daughter into a van and driven off. The father was released in an olive grove outside Tulkarm and the child was taken to a secret location to meet her mother.
A Palestinian newspaper reported that the Jewish gunmen had long beards and had worn Arab headdresses in an apparent effort to disguise themselves as locals.
The girl's father was quoted in Ma'ariv as saying that he and the girl's mother met seven years ago, fell in love and married. The mother converted to Islam and moved to Tulkarm where he had another wife and five children.
Together they had a daughter who was raised as a Muslim.
The mother told Yad L'achim that her husband had pretended to be a French-American Jew, the organization said. She said she hadn't known that her husband was already married and that after their wedding, she wasn't allowed to leave the house.
Yad L'achim said the woman had left her husband and taken her daughter to live in the Israeli city of Ashdod in June this year, but that her husband had subsequently come and taken the daughter back with him to Tulkarm.
"This is one of many cases that we had to take care of with our undercover units that operate in the territories," Rabbi Shalom Dov Lifshitz, the head of Yad L'achim, told Ma'ariv.
The mother was said to be preparing a legal suit to maintain custody of the child.