Prosecutors have reportedly said that the 12-year-old left a backpack containing bomb at the market but the device did not go off because the detonator apparently failed.
"I can confirm that we have started an investigation based on the discovery of a nail bomb in Ludwigshafen," federal prosecutor Stefan Biehl said.
According to reports, the boy has made a number of attempts at detonating the device on different occasions. The explosive device is said to have contained materials from fireworks and sparklers and was not capable of exploding.
The suspect is said to have been born in Ludwigshafen to Iraqi parents in 2004. The boy is too young to face a German court. He was reportedly held and placed into a juvenile care facility after the alleged attempt to blow up the market.
He is thought to have been radicalised recently by Islamic State.