Their bodies were then hung from electricity poles in several districts, the office of the UN Human Rights Commissioner said, citing sources.
A man was also reportedly shot dead in public in central Mosul for ignoring an IS ban on using mobile phones.
Iraqi security forces are continuing their push to take Mosul back from IS.
The killings of the civilians appeared to have been carried out on the orders of self-appointed "courts", according to the UN report.
The 40 civilians were accused of "treason and collaboration" and dressed in orange clothes marked in red with the words "traitors and agents of the ISF" (Iraqi Security Forces).
The UN says 20 civilians were also shot dead on Wednesday evening at the Ghabat military base in northern Mosul, supposedly for leaking information.
The UN also expressed concern at IS's deployment of teenagers and young boys. Children are apparently seen in an IS video issued on Wednesday shooting dead four people for spying.
IS also announced on 6 November that it had beheaded seven militants for deserting the battlefield in the Kokjali district of eastern Mosul, the UN says.
Among the sources cited for the UN's information was a man who played dead during a mass killing by IS fighters.