Born at 19 Llwyn Onn Street in Trecenydd, Caerphilly, Glamorgan, Cooper was delivered by the woman who owned the house in which the family were lodging. His parents were Thomas H. Cooper, a former recruiting sergeant in the British Army turned coal miner, and Catherine Gertrude (née Wright), his English wife from Crediton in Devon.

To escape from the heavily polluted air of Caerphilly, his father accepted the offer of a new job and the family moved to Exeter, Devon, when Cooper was three. It was in Exeter that he acquired the West Country accent that became part of his act. The family lived in the backs of Haven Banks, where he attended Radcliffe College. He also helped his parents run their ice cream van, which attended fairs at weekends. When he was eight an aunt bought him a magic set and he spent hours perfecting the tricks.His brother David (born 1930) opened a magic shop in the 1960s in Slough High Street called D. & Z. Cooper's Magic Shop.