Born a slave in Louisiana in 1825, Cailloux lived his entire life in and around New Orleans. He was owned by members of the Duvernay family until 1846, when his petition for manumission, which was supported by his owner, was granted by an all-white police jury in the city of New Orleans. In 1847, Cailloux married Felicie Coulon, whose mother, Feliciana, a mulatto woman, had participated in the local placage system as the common law wife of a white plantation owner, Valentin Encalada, for several years. Though Felicie was not Encalada’s daughter, she remained his property until her mother bought her freedom from Encalada in 1842. Cailloux and Coulon had four children, three of whom survived to adulthood.