Fisher was born Mary Frances Kennedy in Albion, Michigan. Within two years of her birth, her father, Rex Kennedy, moved the family to Whittier, California to pursue a career in journalism. While Whittier was a Quaker community, Mary Frances was brought up within the Episcopal Church. In addition to being an author, Fisher was an amateur sculptor working mostly in the realm of wood carving. Fisher later bore two daughters. Anne, whose father Fisher refused to name, was born in 1943. Kennedy was born during Fisher's short-lived marriage to Donald Friede, which lasted from 1945 to 1951.