Poor was born in 1747 into slavery on a farm in Andover, Massachusetts owned by John Poor and his son John Poor Jr. He bought his freedom on July 10, 1769 from John Poor Jr. for £27, a year's salary for an average working man at the time.