Born on May 1, 1780, in Brimfield, Massachusetts, Brown was the daughter of Lois Haynes Moffatt and Dr. Joseph Moffatt. She married the Reverend Clark Brown (1771–1817) on December 1, 1799. The pair raised three boys and one girl together until the reverend died in 1817. He was a Congregational minister, and later became an Episcopalian minister. The oldest son was Orus, followed by Manthano, John Mattacks, and finally a daughter, Pherne. John died at age six. Prior to John and Clark's deaths, the family lived in various locales, including Charles County, Maryland, where Clark is buried. Later the family moved to Missouri to follow Brown's brother-in-law Captain John Brown, a sea captain. She supported the family by teaching.