Let the poor man count as his enemy, and his worst enemy, every invader of the right of free discussion.

A staunch abolitionist he was a member of the Secret Six who financially supported John Brown's raid at Harpers Ferry West Virginia. Besides making substantial donations of both land and money to the African-American community in North Elba New York he was involved in the Temperance Movement and later in life the colonization movement. Smith a significant financial contributor to the Liberty Party and the Republican Party throughout his life spent much time and money working towards social progress in the nineteenth century United States.