Stafford was raised in Winter Haven, Florida. In high school, he played in a band along with friends Bobby Braddock, Kent LaVoie (also known as Lobo) and Gram Parsons (of the Byrds and The Flying Burrito Brothers).