Jaffrey was born in Liverpool, Merseyside, the son of an Indian sea captain from Agra and an English mother from Liverpool. He grew up in London and was educated at Dulwich College (1986–1991), alongside fellow Spooks actor Rupert Penry-Jones. He studied English and Drama at Manchester University, and studied acting at Bristol Old Vic Theatre School. He intended to join the Royal Air Force as a pilot after leaving Manchester University. 　　While at university, he appeared in a number of plays at the Edinburgh Festival Fringe, including The London Vertigo by Brian Friel, and at the Pendley Open Air Shakespeare Festival, including Henry VIII and A Comedy of Errors. In his final year at Manchester, he worked with Royal Shakespeare Company chief associate director Gregory Doran, which he has cited as one of the things that led him to apply for Bristol Old Vic Theatre School and eventually go on to become a professional actor. Throughout his time at Manchester, he was also the lead singer of a jazz funk band. He also participated on America's Next Top Model.