Baker grew up in a racing family. His father, George, was a trainer in New York and gave Baker his start at the racetrack. In the 1970s, Baker eventually branched out from his father and went to work for trainer David Whiteley.