One of nine children born to Mr. and Mrs. Samuel Baldwin, Phil received his start in the sport of boxing as a member of the Muskegon Catholic Youth Organization. At the age of 16, he won the state of Michigan’s novice title at 135 pounds.

In February of 1960, Baldwin earned the Michigan Golden Gloves open crown at 147 pounds at the Civic Auditorium in Grand Rapids. It was the beginning of an event-filled year, as Baldwin rapidly climbed to the top of the amateur boxing world. In April 1960, Baldwin, boxing as a member of the Chicago Tribune’s Golden Gloves team, helped the West defeat the East in a boxing exhibition before a crowd of 15,000 at Madison Square Garden. One week later, the Muskegon Heights High School senior won the national AAU 147-pound championship at the University of Toledo with a decision over Roy McMillan of Toledo. The win marked Baldwin’s 27th victory in 29 amateur bouts. The victory also marked the first AAU crown in 31 years for a West Michigan boxer.

One month later, the southpaw joined 78 amateur hopefuls in San Francisco to battle for a spot on the United States Olympic team. Baldwin emerged victorious and was added to the 21-man preliminary roster for the 1960 Rome Olympics. In July, he was sent to Fort Dix, New Jersey, for intensive training with Coach Julius Menendez. In early August, it was announced that Baldwin was named to the 10-man Olympic squad. On August 17, the team boarded a plane for Rome. However, Baldwin was forced to stay behind in New York with pneumonia. He rejoined the team a week later for the Olympics.

Baldwin received a bye in his first bout. In the second round the 5-7, 147-pound 18-year-old TKO’ed Rene Grun of Luxembourg in the third minute of the first round. Baldwin defeated Max Meier of Switzerland two days later to advance to the quarterfinal round. In his next match, he dropped a split decision to Jim Lloyd of Great Britain. During his Olympic adventure, he became acquainted with fellow US teammate Cassius Clay (Muhammad Ali). The two remained close friends throughout their lives.