A manager should stay as far away as possible from his players. I don't know if I said ten words to Frank Robinson while he played for me.

So whenever the Orioles were on the primary ABC game Weaver worked the backup game. He joined the Orioles in 1957 as skipper of their Fitzgerald club in the Georgia-Florida League.

After playing in minor league baseball he retired without playing in Major League Baseball (MLB). He became a minor league manager and then managed in MLB for 17 years with the Baltimore Orioles (1968–1982; 1985–1986).