The loss of the monopoly and the depression ended the mills' expansion. Permanent lay-offs and short-term strikes occurred at Whittier Mills, which had begun to lose money on its Georgia operation. In 1934, the year of the General Textile Strike throughout the country in which so many southern millworkers participated, "Boss" Whittier left the operation. J. J. Scott of Scottdale Mills near Decatur became general manager. Scott divided his time between his own mills and their competition at Whittier Mills in the town of Chattahoochee. Scott put the mill back into the black and in 1936 placed Hanford Sams in the manager's position of both his mills. Sams eventually became vice president of the Whittier Mills board of directors under president Scott, who had taken over that position from Sid Whittier in 1936.
During the 1940s, the new management and wartime contracts brought renewed economic stability to the mill and the village. The employees' newsletter, Whittier Mills & Silver Lake News, reported on the prowess of the company baseball team with detailed accounts of winning seasons, playing against Clarkdale Thread Mill and Celanese (of Rome), and in 1948, the construction of new bleachers for the fans. That year the Osborne family were the big stars: the father, "Tiny," was once a major league player for Chicago and Brooklyn in the National League; "Jeter," one of his seven children, played in the Southern League for the New Orleans Pelicans; and another son, "Bottles" played for Rochester (AAA ball) and Birmingham in the Southern Association. In 1949, the trolley from Atlanta, filled with free riders for the occasion, made its last run to Chattahoochee.