The Congress of Racial Equality (C.O.R.E) in early 1961 organized volunteers willing to ride buses into the Deep South where bus stations were still racially segregated despite ICC regulations and developing case law in the federal courts. John Siegenthaler, then aide to Robert Kennedy, recalls how the Kennedy administration did not approve of the freedom rides, as they came to be called. Attorney General Robert Kennedy feared violence and urged CORE to prosecute the issues in the courts.

Fred Gray, one of the few African American lawyers in Alabama in the 1950s, played a leading role in most of the important civil rights cases of the era. Like Robert Kennedy, he preferred keeping the issues in the courts. Though he knew the depth of white opposition to change, he did not expect the violence that came.

The violence that greeted the Freedom Riders in Anniston, Alabama, where their bus was burned by a mob, and in Birmingham where police commissioner Bull Connor gave local Ku Klux Klansmen fifteen minutes to beat up the riders on a second bus, shocked the state and according to John Patterson, caught state officials completely off guard. Unaware of the second bus leased from Trailways, Patterson and his public safety commissioner had put an undercover state trooper named E.L. Cowling on the Freedom Riders' Greyhound bus when it left Atlanta. Cowling used his pistol to force back the crowd surrounding the burning bus in Anniston and saved the lives of its passengers. According to John Siegenthaler, John Patterson believed the Kennedys were promoting the freedom rides, creating a huge political problem for John Patterson. Albert Brewer, then a member of the Alabama legislature, remembers Patterson's determination to steer clear of the phone calls he began to receive from Washington.

A third group of Freedom Riders decided to continue the journey, from Birmingham to Montgomery, Jackson, Mississippi and New Orleans. John Siegenthaler came to Patterson's office as a representative of the attorney general and the president to ask for protection. Though Patterson was angry at the Kennedys for what he saw as their betrayal, he agreed to protect the Freedom Riders on the advice of public safety director Floyd Mann. John Siegenthaler describes calling Robert Kennedy from Patterson's office and the gulf that had developed between the two former allies.

In Montgomery, Police Commissioner Sullivan went back on his promise to protect the Freedom Riders at the Greyhound bus station and a huge crowd descended to beat and terrorize. Floyd Mann almost single-handedly intervened to quell the marauding crowd until a contingent of state troopers arrived to do the job the city police had conspired not to do. The drama continued later when Martin Luther King arrived in Montgomery and a rally was held at a black church downtown. John Patterson recalls what happened next.

In the end, John Patterson was forced to use the same measure Gordon Persons had used against Phenix City in 1954, declaring martial law and assigning the National Guard to take over policing duties from the Montgomery Police Department. The mob was dispersed but the crowd in the church was advised to remain overnight rather than risk returning to their homes in the dark. Fred Gray was inside the church that night.