In his oral history about his time at the Solicitor General's office, Elman recounted his involvement in Brown v. Board of Education. He explained how the Solicitor General's brief used the phrase "with all deliberate speed":
It's because we were the first to suggest, and all the parties and amicus on both sides rejected it after the government proposed it, that if the Court should hold that racial segregation in public schools is unconstitutional, it should give the district courts a reasonable period of time to work out the details and timing of the implementation of the decision. In other words, "with all deliberate speed".[3]
Elman wanted a "middle ground" between reaffirming the "separate but equal" doctrine of Plessy v. Ferguson and requiring immediate integration of all public schools, even though that meant separating the constitutional principle from the judicial remedy. He believed that otherwise the Supreme Court could not have decided the case unanimously and the American public would not have tolerated the decision.[3]
Since Elman's tenure, only two other political independents have served on the body: Mary Azcuenaga, who served from 1984 to 1998, and Pamela Jones Harbour, who served from 2003 to 2009.[6]
Elman admitted in his oral history that he and Justice Frankfurter conferred privately about the intended remedy in the case, which technically constituted a breach of judicial ethics. He was publicly criticized for this in 1987 by Time and The New York Times. Elman defended both himself and Justice Frankfurter by stating that these discussions took place before the United States became a party to the case, and even then, the United States was not an adversary party but rather an amicus curiae.[8]
