Lytton began on drums at age 16. He played jazz in London in the late 1960s while taking lessons on the tabla from P.R. Desai. In 1969 he began experimenting with free improvisational music, working in a duo with saxophonist Evan Parker. After adding bassist Barry Guy, the ensemble became the Evan Parker Trio. He and Parker continued to work together into the 2000s; more recent releases include trio releases with Marilyn Crispell in 1996 (Natives and Aliens) and 1999 (After Appleby).
A founding member of the London Musicians Collective, Lytton worked extensively on the London free improvisation scene in the 1970s, and aided Paul Lovens in the foundation of the Aachen Musicians' Cooperative in 1976.
Lytton has toured North America and Japan both solo and with improvisational ensembles. In 1999, he toured with Ken Vandermark and Kent Kessler, and recorded with Vandermark on English Suites. Lytton also collaborated with Jeffrey Morgan (alto & tenor saxophone), with whom he recorded the CD Terra Incognita Live in Cologne, Germany.