Rilling was born into a musical family. He received his early training at the Protestant Seminaries in Württemberg. From 1952 to 1955 he studied organ, composition, and choral conducting at the Stuttgart College of Music. He completed his studies with Fernando Germani in Rome and at the Accademia Musicale Chigiana in Siena.
While still a student in 1954, he founded his first choir, the Gächinger Kantorei. Starting in 1957, he was organist and choirmaster at the Stuttgart Gedächtniskirche, conducting the choir Figuralchor der Gedächtniskirche Stuttgart. From 1963 to 1966, he taught organ and choral at the Spandauer Kirchenmusikschule, conducting the Spandauer Kantorei (Spandau chorale).
He is well known for his performances of the music of Johann Sebastian Bach and his contemporaries. Rilling is the first person to have twice prepared and recorded (on modern instruments) the complete choral works of J. S. Bach, a monumental task involving well over 1,000 pieces of music - spanning 170 compact discs. He has also recorded many romantic and classical choral and orchestral works, including the works of Johannes Brahms. In 1988, Rilling conducted the world premiere of the Messa per Rossini that he also conducted at the Rheingau Musik Festival in 2001, where he has traditionally conducted the final concert.[2]
Rilling co-founded the Oregon Bach Festival in 1970, and served as its artistic director until 2013.[3] He also co-founded and led the Internationale Bachakademie Stuttgart since 1981. In 2001, Rilling created the Festival Ensemble to be part of the European Music Festival Stuttgart ("Musikfest Stuttgart").[2] Rilling became the Festival Conductor and lecturer at the Toronto Bach Festival in 2004.