Kaye began his career as an apprentice with Rotherham United, where he turned professional in November 1977. However, he failed to make a league appearance for the Millers and played non-league football for Mexborough Town before returning to The Football League when he joined Chester City in March 1985. He played in Chester's final seven games of the 1984–1985 season and then played three league games and a League Cup tie early the following season. Kaye lost his place to John Butcher and Chester then signed another goalkeeper in Ray Cashley.