The song has Gaye and Wells going back and forth with Gaye apologizing to Wells for leaving her behind for someone else begging Wells to give him one more chance.
At first, Wells says taking Gaye back would "hurt (her) pride" and that she cannot let him take her out for the night but after Gaye pleads for Wells to come back to him, Wells relents but still passes on him taking her out for the night. The song begins and ends with the two harmonizing together.
Chart performance
"What's the Matter with You Baby" peaked at number nineteen on the U.S. Pop Singles chart when released.[3][2] On Cash Box "Top 50 in R&B Locations", the song went to number two.[4]