He was descended from a family that probably originated from Gosel in the Egerland and emigrated to Gürth (now in Bad Brambach) before 1630. He was the son of Karl Gustav von Goßler (1810–1885), the Kanzler (in this case, provincial justice minister) of the province East Prussia, legal advisor to the crown and President of the Oberlandesgericht in Königsberg, and Sophie von Mühler (1816–1877), who was the daughter of the Prussian Justice minister Heinrich Gottlob von Mühler (1780–1857). This marriage produced four children: the Prussian Minister of State and Oberpräsident of the province of West Prussia, Gustav (1838–1902), the Prussian Minister of War and General of the Infantry, Heinrich (1841–1927), the Prussian General of the Infantry, Konrad (1848–1933), and the Prussian Lieutenant General, Wilhelm von Goßler (1850–1928).
On 31 August 1872, Gossler married Emma von Sperber (born 14 November 1848; died 3 October 1914), the daughter of the landowner Eugen von Sperber, and of Emilie Donalitius.
He also fought in the Franco-Prussian War in 1870–1871, with distinction. After the war, he was assigned to the Ministry of War as a captain until 1875. He then commanded a company in the 2nd Grenadier Regiment until 1878, when he was once again moved to the War Ministry, to the Department for Army Affairs. Here he was later made a member of the higher examination commission. In 1885 Goßler became a department head in the War Ministry.
After being appointed a colonel in 1888, commander of the 3rd Guard Regiment in 1889 and a Major General and brigade commander in 1891, he was made Director of the General War Department in the War Ministry in October 1891 and simultaneously, representing the War Minister, a member of the Defence Commission. He was also the deputy plenipotentiary in the Bundesrat, chairman of the Reichsrayonkommission and a member of a disciplinary commission.
Goßler was a Lieutenant General after 1895 and commanded the 25th Division in Darmstadt. On 14 August 1896 he was appointed Prussian Minister of War, and in September was also made the Prussian plenipotentiary in the Bundesrat. On 15 August 1903 he retired, having been made a General of the Infantry in 1899.
