The history of phonetic alphabets doesn't seem to be very well documented before Henry Sweet, 1874 (of Sweet's Anglo-Saxon Reader and "hebban olla vogola"). This particular phonetic alphabet is in Charles de Brosses, Traité de la Formation Méchanique des Langues et des Principes de l'Étymologie, 1765.
De Brosses holds that the only phonetically distinctive part of a language are its consonants; the vowels, he argues, are much more like the arbitrarily selected tones on a continuous scale. He divides up consonants in six groups: those formed with lips, throat, teeth, palate, tongue, and nose. Between these he distinguishes further between soft, rude, and moderate. For the Brosses this essentially captures all the phonetic differences in the world, the rest depending on undistinctive "local colour", though he allows for some finer contrast in the "organic" transcription of sounds.
To increase confusion, he presents two alphabets: one that more or less "pictures" the sounds (with letters that look like teeth, noses, tongues &c.), and one for more facile transcription. In good eighteenth-century fashion, he offers a phonetic transcription of the Paternoster in four different languages.
Now even if this is Eurocentric (yes), not quite as encompassing as IPA (yes), and somewhat internally inconsistent (yes), and the distinction between rude, moyen and doux doesn't quite match with the current voiced/unvoiced distinction, it is on a high level of conceptual sophistication. So I'm curious to find out on what precedents De Brosses drew.
There were phonetic alphabets from the early 17th century onwards, only I haven't yet seen what they look like. The HSK handbook on the History of the Language Sciences is not particularly helpful in this regard. Any nerds out there that know more?
The Traité is on the internet archive - unfortunately, without the tables.