To elaborate on how frequency modulation "gives the triangle wave a grittier sound", FDS's FM and Yamaha's phase modulation produce sidebands, or extra frequencies, at (harmonic * carrier frequency ± n * modulator frequency). The intensity of each sideband is proportional to FFT(carrier)[harmonic] * f(n) (for some odd f involving Bessel functions and FFT(modulator), which is zero for large |n|, and the number of nonzero n increases as the modulation index increases). In this case, the modulator frequency isn't a small rational factor of the carrier frequency, creating gritty inharmonic sidebands.

As an example of FDS FM, you can look at this video (https://www.youtube.com/watch?v=Tgu1mCBU2vA) describing FM emulation errors in older versions of FamiTracker, and how some chiptune songs are incompatible with the more accurate emulation core I imported into Dn-FamiTracker.

FDS FM is the chip's defining characteristic in modern chiptune, though it was sadly barely explored by official games in the console's original lifespan (aside from being used as vibrato). The only game I currently know which used FM at audible frequencies was Bio Miracle Bokutte Upa (https://www.youtube.com/watch?v=_alQrPMNBT0), which also required cycle-accurate console emulation and would otherwise degrade into inharmonic chaos (like the "FM emulation errors" video above). Unlike regular FM on Yamaha sound chips, the FDS allows you to use complex carrier wavetables. Combined with how the modulator is naturally slightly detuned from the carrier (due to hardware rounding errors), this results in growling evolving sounds as the waveform stretches and squeezes in the time domain, and harmonics and sidebands beat in and out of phase in the frequency domain.

Strangely, the FDS's modulator doesn't draw from an array of samples like the carrier waveform, but instead an array of 3-bit delta values (https://github.com/NovaSquirrel/Mesen-X/blob/master/Core/Mod...) representing adding [+0, +1, +2, +4, reset=0, -4, -2, -1] (modulo [-64..64)) to the current modulator amplitude. This means you have to design modulator tables carefully, or else the carrier frequency will have a net offset (as above) or even drift endlessly until it wraps around. Interestingly, FamiTracker shows the mod table as only having 32 elements (compared to the wavetable's 64 elements), and the nesdev wiki (https://wiki.nesdev.org/w/index.php/FDS_audio#Mod_table_writ...) describes the mod table as having 32 elements. This conflicts with how Mesen (https://github.com/NovaSquirrel/Mesen-X/blob/master/Core/Mod...), cxNES (https://github.com/perilsensitive/cxnes/blob/master/boards/a...), and nsfplay (https://github.com/bbbradsmith/nsfplay/blob/master/xgm/devic...) implement the mod table as a 64-element array where you can only write to 2 adjacent elements at a time. I'm not sure which is accurate, as I don't have hardware on hand to test.

Then on each sample (https://github.com/NovaSquirrel/Mesen-X/blob/master/Core/Mod...), the modulator amplitude is multiplied by the modulation strength (dropping a few lower bits and rounding oddly), clamped between [-64..192) (asymmetrical for some reason), and then used to change the carrier's instantaneous frequency by 0x through 4x (https://github.com/NovaSquirrel/Mesen-X/blob/master/Core/Fds...).