What could cause MAX17552 to switch the LX pin only at 8% duty cycle when the FB pin reads 100mV?

LX on-time appears to be 0.5uns, the off-time is 6.5 us

That would imply an output of 1.8V. Are you sure it isn't also doing some pulse skipping?

From the datasheet, it looks as if your problem may relate to having selected the mode as PFM (MODE pin open) with a low value inductor.

All their typical application examples with coils less than 150uH are with PWM operation. The only ones using the PFM mode (the ones marked 'high efficiency') have inductors higher than that.

Depending what your board layout is like, you may be able to test that by grounding the MODE pin (before you apply power - it says not to change mode on the fly) to set PWM operation and see if it springs into life and gives you the 6.1V.

I think you're right about the reason. It sounds (from a quick read, so don't rely on this) as if in PFM mode there's a current limit and with a small inductor it hits it too early and never gets the output up to the operating voltage. The trade-off that they are giving you here with the choice of modes is that if you want the higher efficiency, you need a larger inductor and consequentially the footprint is larger.

It doesn't seem to pulse skip at all. I wish I could ground the MODE pin but it is too small for hand soldering. I didn't expect any trouble when I designed the circuit.

I replaced the inductor with 220uH. Sadly the behavior hasn't changed which is funny because now the circuit is basically the same as Maxim's reference design. It must be a small stupid mistake but I can't find it. This is what the design looks like:

https://ibb.co/ich62o

https://ibb.co/ghjx8T

https://ibb.co/ciJPoT

It doesn't seem to pulse skip at all. I wish I could ground the MODE pin but it is too small for hand soldering. I didn't expect any trouble when I designed the circuit.

I replaced the inductor with 220uH. Sadly the behavior hasn't changed which is funny because now the circuit is basically the same as Maxim's reference design. It must be a small stupid mistake but I can't find it. This is what the design looks like:

https://ibb.co/ich62o

https://ibb.co/ghjx8T

https://ibb.co/ciJPoT

Has the voltage at the FB pin increased or is it still down at 100mV?

The first of those links doesn't work for me (presumably it's the schematic).

It looks to me like you need at least 180-190uH to be under the current limit (with Vin of 25V) by the time the minimum period is up, so 220uH should just about clear it (though given a fairly wide tolerance on the coil and particularly if the input voltage could go higher than 25V you might want to review that).

Have you looked at their dev kit for the part. They use a 220uH part there.

https://datasheets.maximintegrated.com/en/ds/MAX17552ATBEVKIT.pdf

It might be worthwhile your looking at their board layout.