do not overwrite storage dtypes

[MdSadiqMd]

Description

When TransformerBridge.boot_transformers loads a quantized HuggingFace model, its post-load dtype normalization via cast_floating_params_to_dtype casts every floating-point parameter to the requested dtype. This includes floating-point parameters owned by the quantizer, such as FP8 scale tensors (e.g., float8_e8m0fnu).

On MXFP4 checkpoints like deepseek-ai/DeepSeek-V4-Flash-0731, loading with dtype=torch.bfloat16 changes the expert scales from torch.float8_e8m0fnu to torch.bfloat16. The packed int8 expert weights are left unchanged, so the weight/scale pair no longer describes the same quantization format, causing forward pass failures.

Fix implemented at two levels:

1. Call-site skip: Skip cast_floating_params_to_dtype entirely when the model has an active quantization_config
2. Function guard: Skip one-byte floats (FP8 dtypes) in cast_floating_params_to_dtype as a defensive guard

Fixes #1713

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

[jlarson4]

Hey @MdSadiqMd, thanks for the fix! A couple notes on the test coverage

[jlarson4]

@MdSadiqMd I missed this the first time around, could you please rebase this to dev?

[MdSadiqMd]

sure

[jlarson4]

Hi @MdSadiqMd, just pushed a CI update to your branch that should help stabilize Full Code Coverage, there was a flaw in dev that was causing failures. Everything looks good, once this run passes I'll merge, thanks for your patience!