Configuring group_size > 128

[hzfantasy]

To meet the requirements of the hardware, I need to quantize with group_size >= 512. Is it possible to do this in GPTQModel?

[jeyapradhaj]

Group size is configurable in 'QuantizeConfig'. For example,
quant_config = QuantizeConfig(bits=4, group_size=256)

[Qubitium]

@hzfantasy Can you tell me what exactly the hardware requirement that will need group_size 512? Quality of post-quant recovery may be very bad with > 128 gpsize.