How to use MAISI's VAE encoder to encode a CT?

[paproaskwar]

Really appreciate your wonderful job!!!

Here is my code when I use AutoencoderKlMaisi to encode a CT:

The code looks good, but when I save the latent(z_mu) as a .nii file and displayed in 3D Slicer, the latent looks pretty noisy.
is that normal?

autoencoder = define_instance(args, "autoencoder_def").to(device)
checkpoint_autoencoder = torch.load(args.trained_autoencoder_path, weights_only=True)
autoencoder.load_state_dict(checkpoint_autoencoder)
autoencoder.eval()

with torch.no_grad(), torch.cuda.amp.autocast():
     z_mu, z_sigma = autoencoder.encode(images)

[Can-Zhao]

Yes

[sara-create]

By the way I have a question , the embeddings of the labels are learned directly with the training script of the ControlNet ? Or there is a script for its training that I don't see.

[Can-Zhao]

There is no learned embeddings of the labels used for ControlNet. You can find how labels are processed in controlnet training script

tutorials/generation/maisi/scripts/train_controlnet.py

Line 196 in 8b90a16

controlnet_cond = binarize_labels(labels.as_tensor().to(torch.uint8)).float()

[sara-create]

Thank you for your response. If I am not mistaken the labels are also sent into the latent space . In the ControlNet definition ( monai->networks->nets->controlnet.py ) there is a block called ( ControlNetConditioningEmbedding : Network to encode the conditioning into a latent space ) . My question is how is this block trained . Is it trained separately from the ControlNet or optimized with it. In the forward of the controlnet we call it through : controlnet_cond = self.controlnet_cond_embedding(controlnet_cond).

[Can-Zhao]

It's inside the controlnet, so it was trained together with the controlnet, you won't need to worry about it