How to set "n_z" parameter of hyperLSTM? Are there any guidelines?

LabML Team, Excellent work!! I am test-running transformers/retro however it seems some files are missing in the repo, getting below error while running 'train.py' ` ModuleNotFoundError 5 from labml import monit, lab, tracker, experiment, logger 6 from labml.logger import Text ----> 7 from labml_...

annotated_deep_learning_paper_implementations/labml_nn/diffusion/ddpm/unet.py Line 188 in 05632f9 attn = attn.softmax(dim=1) According to my understanding of Self-Attention, the softmax...

annotated_deep_learning_paper_implementations/labml_nn/normalization/deep_norm/__init__.py Line 105 in 25ad4d6 self.layer_norm = LayerNorm(normalized_shape, eps=eps, elementwise_affine=elementwise_affine)...

annotated_deep_learning_paper_implementations/labml_nn/transformers/switch/__init__.py Line 139 in 25ad4d6 final_output[indexes_list[i], :] = expert_output[i] here final_output.dtype is...

the code comments shows it should be (seq_len, batch_size, d_model), but in the experiment.py, the input is the output of nn.embedding, so its shape may be (batch_size, seq_len, d_model) ?

Hello, Would it possible to fine-tune RETRO on a suitable downstream task such as text summarisation? how would one go about implementing it. Thank You!

line 98 route_prob = self.softmax(self.switch(x)) line 102 route_prob_max, routes = torch.max(route_prob, dim=-1) As far as I know, the torch.max operation is discontinuous, so that the parameters of self.switch cannot be trained, which means that the experts will be chosen randomly each time, I don...

The given implementation puts all layers into a list blocks , and the condition to set stride to 2 is len(blocks)==1. So The size of image will only modify in the first block, and the image size is always the same in all other blocks. It does not match the comments of the implementation code and the...

Hi, Running the exact code on github for deepmind's retrieval transformer - RETRO, getting the following error: RuntimeError: stack expects each tensor to be equal size, but got [2, 32] at entry 0 and [1, 32] at entry 29 Could you please help me with this? I used the same dataset as in the code.