LLM Augmented LLMs achieve new capabilities by combining existing base models with more specific models. CALM (Composition to Augment Language Models) introduces cross-attention between models to combine their representations and achieve new capabilities. Its key advantages include: (i) Scaling up LLMs on new tasks by "reusing" existing LLMs with a small amount of additional parameters and data; (ii) Preserving the weights of existing models, therefore retaining their existing capabilities; (iii) Applicability to different domains and settings. Experiments show that augmenting PaLM2-S with smaller models trained on low-resource languages resulted in absolute improvements of up to 13% on tasks such as English translation and arithmetic reasoning in low-resource languages. Similarly, when PaLM2-S was augmented with code-specific models, we saw up to 40% improvement in code generation and interpretation tasks compared to the base model, comparable to fully fine-tuned counterparts.