LongLLaMA is a large language model capable of processing long-form text. It is based on OpenLLaMA and fine-tuned using the Focused Transformer (FoT) method. It can handle text as long as 256k tokens or even more. We provide a smaller 3B base model (not instruction-tuned) and inference code with support for longer context on Hugging Face. Our model weights can be used as a replacement for LLaMA in existing implementations (for short contexts up to 2048 tokens). Additionally, we provide evaluation results and comparisons with the original OpenLLaMA model.