Popular GitHub Repositories for Optimizer

torch-optimizer -- collection of optimizers for Pytorch

AdalFlow: The library to build & auto-optimize LLM applications.

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch

Must-read Papers on Large Language Model (LLM) as Optimizers and Automatic Optimization for Prompting LLMs.

Explore the energy-efficient dataflow scheduling for neural networks.

APOLLO: SGD-like Memory, AdamW-level Performance

LAMB Optimizer for Large Batch Training (TensorFlow version)

The official implementation of "CAME: Confidence-guided Adaptive Memory Optimization"

Distributed K-FAC Preconditioner for PyTorch

IVON optimizer for neural networks based on variational learning.

? Pytorch and Jax code for the Madam optimiser.

Add gradient noise to any Keras optimizer

PyTorch Impl. of Prediction Optimizer (to stabilize GAN training)

Review materials for the TWiML Study Group. Contains annotated versions of the original Jupyter noteboooks (look for names like *_jcat.ipynb ), slide decks from weekly Zoom meetups, etc.

[EMNLP22] Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models

Jax implementation of the AdaHessian optimizer