Popular GitHub Repositories for Optimizer

APOLLO: SGD-like Memory, AdamW-level Performance

torch-optimizer -- collection of optimizers for Pytorch

AdalFlow: The library to build & auto-optimize LLM applications.

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch

Must-read Papers on Large Language Model (LLM) as Optimizers and Automatic Optimization for Prompting LLMs.

Explore the energy-efficient dataflow scheduling for neural networks.

LAMB Optimizer for Large Batch Training (TensorFlow version)

The official implementation of "CAME: Confidence-guided Adaptive Memory Optimization"

Distributed K-FAC Preconditioner for PyTorch

IVON optimizer for neural networks based on variational learning.

? Pytorch and Jax code for the Madam optimiser.

Add gradient noise to any Keras optimizer

PyTorch Impl. of Prediction Optimizer (to stabilize GAN training)

This project implements optimizers for TensorFlow and Keras, which can be used in the same way as Keras optimizers. Machine learning, Deep learning

Review materials for the TWiML Study Group. Contains annotated versions of the original Jupyter noteboooks (look for names like *_jcat.ipynb ), slide decks from weekly Zoom meetups, etc.

[EMNLP22] Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models