Octopus is an environment-based visual language programming tool that can efficiently parse agents' visual and textual task goals, formulate complex action sequences, and generate executable code. Octopus is designed to enable agents to handle a wide range of tasks, from everyday chores in simulators to complex interactions in complex video games. Octopus is trained in our experimental environment OctoVerse by leveraging GPT-4 to control the exploration agent, generating training data in the form of action blueprints and corresponding executable code. We also collect feedback to enable a reinforcement learning with environment feedback (RLEF) training paradigm. Through a series of experiments, we elucidate Octopus' capabilities and present compelling results, demonstrating that the proposed RLEF approach effectively enhances agent decision-making. By open-sourcing our model architecture, simulator, and dataset, we aim to foster innovation and promote collaborative applications within the broader AI community.