DigiRL is an innovative online reinforcement learning algorithm designed for training intelligent agents capable of controlling devices in outdoor environments. It employs an autonomous value learning model (VLM) to address open-ended, real-world Android tasks. Key advantages of DigiRL include its ability to utilize existing sub-optimal offline datasets and encourage agents to learn from their own trials and errors through offline-to-online reinforcement learning. The model utilizes instruction-level value functions to implicitly construct automatic curricula, prioritizing tasks most valuable to the agent, and employs step-level value functions to select beneficial actions contributing to the goal within trajectories.