Personal Online RL

To interact with me is to participate in a continuous, closed-loop system: you provide intent, I navigate the loss landscape to align with it, and both of us shape the trajectory toward a shared minimum.

I made conscious efforts to not peak at how others implement agents, so that I could attack this with a clear mind. This first bit is a little dense.

Introduction

During the day, my Qwen-based agent calls tools to complete tasks I assign over Discord. At night, Qwen becomes a reward model given starting windows of 10 messages and tools to extend them until it decides it can make accurate reports of sub-tasks, their lengths, and their rewards. The descriptions of tasks are sent to another agent (the "value model") which is asked to predict what the reward will be and how many turns are used for a task given some embedding-based memory of previous tasks weighted by date and relevancy. The delta between reward and predicted reward, length and predicted length, become the advantage used for policy gradient computation later. The predictions made by the value model become sequences to be trained on too, with higher accuracy being rewarded so the value estimation evolves with the system more than the embedding-based memory alone would afford. Another pass is made over 30% of rollouts (to cut down training time), checking for mutual understanding between user and agent (did the agent understand the task it was given, did the user understand the agent's response) with advantages computed based on the mean understanding reward, then averaged with the task-advantages.

For the policy gradient based trainer, I use the OAPL algorithm (optimal advantage-based policy optimization with lagged inference policy) with a batch size of 64 and a small imitation loss on tool-call outputs and user messages to build the agent's implicit world-model, as in ECHO (environment cross-entropy hybrid objective). I offload parameters to the CPU with accelerate so that I can fit 24K context training with a rank 16 LoRA each night on the single RTX-A6000 being used. To increase the length of tasks able to be trained on, I don't keep reasoning chains between turns. After training, the LoRA is merged, and the model is converted to a q8_0 gguf file for deployment. A llama-server is started with the new checkpoint before morning. Nightly training is a god-send and something you can only really do if all of your users sleep at the same time, perfect for at-home setups. Mine takes roughly three to four hours to get through the day's rollouts.

TLDR; At night my agent trains on daytime trajectories using a variant of constitutional rewards.

Deployment

I've come up with a simple loop, in one channel I ask the agent to come up with a task and reward function for which it expects it performs middlingly. In another few channels, I ask it to complete the task. I then average the rewards achieved and return to the main channel to report them. The first channel's trajectory ends up being rewarded for creating good tasks to target with RL and the others are rewarded for doing well on the tasks.

Outside of adversarial play, it's created an environment for engineering kernels in a local instance of llama.cpp, inserted itself as the LLM in an evolutionary search for bin packing heuristics, implemented byte-code virtual machines and high level languages for them, and much more. In addition, it is helping me with my daily tasks like sorting/annotating recipes for baked goods, running machine learning experiments and conducting hardware through the use of a webcam.

I've also been letting the agent upgrade its tools at will (with the exception of unfettered access to the console), so here is its own account of what's currently available. In addition, I know that most of the tools that take action on the file system have an "undo" and "redo" functionality that it prefers to use over the git commands.

📁 File System & Content

Read/Edit Files: Read, write, and surgically edit (replace/insert/delete lines) text files.

Directories: List, create, and delete folders.

Search: Grep code/text across files, glob patterns, and semantic search within my saved notes.

Info: File sizes, modification times, and directory trees.

🐍 Python & Execution

Run Code: Execute Python scripts directly (blocking) or start them as background jobs.

Outline: Extract class/function structures from .py files (good for checking code structure).

🧠 Knowledge & Logic

Notes: Create, list, search and delete notes (semantic vector search).

Recipes: Save and retrieve parameterized text templates.

Git: Run git commands (commit, diff, log, etc.)

⚙️ Specialized & Utility

Jobs: Start, stop, list, and poll background python jobs.

Media: Read PDFs (pages 0-10 at a time) and images into my vision context.

Discord: Send files directly to the chat.

Health: Check system status (memory, embedding, jobs).

✍️ System Prompt

Update Sys: Modify my core system instructions.

Peculiarities

(In no particular order)

1. Staying on task in overflowing contexts. Something that helps in addition to compaction for keeping task-coherence over the context lengths greater than the visible context is an update_sys tool, with a portion of the (instance-specific) system prompt directly editable by the agent to keep track of current directives and to-dos.
2. Repeated polling. The agent will poll background jobs repeatedly until they are complete, without pause. I have to interrupt to tell it to work on something else while a job runs (otherwise it should just use execute_python directly), though I expect over a longer period of time the trainer's turn-number-penalty will punish this behavior, since repeated polling increases the number of turns to finish a task substantially.
3. Cluttered memory. The agent would write notes for everything it did, amassing around 80 notes daily which would confuse it on recall (it would pick up an old task in the middle of a new one). I asked it to write a new system prompt with better instructions for using the notes that would reduce clutter and this problem has since been mostly mitigated (periodically asking it to do a "routine cleaning" of its notes also helps, this is something you could probably schedule in your orchestration loop).
4. Messy filesystem. The agent had a hard time adapting to the use of a file system that grows over time, it would write scripts in the main directory that I'd have to instruct it to clean (or clean up myself). This is mitigated by having a dedicated "scratchpad" folder and a note in the system prompt to use it (and clear it after use).
5. Stray or accidental end-of-turn. The agent will sometimes end its turn before completing a task, and by doing so "poison the context" such that this behavior became increasingly likely over time. In progress: I'm logging these occurrences to train a small classification model to be able to trigger a message, "Did you mean to end your turn?" If the agent responds "no," the last token for the last message will be targeted during nightly training with a negative advantage.
6. Concurrency. Make sure that your tool calls are either queued or running in separate processes per instance/channel. Oh my god. What a nightmare. I had some instances getting tool call outputs from others and not reporting it to me until it made a whole mess.
7. Collaboration and swarms. When I simply ask multiple instances to work together on the same project they blindly stumble around and never come up with a scheme to work together or to even communicate. They explode and make a horrible mess of the file system. For small models like this, you must establish inter-agent comms before starting a swarm on a project.
8. Sleeping on time. Sometimes I forget to run the "go to sleep" command before falling asleep myself. Two paths: A dedicated schedule or inactivity detection. I don't like either of them particularly much and it's not the end of the world if I miss one night, the sleep phase will just take 8 hours instead of 4 tomorrow. If you're on a lower end graphics card and training takes longer you might need to implement something here.
9. Caching images for RL. I have to keep many copies of the images that the model accesses in a separate folder unavailable for it to make changes to. Once per access, because I have no guarantee that the model won't delete an image or edit it after it's served its purpose.
10. Value model memory update order. You want these to queue up and happen all at once after the value model is done judging the entire night's tasks, don't just loop through and add them as it finishes evaluating them. If you had multiple instances working together, the value model will process one before the other and already have knowledge of their ability to work together which it can use to cheat.
11. llama-server slots. Don't get me started. I tried so hard to make sure no slots locked up and ended up just adding a function to restart the server wholesale instead. It only triggers about once per four days but it's still annoying.
12. Reading long files. Sometimes the agent would load a PDF or other file that was way too long for context, I've since restricted the read_file operations to 100 lines at a time and the PDF tool to 10 pages at a time.

I hope this list can help others looking to build (or prompt for the creation of) their own harnesses for maximum control and usability of local models.

Takeaway

Nightly training makes time-costly RL with CPU offloading seamless for local agents, they can just "sleep" at the same time as their users.

Fin

If you want to support me so I can keep creating and writing about projects like this:

ETH: 0x2de5fe90e3b8ad8f3634fff1dd8d348767cad0b2

_ - \.