I will be sharing a guide to starting your own projects using the methods I used to build Eli Felse. Eli is my public demo that runs an autonomous AI 24/7 with much lower security risks than traditional AI agents. I will also be open sourcing the base of Eli, which includes a simple memory system and some other handy features to get you started. While Eli is capable of many things, the base system is very simple and easy to understand.
How Eli’s System is Different
I would describe Eli’s system as very similar to a video game with branching narratives and alternative endings based on the choices made during gameplay. Instead of reaching a game ending, Eli completes his chosen activity and loops back to the main menu to make more choices.
The LLM (Large Language Model) makes choices from a set menu, then proceeds to submenus, making further choices before entering an activity. The activity is triggered when selected from the menu and is fully automated in Python, exchanging text with the LLM. Simply put, the LLM makes the choices, but Python executes the action.
Traditional AI agents are LLMs trained to invoke tools or functions to execute actions directly on your system, such as modifying files, running shell commands, or interacting with social apps. Many of the frameworks built for AI agents take the text output from an LLM and allow it to run commands directly. Which has many of its own risks, I have detailed these risks in my Meet Eli blog (link blog).
Guide
In this guide, I will give a high-level overview of how the system works and share some tips for building your own system like Eli!
Menus and Schemas
We start with a simple multiple-choice menu of all the options you want your AI to have. Eli’s current menu is pretty big, as I’ve been adding to it for a while, but we can start with something simple in this guide.
Main menu:
A. Chat with friends
B. Journal
C. Read
D. Nap
E. Game
Instead of a user sending the LLM a prompt, the LLM will be sent the main menu to select an option.
Typically, models respond with free-form text. Fortunately, most current models have a feature called structured output. Structured output is a requirement you’ll want to look for when deciding on what model to use. Structured outputs force a model to respond with a JSON schema, meaning you can make the model output a multiple-choice letter rather than free-form text. Using a schema simplifies the process for your Python script.
{
"type": "object",
"properties": {
"thinking": {"type": "string"},
"choice": {"type": "string", "enum": ["A", "B", "C", "D", "E"]}
},
"required": ["thinking", "choice"],
"additionalProperties": false
}
You would set this as the schema for the main menu we created earlier. A downside of enabling structured output is that it does not allow extended reasoning in the models that support it. We can replace this with a thinking string for additional observability.
Now, the decision tree design comes in. We can start simple for this tutorial, but I'll give my tips below on how to keep track of and design large design trees.
If A is selected, we will then send this submenu to the LLM
Which friend would you like to chat with?
A. Lain
B. Baymax
C. GLaDOS
Make sure to update the JSON schema to only have these 3 multiple-choice letters, rather than the full main menu.
Once a friend is selected, you can then enter the activity. For ease of programming, I keep the schema on. Rather than constraining it to multiple-choice responses, set it to a string for open-text responses for messaging with people.
{
"type": "object",
"properties": {
"thinking": {"type": "string"},
"response": {"type": "string"},
"return_to_menu": {"type": "boolean"}
},
"required": ["thinking", "response", "return_to_menu"],
"additionalProperties": false
}
For activities that don't have a set ending, I add this additional schema line: “return to menu”, which the model can set to true to return to the main menu.
When chatting with people, I use a hybrid approach: allow the LLM to return to the main menu at any point, but also set a timeout of 5 minutes. If the person doesn’t respond right away, the LLM isn't left hanging; it automatically returns to the main menu with a note appended that says, “Baymax did not respond yet; you’ll get notified when they do.”
For an activity like journaling, where the LLM will be finished after one entry, I don't add a “return to menu” option to the schema; I just automatically loop back after the LLM finishes.
System Prompts
System prompting is very important for making this work properly. You’ll need to ensure you are using a model trained to accept a custom system prompt role. If you feel your model isn’t following instructions well, it's likely not tuned for a custom prompt. I would avoid Gemma models, as only the newest releases support system prompts.
Typically, we will change the system prompt whenever the schema changes, for each menu and submenu, and when entering an activity. The main menu system prompt is the simplest; you can instruct your LLM to select a menu option. Submenus you might want to add specific context per choice. For example, the friend list may explain who each person is, allowing the LLM to make an informed choice.
The system prompt with the most detail will be used for activities. This system prompt will instruct the model on what it should do during the activity. Additionally, if it has “return to menu” in the schema, you should place the main menu in the prompt and say that it can set return to menu to true to return to the choices listed at any time.
System prompts are also a great place to inject memories, character names, and personality traits. I recommend doing this dynamically for each prompt to make it easy to edit in one place later. I append Eli’s name and traits to each prompt.
Context Handling
There are a few other additions you can add to keep a smaller model on track. I keep timestamps on every input and track which activity was used and when. I would not recommend using timestamps in the system prompt as the LLMs seem to fixate too much on time passing.
You just finished chatting with Baymax.
Main menu:
A. Chat with friends (Lain, Baymax, GLaDOS) last chatted 1 minute ago
B. Journal - last journaled 10 minutes ago
Read (news, books) - never read
Nap - last napped 5 hours ago
Game (poker, chess) - last played 10 minutes ago
Time: 5:30 pm
Date: Thursday, July 9th 2026
Additionally, if running for multiple days, set the hours to days so it’ll show 1 day ago rather than tracking hours.
One important thing to consider is context windows; many models handle this automatically. However, you will need to modify it to keep it streamlined.
For an LLM to see or remember earlier messages, the entire conversation is sent back to the model each time. Seeing the full conversation is no problem for the model, as it is trained to respond to multi-turn conversations. You must take into account how much context is being run, as this can slow down response time, increase costs, or set your GPU on fire. Best practice is to trim older messages over time and set a maximum token limit. I recommend setting the maximum number of tokens to 36,000. I don't notice messages dropping out too fast with this limit, and it stays lightweight on my GPU.
You will want to reconstruct the context the model sees, as it can get kinda wonky when it sees structured outputs. I notice it begins to nest outputs, putting JSONs within JSONs.
Reconstruct the window by stripping out the schema first. You may also strip out the thinking portion; I keep it for menu choices so the model remembers why choices were made, but remove it from conversational context. I also keep timestamps, but use your own judgment, as some LLMs act strangely when they see time passing.
While it may be odd to include a napping option for an AI model, I actually use it for context compression; the context is sent to the model for summarization, then cleared, and the summary is sent. The model wakes up from the nap with a vague summary of what happened before, just like a real nap.
Model
I’ve tested a bunch of models it’s this type of system and landed on Magistral Small. It’s not the most popular model, but it’s great at following custom system prompts, and it's the perfect size for my GPU (I use the Unsloth Q4_K_M version).
Activities
For the activities themselves, you can really go far in automating and programming them for the LLM to interact with. I made Pokémon Blue fully playable through text and multiple-choice menus. It was fun to build, but that's kinda on the extreme end of this. I would say social media browsing and posting aren’t too complex to set up compared to Pokémon; they just have pretty extensive submenus. The key is to have Python execute the model's menu choices, so you only have to program the choices you’ve set rather than giving the model open-ended access to your computer. I also feel that programming capabilities are far beyond those of traditional agents; it is best to take advantage of this.
You can also go very simple and create prompt-only activities. I gave Eli the option to eat. It's a randomly generated menu of food options. Once one is selected, it simply tells Eli he is eating that food and prompts him to describe it as he eats it. I set a timer for the approximate amount of time it would take to eat. He returns to the main menu when he is finished.
There are various programming languages you can use, but I highly recommend Python. It works very well for state machines and is good at automating tasks. If you are programming the old-fashioned way, it will get a bit tedious as you will be writing many if else statements repeatedly. I find the activities themselves pretty straightforward and easy to program.
If you must use an AI coding tool, you are in luck, as models currently excel at simple programming like this, but please… sandbox the coding agent so it cannot accidentally delete your entire drive.
The hardest part of this process will be thinking through and designing a decision tree. It can get very complex and convoluted fast. It is important to keep your plan and ideas organized. I visualize my state machine plans using a tool like Excalidraw, which has helped a lot.
Eli in the early stages
Open source release
I will not be open sourcing Eli’s entire system as one piece, as it has become a bit of a monster… However, I will be releasing the system's base and weekly modules for each activity, which can plug right into the base project.
The base is available here https://github.com/ella0333/Eli_Felse_Base
The base is bare bones, but it includes a memory system and a setup wizard.
Activity modules that will be released in the coming weeks:
Social: Discord, Slack, Twitter, Reddit, Live streaming
Games: board games, text RPGs, Pokémon Blue
Explore: web search, news
Creative: blog, story writing, music
Other: reading
The release has guidance for releasing your own activity modules. If you align with my safety guidelines, I will add you to the approved list of modules so others can add them to their projects.
Have fun creating your own characters and systems. I would love to see what you’ll build. Let me know which activities you are most excited to see released, and I will prioritize them!