The Unreasonable Effectiveness of an LLM Agent Loop with Tool Use

Project Description

This project explores the effectiveness of a simple agent loop for LLMs with tool use, specifically in the context of an AI Programming Assistant called Sketch. The core idea is a nine-line Python loop that continuously feeds user input to an LLM, processes tool calls from the LLM's output (in this case, primarily bash commands), and then feeds the results back to the LLM.

def loop(llm):
    msg = user_input()
    while True:
        output, tool_calls = llm(msg)
        print("Agent: ", output)
        if tool_calls:
            msg = [ handle_tool_call(tc) for tc in tool_calls ]
        else:
            msg = user_input()

Usage Instructions

The primary usage involves interacting with an AI programming assistant that leverages this agent loop. Users provide prompts or questions, and the system, powered by an LLM (such as Claude 3.7 Sonnet) with access to bash and other specialized tools, attempts to resolve the task. This can range from complex git operations to refactoring code and handling type checker errors. The system can be persistent, adapting to the environment and even installing necessary tools or adjusting to command-line differences.

Key Features

• Simple Agent Loop: A concise, nine-line core logic for LLM-tool interaction.
• Tool Use Integration: LLM returns output that corresponds to a schema, enabling tool calls.
• Bash Tool: Primary and powerful tool, allowing the LLM to execute shell commands.
• Specialized Tools: Beyond bash, Sketch incorporates additional tools to improve quality, speed iterations, and facilitate developer workflows (e.g., for text editing).
• Persistence and Adaptability: The agent can learn and adapt to its environment, such as installing missing tools or adjusting to different command-line options.
• Problem Solving: Capable of handling various programming tasks, from git operations to code changes and error handling.

Target Users

• Software Developers: Individuals who want an AI assistant to streamline daily programming tasks, automate repetitive operations, and assist with complex coding challenges.
• DevOps Professionals: Those seeking to automate system administration tasks and scripting.

Project Links

• Blog Post: sketch.dev/blog/agent-loop
• Alternative Blog Post: philz.dev/blog/agent-loop/

Application Scenarios

• Git Operations: Automating esoteric git commands or assisting with merges.
• Code Refactoring: Making large-scale code changes, such as changing a type across a codebase and resolving resulting type checker errors.
• Development Environment Setup: Automatically installing missing tools or adapting to different command-line utilities.
• Debugging and Log Analysis: Correlating stack traces with git commits, performing an initial pass on complex debug information.
• General Automation: Automating day-to-day tedious tasks that are too specific for general-purpose tools or too unstable for traditional automation.