Demo

Coding Agent痛点

自作主张。即使使用了plan mode/openspec一类的计划工具，agent仍然会在实际执行过程中偏离人类意图。原因包括上下文窗口大小、计划本身没有涵盖全部细节等。理想情况下agent应该能够在遇到关键决策时咨询人类，但是实际技术水平还未达到。现在human in the loop实际上要求human on the watch，这样开发效率很低。
作弊。这实际上是上一点的子问题。由于各模型厂商的商业需求，模型被训练得越来越倾向于长时间运行且一次性解决整个问题而不是先问人类，这对宣传和卖token都大有裨益。但这也就使得模型倾向于走捷径。在不太严重的情况下，它会留下TODO然后绕过一些case；在严重的情况下，他会通过硬编码等方式作弊。
遗忘/长视野问题。老生常谈的问题了。

尝试

由前面三点，一个自然的推论是我们应该切分子任务，降低它的Context压力以及给出更明确的指导。
另一个可能的方向是：更细粒度地操控agent的行为。现有的LangGraph一类的工作流系统，虽然可以在一定程度上操控agent的行为，但本质上是从构建一个multi-agent系统然后从整个系统外部去操控/观察agent的行为。
结合这两点，我们作如下尝试：做一个API允许写prompt，并且在prompt里允许引用程序里的表达式。

Semantic Sorting

这个例子展示了如何在程序中调用agent。这里写了一个插入排序对一个作家列表按照伟大程度排序。其中“比较两个作家谁更伟大”这个comparator是一个agent完成的。

Code

use agent_lang::{Agent, Config, prompt};

// -----------------------
// 核心代码开始
// -----------------------

fn writer_list() -> Vec<String> {
    ["郭沫若", "曹雪芹", "韩寒", "鲁迅", "张爱玲", "莫言"]
        .into_iter()
        .map(str::to_string)
        .collect()
}

fn local_rank(writer: &str) -> usize {
    match writer {
        "曹雪芹" => 0,
        "鲁迅" => 1,
        "张爱玲" => 2,
        "莫言" => 3,
        "郭沫若" => 4,
        "韩寒" => 5,
        _ => usize::MAX,
    }
}

fn main_program(config: Config) -> Vec<String> {
    let mut writers = writer_list();

    let agent = Agent::with_config(config).expect("backend should be configured");

    let writer_is_greater = |left: &str, right: &str| -> bool {
        if playground_uses_echo() {
            return local_rank(left) < local_rank(right);
        }

        let mut context = agent.fork();
        prompt!(
            &mut context,
            "
            Which writer is greater in overall literary achievement and historical influence?

            Writer A: {left}
            Writer B: {right}

            If Writer A is greater, output exactly:
            true

            If Writer B is greater, output exactly:
            false

            Do not output reasoning, Markdown, punctuation, or any other text.
            "
        )
    };

    // Insertion sort
    for index in 1..writers.len() {
        let mut current = index;
        while current > 0 && writer_is_greater(&writers[current], &writers[current - 1]) {
            writers.swap(current, current - 1);
            current -= 1;
        }
    }

    writers
}

// -----------------------
// 核心代码结束
// -----------------------

fn assert_same_members(sorted: &[String], original: &[String]) {
    let mut sorted = sorted.to_vec();
    let mut original = original.to_vec();
    sorted.sort();
    original.sort();
    assert_eq!(sorted, original);
}

fn assert_ordered_before(writers: &[String], earlier: &str, later: &str) {
    let earlier_index = writers
        .iter()
        .position(|writer| writer == earlier)
        .unwrap_or_else(|| panic!("missing writer {earlier}"));
    let later_index = writers
        .iter()
        .position(|writer| writer == later)
        .unwrap_or_else(|| panic!("missing writer {later}"));
    assert!(
        earlier_index < later_index,
        "expected {earlier} to sort before {later}; sorted writers: {writers:?}"
    );
}

let mut config = playground_config("main");
config.max_turns = 8;
config.limits.max_model_calls = 96;
config.system.push(
    "For literary comparison prompts, output exactly true or false with no reasoning, Markdown, punctuation, or extra text. Prefer broad, long-standing literary-critical consensus over sales or current popularity."
        .to_string(),
);

let original = writer_list();
let writers = main_program(config);
println!("[sorted writers]\n{}", writers.join(" > "));

assert_same_members(&writers, &original);
assert_ordered_before(&writers, "鲁迅", "韩寒");
assert_ordered_before(&writers, "曹雪芹", "韩寒");

Output

Click Run.

Programming by Examples

这个例子是program synthesis领域的经典例子。给出一个DSL，以及一堆输入输出对，要求用这个DSL写一个变换器，把输入变换为输出，并且对所有例子都成立。

传统的做法是用搜索算法在DSL空间中搜索出一个正确的DSL程序。这里我们尝试用agent来做这个搜索。agent的prompt里包含了DSL定义和一个DSL的解释器。然后我们逐个给agent输入例子，要求它输出一个DSL程序对这些例子都成立。

Code

use agent_lang::{Agent, Config, TraceObserver, program, prompt};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use std::sync::Arc;

// -----------------------
// 核心代码开始
// -----------------------

fn main_program<F>(
    mut config: Config,
    dsl_schema_json: &str,
    interpreter: agent_lang::runtime::Program<RunDslInput, RunDslOutput, F>,
    examples: &[Example],
) -> Expr
where
    F: Fn(RunDslInput) -> RunDslOutput + Send + Sync + 'static,
{
    if playground_uses_echo() {
        let learned = Expr::Concat {
            parts: vec![
                Expr::Upper {
                    expr: Box::new(Expr::Replace {
                        expr: Box::new(Expr::Between {
                            left: "invoice_".into(),
                            right: "_final".into(),
                        }),
                        from: "-".into(),
                        to: " ".into(),
                    }),
                },
                Expr::Lit { value: " / ".into() },
                Expr::Prefix { n: 10 },
            ],
        };

        for (index, example) in examples.iter().enumerate() {
            println!(
                "[learned DSL after example {}]\n{}",
                index + 1,
                Dsl.pretty(&learned)
            );
            println!(
                "  {} => {:?}",
                example.input,
                Dsl.eval(&learned, &example.input)
            );
        }

        return learned;
    }

    config.system.push(format!(
        "
        You are a stateful programming-by-example learner.

        DSL definition JSON schema:
        {dsl_schema_json}

        After each example, revise one DSL program that satisfies every example seen so far.
        "
    ));
    let mut agent = Agent::with_config(config).expect("real backend should be configured");

    let mut learned = None;
    for (index, example) in examples.iter().enumerate() {
        let raw: String = prompt!(
            agent.context_mut(),
            "
            New example JSON:
            {example}

            - Revise your previous DSL program using all examples seen so far.
            - Before returning, call {interpreter} with an object containing ast and input to test your candidate DSL program on every example seen so far.
            - Revise until every result exactly matches the example output.

            Return ONLY one Expr JSON object.
            "
        );

        let program: Expr = parse_json_object(&raw);
        println!(
            "[learned DSL after example {}]\n{}",
            index + 1,
            Dsl.pretty(&program)
        );
        learned = Some(program);
    }

    learned.expect("main_program should receive at least one example")
}

/// example 序列
fn example_sequence() -> Vec<Example> {
    vec![
        ex(
            "2024-01-15_invoice_acme-corp_final.pdf",
            "ACME CORP / 2024-01-15",
        ),
        ex(
            "2024-02-03_invoice_zenith-labs_final.pdf",
            "ZENITH LABS / 2024-02-03",
        ),
        ex(
            "2024-04-01_invoice_north-atlantic-research_final.pdf",
            "NORTH ATLANTIC RESEARCH / 2024-04-01",
        ),
        ex(
            "2024-05-09_invoice_deep-space_final.csv",
            "DEEP SPACE / 2024-05-09",
        ),
    ]
}

/// DSL 定义
#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
#[serde(tag = "kind", rename_all = "snake_case")]
enum Expr {
    Lit {
        value: String,
    },
    Prefix {
        n: usize,
    },
    Between {
        left: String,
        right: String,
    },
    Split {
        sep: String,
        index: usize,
    },
    Replace {
        expr: Box<Expr>,
        from: String,
        to: String,
    },
    Upper {
        expr: Box<Expr>,
    },
    Concat {
        parts: Vec<Expr>,
    },
}

// -----------------------
// 核心代码结束
// -----------------------

#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
struct Example {
    input: String,
    output: String,
}

#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
#[serde(untagged)]
enum EvalResult {
    Output(String),
    Error { error: String },
}

impl EvalResult {
    fn output(value: impl Into<String>) -> Self {
        Self::Output(value.into())
    }

    fn error(error: impl Into<String>) -> Self {
        Self::Error {
            error: error.into(),
        }
    }
}

struct Dsl;

impl Dsl {
    fn eval(&self, expr: &Expr, input: &str) -> EvalResult {
        match expr {
            Expr::Lit { value } => EvalResult::output(value),
            Expr::Prefix { n } => EvalResult::output(input.chars().take(*n).collect::<String>()),
            Expr::Between { left, right } => self
                .between(input, left, right)
                .map(EvalResult::output)
                .unwrap_or_else(|| EvalResult::error("marker_not_found")),
            Expr::Split { sep, index } => input
                .split(sep)
                .nth(*index)
                .map(EvalResult::output)
                .unwrap_or_else(|| EvalResult::error("split_index_out_of_range")),
            Expr::Replace { expr, from, to } => {
                self.map(self.eval(expr, input), |s| s.replace(from, to))
            }
            Expr::Upper { expr } => self.map(self.eval(expr, input), |s| s.to_uppercase()),
            Expr::Concat { parts } => {
                let mut output = String::new();
                for part in parts {
                    match self.eval(part, input) {
                        EvalResult::Output(s) => output.push_str(&s),
                        error @ EvalResult::Error { .. } => return error,
                    }
                }
                EvalResult::output(output)
            }
        }
    }

    fn accepts_all(&self, expr: &Expr, examples: &[Example]) -> bool {
        examples
            .iter()
            .all(|e| self.eval(expr, &e.input) == EvalResult::Output(e.output.clone()))
    }

    fn pretty(&self, expr: &Expr) -> String {
        match expr {
            Expr::Lit { value } => format!("\"{value}\""),
            Expr::Prefix { n } => format!("prefix({n})"),
            Expr::Between { left, right } => format!("between(\"{left}\", \"{right}\")"),
            Expr::Split { sep, index } => format!("split(\"{sep}\", {index})"),
            Expr::Replace { expr, from, to } => {
                format!("replace({}, \"{from}\", \"{to}\")", self.pretty(expr))
            }
            Expr::Upper { expr } => format!("upper({})", self.pretty(expr)),
            Expr::Concat { parts } => format!(
                "concat({})",
                parts
                    .iter()
                    .map(|e| self.pretty(e))
                    .collect::<Vec<_>>()
                    .join(", ")
            ),
        }
    }

    fn map(&self, value: EvalResult, f: impl FnOnce(String) -> String) -> EvalResult {
        match value {
            EvalResult::Output(s) => EvalResult::output(f(s)),
            error @ EvalResult::Error { .. } => error,
        }
    }

    fn between(&self, input: &str, left: &str, right: &str) -> Option<String> {
        let start = input.find(left)? + left.len();
        let end = input[start..].find(right)? + start;
        Some(input[start..end].to_string())
    }
}

#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
struct RunDslInput {
    ast: Expr,
    input: String,
}

#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
struct RunDslOutput {
    pretty: String,
    result: EvalResult,
}

fn parse_json_object<T: for<'de> Deserialize<'de>>(text: &str) -> T {
    let trimmed = text.trim();
    let json = trimmed
        .split_once("```json")
        .and_then(|(_, rest)| rest.split_once("```").map(|(json, _)| json))
        .or_else(|| {
            trimmed
                .split_once("```")
                .and_then(|(_, rest)| rest.split_once("```").map(|(json, _)| json))
        })
        .unwrap_or(trimmed)
        .trim();
    serde_json::from_str(json)
        .unwrap_or_else(|error| panic!("failed to parse JSON: {error}\n{json}"))
}

fn demo_config(trace: TraceObserver) -> Config {
    let mut config = playground_config("main");
    config.max_turns = 12;
    config.limits.max_model_calls = 64;
    config.limits.max_tool_calls = 64;
    config.observer = Arc::new(trace);
    config.system.push(
        "Call required tools before answering. Return exact JSON when asked for JSON.".into(),
    );
    config
}

fn ex(input: &str, output: &str) -> Example {
    Example {
        input: input.into(),
        output: output.into(),
    }
}

let model_trace = TraceObserver::new();
let config = demo_config(model_trace.clone());
let expr_schema_json = serde_json::to_string_pretty(&schemars::schema_for!(Expr)).unwrap();
let interpreter = program(|request: RunDslInput| {
    let dsl = Dsl;
    RunDslOutput {
        pretty: dsl.pretty(&request.ast),
        result: dsl.eval(&request.ast, &request.input),
    }
});
let examples = example_sequence();
let learned = main_program(config, &expr_schema_json, interpreter, &examples);
let dsl = Dsl;

assert!(dsl.accepts_all(&learned, &examples));
assert_eq!(
    serde_json::from_value::<Expr>(serde_json::to_value(&learned).unwrap()).unwrap(),
    learned
);

let model_trace = model_trace.trace().pretty();
println!("[model trace]\n{model_trace}");
if !playground_uses_echo() {
    assert!(model_trace.contains("ModelCallStart"));
}

Output

Click Run.

小结

这两个例子展示了agent/prompt如何调用program，program如何调用agent/prompt。可以想象prompt与program互相嵌套可以形成一个无限嵌套的结构。
这两个例子模糊地展示了我们如何接近前面提到的两个方向：切分子任务和更细粒度地操控agent的行为。下一步我们希望把这个设计推广到更复杂的场景中，如SWE-bench之类的实际任务里，考察这些设计在实际任务中的表现。
在这两个例子的实现过程中，发现了一些Rust语言的特性与这个设计的摩擦：
- 当我们想把程序本身（代码原文）暴露给agent时，Rust的宏系统不够灵活。
- Rust的Generics参数个数限制了我们在prompt里Rust表达式的灵活性。Rust的泛型参数个数是有限制的，我们不能在prompt里引用任意个数的program表达式。
这里我们仍然是把agent当作一个黑盒子来使用的。但是根据Agent=Model+Context的观点，如果我们把Model看作常量，那么agent的行为就完全由Context决定了。我们可以把Context看作一个continuation，然后去操控这个continuation本身。