LLMs suck at counting and therefore suck at writing valid lisp. (Source: I’ve seen people trying to use code agents with clojure))
Sorry it seems I miscounted the (parenthesis)
You are totally right my fix made the syntax valid but doesn’t work…
So am I. Add a linter.
A linter will tell you that you counted wrong it will not tell you where.
Citation needed.
I asked a LLM to start a stdio REPL, write code and then copy the code into the REPL. It closely mirrors actual Lisp development process - you write a couple of lines, eval it, see why it didn’t work, and repeat. Here’s what it came up with:
Starting Lisp
$ : > /tmp/lisptest-repl.commands; : > /tmp/lisptest-repl.log; (tail -n +1 -f /tmp/lisptest-repl.commands | sbcl --noinform > /tmp/lisptest-repl.log 2>&1) & echo $! > /tmp/lisptest-repl.pid; sleep 1; printf '(require :asdf)\n(asdf:load-asd #P"/home/user/lisptest/lisptest.asd")\n(asdf:load-system "lisptest")\n' >> /tmp/lisptest-repl.commands; sleep 2; tail -40 /tmp/lisptest-repl.log (timeout 10s) * ("ASDF" "asdf" "UIOP" "uiop") * #<ASDF/FIND-SYSTEM:DEFINE-OP > #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}> * T * Took 3.1sInvalid input
Feeding a single
)into the REPL. The LLM later recovered it by sending the abort command to the debugger.$ printf ')\n' >> /tmp/lima-repl.commands; sleep 1; tail -25 /tmp/lima-repl.log (timeout 10s) * #<ASDF/FIND-SYSTEM:DEFINE-OP > #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}> * T * #<SB-THREAD:THREAD tid=151 "lima-echo" RUNNING {1203EB14E3}> * Echo server listening on 127.0.0.1:5555 T * NIL * #<SB-THREAD:THREAD tid=0 "lima-echo" RUNNING {1201C0DDC3}> * Echo server listening on 127.0.0.1:5555 T * debugger invoked on a SB-INT:SIMPLE-READER-ERROR in thread #<THREAD tid=142 "main thread" RUNNING {12003D8003}>: unmatched close parenthesis Stream: #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}> Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [CONTINUE] Ignore it 1: [ABORT ] Exit debugger, returning to top level. (SB-IMPL::READ-RIGHT-PAREN #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}> #<unused argument>) 0] Took 1.0sHotpatching
This thing activates a new function definition. Next time it’s called, the new one gets executed.
printf '(load #P"/home/user/lisptest/src/server.lisp")\n' >> /tmp/lisptest-repl.commands(defn process-scores [user & scores] (println "User:" user) (reduce + scores (filter pos? scores))can be either
(defn process-scores [user & scores] (println "User:" user) (reduce + scores) (filter pos? scores))or
(defn process-scores [user & scores] (println "User:" user) (reduce + scores (filter pos? scores)))Yep, can happen. Do we know if this tends to happen more often with lispy (than curlybracy) languages?
J as all the benefits of lisp, and more, without annoying ((())). LLMs are bad at understanding it though (which is one of the claims that Graham makes for lisp here). Lisp would have the same lack of training data compared to python/js to have models good at programming in lisp?
is the plan here to give an llm access to a running instance of the app and just… make changes? fantastic, can’t imagine what might go wrong
Yep, and it works surprisingly well. Of course you’ll need to run the LLM and the app both in a container to discourage accessing your SSH keys and such.