AI LLMs can accidentally stumble through a text-adventure game

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

I am not impressed with this. If you give an LLM a limited set of options, it can pick from the options. That doesn’t strike me as a big deal. A much more ambitious project would be for an LLM to create a text-adventure game, and there they seem to fail, because of a lack of object permanence. I’ve tried playing text-adventure games with ChatGPT and it doesn’t enforce any limits on me. If I suggest that I have a magic sword, it will simply accept my assertion that I have a magic sword. Training an AI LLM to enforce limits would be an interesting advance.

A wide range of real-world applications is characterized by their symbolic nature, necessitating a strong capability for symbolic reasoning. This paper investigates the potential application of Large Language Models (LLMs) as symbolic reasoners. We focus on text-based games, significant benchmarks for agents with natural language capabilities, particularly in symbolic tasks like math, map reading, sorting, and applying common sense in text-based worlds. To facilitate these agents, we propose an LLM agent designed to tackle symbolic challenges and achieve in-game objectives. We begin by initializing the LLM agent and informing it of its role. The agent then receives observations and a set of valid actions from the text-based games, along with a specific symbolic module. With these inputs, the LLM agent chooses an action and interacts with the game environments. Our experimental results demonstrate that our method significantly enhances the capability of LLMs as automated agents for symbolic reasoning, and our LLM agent is effective in text-based games involving symbolic tasks, achieving an average performance of 88% across all tasks.

Post external references

  1. 1
    https://arxiv.org/abs/2401.09334
Source