Large Language Models and the Socratic Method

Table of Contents

Imagine a lively exchange between Socrates, the father of Western philosophy, and an advanced Large Language Model (LLM): Socrates, whose probing inquiries have inspired thinkers for millennia, sparring with an LLM, trained on an unprecedented wealth of human knowledge.

This blog post is an intellectual journey through various ideas ranging from ancient philosophy to cutting-edge AI. Fasten your seatbelts as we delve into concepts such as the Socratic method, System 1 and System 2 thinking, and the Tree of Thoughts prompting technique.

A meeting of minds #

I know that I know nothing.
– Socrates

Socrates’ dialogues, as recorded by his student Plato, present an enlightening means of dissecting complex philosophical concepts.

The Socratic method #

This practice, known as the Socratic method, leverages incisive questions to stimulate critical thinking and expose the assumptions underlying one’s beliefs. It represents an intellectual journey towards logical conclusions, employing a process of elimination known as “elenchus”.

Just as Socrates sought to expose truths through patient questioning, modern LLMs can be pushed to uncover hidden knowledge. Think of the Socratic method as a persistent child, incessantly asking “why?” until every assumption is laid bare.

Plato chose this dialogue form to situate the reader at the heart of the intellectual process, allowing for an active engagement with the content. By observing the evolution of arguments, we cultivate our critical thinking skills.

Thesis-antithesis-synthesis #

Many times, this progression is guided by the thesis-antithesis-synthesis formula, a concept often attributed to Hegel. This model posits that an initial proposition (the thesis) meets its opposite (the antithesis), and their tension is resolved at a higher level of truth (the synthesis).

graph LR A[Thesis] B[Antithesis] C[Synthesis] A --> B B --> C A --> C

Such tools promote critical thinking, challenge assumptions, and foster a deeper understanding of complex ideas. In essence, they lay the foundation for fruitful discourse, enabling conflict resolution and consensus-building.

Fast and slow thinking #

As we examine the strengths and limitations of LLMs, we might draw parallels with the System 1 and System 2 thinking models, terms popularized by Daniel Kahneman.

graph TD A[Thinking Systems] --> B[System 1] A --> C[System 2] B --> D[fast] B --> E[instinctive] B --> F[emotional] C --> G[slow] C --> H[deliberative] C --> I[logical]

System 1 #

System 1 thinking, characterized by swift, automatic, and often unconscious thought processes, mirrors the LLMs’ aptitude for fast, data-driven responses. Here, LLMs demonstrate superior prowess, producing text at volumes and speeds unmatchable by any human.

System 2 #

However, LLMs struggle with System 2 tasks requiring strategic planning and complex reasoning, reflecting the slower, deliberate, analytical thinking in humans. This difference primarily arises from LLMs’ missing ability to strategically plan ahead, as they only predict one next token at a time.

If we consider Socrates in this frame, he would undeniably operate within the System 2 model, dissecting and analyzing ideas, something our current AI models are striving to accomplish.

Trees of thoughts #

How can LLMs become better at System 2 tasks?

Branching out and going deep #

The “Tree of Thoughts” method, proposed by Yao et al. and Long in 2023, provides a solution. The method allows the AI to simulate a multi-branched conversation.

Visualize this method as a tree: the central question forms the trunk while the possible answers branch out, each representing a different outcome. It then evaluates each branch before choosing the one that best suits the question.

Depending on how you want to use the method, the tree can become very branched out or “deep”. Consider the following example where three answers branch out for a central question:

graph TD A["Central question"] A --> B["Answer 1"] A --> C["Answer 2"] A --> D["Answer 3"] B --> E["Sub-question"] C --> F["Sub-question"] E --> G["Answer 1a"] E --> H["Answer 1b"] F --> I["Answer 2a"] F --> J["Answer 2b"] G --> K["Synthesis"] J --> K K --> O["Final answer"]

Let’s say you rule out Answer 3 but want to go deeper for the others. You pose a sub-question, leading to further answers. You select Answer 1a and 2b and ask the LLM to find a synthesis of the two, leading to the final answer.

The Tree of Thoughts method augments the quality of AI’s outputs, aligning them more closely with System 2 thinking.

Self-talking LLMs #

Now, how could we apply the Tree of Thoughts in practice while keeping our manual effort low?

One way to do it is just “programming” the LLM to execute the method. Let’s employ Hulbert’s prompt technique, introduced in 2023, to obtain the following example prompt:

Imagine three different experts answering the question, ‘What is the meaning of life?’ All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realizes they’re wrong at any point then they leave.

This leads the LLM into an internal dialogue, with the ’experts’ talking to each other. This is where the thesis-antithesis-synthesis formula might come into play, enabling the AI to consider multiple viewpoints, challenge them, and arrive at an improved understanding.

Here’s how GPT-4 evaluates the prompt from above:

Round 1
Philosopher: The question is indeed complex, and its answer may lie in a myriad of personal and shared experiences. To begin, it would be prudent to dissect the nature of ‘meaning’. What do we constitute as ‘meaning’, and how does that apply to life itself?
Biologist: The meaning of life from a biological perspective could be seen as survival and reproduction, the passing of genes from one generation to the next. This is a fundamental process observed in all life forms.
Religious Scholar: From a religious perspective, the meaning of life often ties to the purpose given by divine entities or forces. This purpose varies across religions and individual interpretations. However, for the sake of analysis, I’ll take a monotheistic standpoint where life’s purpose is to serve and love God. Round 2
Philosopher: Having defined meaning as an interpretation of events that provides purpose, I posit that life’s meaning may be individualistic, influenced by one’s experiences, goals, and personal growth. It could revolve around the pursuit of knowledge, happiness, or societal contribution.
Biologist: I stand by my initial point on survival and reproduction. It is a universal constant in biology and a driving force of evolution. However, given the complexity of human behavior and society, it’s clear that many individuals find additional, more abstract purposes for their lives.
Religious Scholar: Expanding on my initial point, many religious texts imply that life is a trial or a journey. It is a transient phase where actions and decisions influence an eternal afterlife. The purpose is not just to serve and love God but also to learn, grow, and refine one’s moral and ethical self.
Round 3
Philosopher: Reflecting on the inputs from my peers, I maintain that while biological and religious perspectives offer valid interpretations, they may not encompass the entire scope of human experience. The meaning of life is a deeply personal and subjective concept, potentially a synthesis of various factors – biological, psychological, spiritual, and societal.
Biologist: The perspectives shared have reaffirmed the complexity of the question. From a purely biological point of view, I could argue survival and reproduction. However, in the context of human consciousness and society, this explanation feels incomplete.
Religious Scholar: With the understanding from both the philosopher and the biologist, I concede that while religious texts provide a prescribed meaning of life, individual experiences and interpretations might differ, adding layers to the understanding of life’s meaning.
At this point, none of the experts are necessarily wrong. They’ve simply offered different perspectives that reflect the complexity and diversity of human understanding and experience. The dialogue could continue, with each expert drawing from the others to build a richer, more nuanced understanding of the meaning of life.

So, what is the meaning of life?

In the face of this timeless question, don’t expect a satisfactory answer. It should not come as a surprise that the experts did not reach a consensus.

But that’s fine.

The resulting exchange of opinions is one step closer to obtaining a final answer. It can serve as a good starting point for further questions. For example, you could dig deeper into some specific aspects that were mentioned by the experts.

Or, you could force the LLM to come to a conclusion, such as in the following way:

If the experts had to agree on a unified statement, what would it be?

graph TD A("Central Question: 'What is the meaning of life?'") --> B1("Branch 1: Philosopher's Perspective") A --> B2("Branch 2: Biologist's Perspective") A --> B3("Branch 3: Religious Scholar's Perspective") B1 --> C1("Understanding 'Meaning'") C1 --> D1("Life's Meaning as Individualistic") D1 --> E1("Synthesis of Various Factors") B2 --> C2("Survival and Reproduction") C2 --> D2("Biological Imperative and Human Behaviour") D2 --> E2("Influence of Consciousness and Society") B3 --> C3("Purpose Given by Divine Entities") C3 --> D3("Life as a Trial or Journey") D3 --> E3("Refinement of Moral and Ethical Self") E1 --> F("Unified Statement") E2 --> F E3 --> F

GPT-4’s answer:

The meaning of life, a concept universally intriguing yet inherently subjective, can be seen as a complex interplay of our biological imperative for survival and reproduction, personal interpretations influenced by individual experiences and aspirations, and spiritual or religious beliefs that may prescribe purpose or moral directives. This multifaceted understanding underscores the richness and diversity of human life, affirming that the pursuit of meaning remains an integral part of our shared human experience.

The Socratic dialogue in LLMs #

Let’s connect that idea of self-talking LLMs with the philosopher who stood at the beginning of this article, Socrates, to end up with the following synthesis:

What if we prompt an LLM to simulate a Socratic dialogue?

Take a look at the following example:

Imagine there are two friends with different points of view about the question “What is better: to work hard or work smart?”. How would a Socratic dialogue between the two look like?

GPT-4 outputs:

Alice: What do you think, Bob? Is it better to work hard or work smart?
Bob: I think working hard is the best way. There’s no substitute for effort.
Alice: Interesting viewpoint. But have you ever thought of a scenario where you work hard, but not in the right direction? Would the effort still be worthwhile?
Bob: Well, I guess it would be wasted, but you would eventually find the right path through hard work.
Alice: Right. But don’t you think that if you worked smart initially, you could potentially avoid wasted effort and reach your goal more quickly?
Bob: But isn’t working smart just a fancy term for being lazy and trying to avoid putting in the effort?
Alice: Not necessarily. To work smart can also mean to prioritize, strategize and make decisions that maximize your productivity. Would you agree that productivity is not solely defined by the hours spent on a task, but rather by the results obtained?
Bob: I agree that productivity is about results. But those results still come from hard work.
Alice: Indeed, hard work is a crucial component. But consider this: two people have the same task, one works for ten hours straight without much strategy, while the other strategically plans and completes the task in five hours. The results are similar, but who worked smarter?
Bob: I see what you mean. The one who completed the task in five hours definitely worked smarter. But wouldn’t he or she also be able to accomplish more by then working hard for the next five hours?
Alice: Yes, that’s a valid point. So, it seems like the real solution lies not in choosing to work hard or smart, but in effectively integrating the two. Working smart to optimize productivity, and working hard to capitalize on that optimization. What do you think?
Bob: I think I understand your perspective better now. Working smart doesn’t exclude working hard. It’s about making the best use of our time and effort. I think I agree with that.
Alice: Exactly. And as the saying goes, it’s not about working harder, it’s about working smarter. But always remember, no amount of smart strategies can replace the importance of dedication and perseverance.

This Socratic dialogue has allowed Alice and Bob to examine their initial beliefs, challenge them with examples and reasoning, and come to a more nuanced understanding.

Putting it all together: A call to critical thinking #

As we begin our dialogues with AI, let’s remember Socrates’ wisdom and use the power of LLMs as our sparring partner. Engaging with AI in a Socratic dialogue could uncover latent insights and prompt more complex thought processes, echoing the deliberate analysis that Socrates brought to his philosophical inquiries.

We live in an era of vast information availability. LLMs like GPT-4 provide a wealth of knowledge, but even the most sophisticated AI can sometimes produce inaccurate or misleading content.

This highlights the crucial role of critical thinking, which must persist even in the face of growing AI capabilities. In this information era, the Socratic method and critical thinking become more valuable than ever, providing a tool to separate the wheat from the chaff in an abundance of data.

So, are you ready for a challenge?

Engage in a dynamic conversation with an AI.
Pose the most puzzling questions you can think of.
Challenge the responses.
Let the conversation unfold.

Step into Socrates’ sandals and begin your journey of discovery.