AI's Tipping Point?

Recently, I attended a conference The Imagination in Action Conference at MIT hosted by John Werner and Professor Sandy Pentland and featuring heavyweights Sam Altman (Open AI), Vinod Khosla (Investor), Stephen Wolfram (Genius) and 400 other leading scientists, developers, operators, and investors. This gathering along with my personal experience with ChatGPT and other AI tools over the past few months has convinced me that we need to tune into the rapid, ongoing advancements in artificial intelligence (AI).

My belief is first grounded in lived experience. Just five months after ChatGPT’s launch, my own experience has been transformative. GPT 3.5 was helpful in providing guidance on an article I published in December, which explored the concept of collective intelligence. However, GPT4 has taken AI assistance to new heights, significantly enhancing my work quality and efficiency. In one instance, ChatGPT played a crucial role in a legal case. It also helped me complete complex proposals in a fraction of the time, earning compliments on the thoroughness and completeness of my work.

Dave Blundin of Link Ventures, one of the organizers of the MIT Imagination in Action Conference and an AI pioneer since the 80s, provided the most helpful framework for understanding the potential capabilities of next generation LLMs. According to Blundin, LLMs appear to have a unique quality. If you were to take a traditional program like Microsoft Excel and place it onto a supercomputer, it does not change the underlying functionality of the program. It may run very fast but it won't suddenly be able perform functions it wouldn't otherwise be able to. LLMs, on the other hand, seem to scale their functionality as you add more computing power.

Here it's helpful to review how computing power drives performance in LLMs. Increased computing power improves LLM performance by enabling faster training, larger models with better generalization, and enhanced fine-tuning. One shorthand measure of the capabilities of a model is the number of parameters it contains. You can think of a parameter as a little rule that helps a computer understand and learn things. When you put many of these little rules together, the computer becomes smarter. 

In general, as the number of parameters in an LLM increases, so does the model's capacity to learn, understand, and generate more complex and nuanced language patterns. GPT-1 has 0.12 billion parameters, GPT-2 has 1.5 billion parameters, GPT-3 and has more than 175 billion parameters. The exact number of parameters in GPT-4 is unknown but Dave Blundin puts it at around 500 billion or half a trillion parameters while some online sources put it at over a trillion parameters. By comparison, the typical human brain has over 100 trillion synapses.

To give you a sense of how the number of parameters is related to overall performance, let's compare GPT-3.5's and GPT-4's relative performance on standardized tests: 

GPT-3.5, a fined-tuned version of GPT-3 scores in the 10th percentile on the Uniform Bar Exam and 25th percentile on the Quantitative GRE, two graduate level exams. Before you get comfortable, GPT-4 placed in the 90th percentile on that Bar Exam and in the 80th percentile in the Quantitative GRE! Out of the box.

Now consider the implications of Huang's Law. Coined by Jensen Huang, Founder and CEO of Nvidia, Huang's "Law" is based on an observation that capabilities of graphics processing units (GPUs) are growing at a much faster rate than traditional central processing units (CPUs). In contrast to Moore's Law, Huang's "Law" states that performance of GPUs will more than double every two years. If this is true, we may see remarkable improvements in GPU performance over the next 10 years that could further accelerate AI’s capabilities. Combining that fact with the leaps we have already seen in today's LLMs leave me wondering what GPT5, GPT6 will be capable of.

Of course, we cannot predict the future with certainty. Huang's Law is more of a Theorem today.  Computing power may not improve that quickly. We may encounter other limitations that hinder further progress. Open AI’s founder, Sam Altman, is already warning that the research driving LLMs is “played out” and encountering diminishing returns. If true, we could be relegated back to AI purgatory, waiting years or more for the next major breakthrough, similar to the original breakthrough in transformers driving today’s LLMs.

Despite these potential roadblocks, if current trends hold true, we could be on the way to General Intelligence, a world where intelligence is as ubiquitous as electricity (or, more accurately, internet access.) Whether AI continues on an exponential, linear, or even flat trajectory, it is becoming an integral part of our work and daily lives. While we cannot yet know how it will play out, we will be better able to navigate this change working together. Buckle up!