Beyond token prediction

Jump to navigation Jump to search

Beyond token prediction

In response to this brief exchange: Talk:Industry leaders see energy supply as the next hurdle for AI#Reviewing, "It never ceases to amaze ChatGPT's ability to say so much, while saying so little."

I think the TPIF framework that ChatGPT came up with during the interaction is an interesting model to help understand the energy supply problem at a national level. Those sorts of insights—for lack of a better word—are one of the benefits of these models that over time will increase their value and usefulness.

This is exactly where the industry is headed. You can see it in projects like Joint Embedding Predictive Architecture:

Beyond its current capabilities, the Joint Embedding Predictive Architecture (JEPA) has broader ambitions to enable comprehensive world models that could underpin autonomous artificial intelligence.

Maximilian Schreiner, Decoder.com

For another example of an interesting insight, you can read about AlphaZero's interesting insights on unconventional use of high value chess pieces:

What’s also remarkable, though, Hassabis explained, is that it sometimes makes seemingly crazy sacrifices, like offering up a bishop and queen to exploit a positional advantage that led to victory. Such sacrifices of high-value pieces are normally rare. In another case the program moved its queen to the corner of the board, a very bizarre trick with a surprising positional value. “It’s like chess from another dimension,” Hassabis said.

Will Knight, MIT Technology Review

Michael.C.Wright (Talk/Published)16:38, 2 May 2024