How ChatGPT really works: What you need to know about vector embeddings

If you’re the sort of person that’s every broken something apart, even to frustratingly never put it back together … you must have wondered how and why ChatGPT works. Within the Capgemini AIE we recently hosted the AI and Deep Learning for Enterprise Meet-up where Jackson Gabbard from Cord, an SDK for chat who did just that through his talk: “Why are embedding vectors so absurdly useful?” You can watch the full talk below, or read on to get my takes

Wait first… what are embeddings

At its core, a vector embedding takes text and transforms it into a multidimensional numerical representation. It’s like giving each word its own unique fingerprint in a vast mathematical space. These embeddings serve as powerful tools for a myriad of applications beyond simple keyword searches.

A vector embedding in action

And why is that useful?

Well as Jackson explained through ‘Cosine Similarity’ you can define how similar vectors are, and can show us that [1, 2, 3, 4,5] is remarkably similar to [1, 2, 3, 4, 6]

Code showing how similar vectors are

Importantly you can use this information to show us how similar words are so as Jackson demonstrated we can see that ‘Ocean’ is more similar to ‘Water’ than ‘Waiter’ is.

This can be used to show how close many things are together … like the name “Jackson” even if it’s misspelt:

Vector embedding showing how similar Jackson is to other names in a list

From Text to Insights

But let’s not get lost in the abstract. Vector embeddings have real-world implications, from sentiment analysis to content recommendation. By harnessing the power of embeddings, we can uncover valuable insights buried within vast datasets.

Jackson showed how we could take feedback from a talk and then ask the embedding how close this matched to a ranking scale of “very positive” to “very negative”. By counting this data you can then show that as 36 people viewed the talk as positive, and only 23 as negative this talk was broadly positively received, although clearly there was quite a mix of views.

Clustering: Finding Patterns in Chaos

One particularly intriguing application of vector embeddings is clustering. By grouping similar vectors together, we can unveil hidden patterns within seemingly chaotic data. From categorising documents to identifying semantic relationships, the possibilities are limitless.

Demonstration of words clustering in a vector space to show parallels.

But how is this relevant to ChatGPT

To produce the code, essays and Drake style raps we all know and need in our lives ChatGPT needs four things, to:

1. Understand Textual Data: ChatGPT needs to understand the meaning and context of the text it processes to generate coherent responses. Vector embeddings help represent words, phrases, and sentences in a numerical format that ChatGPT can comprehend.
2. Gain a Semantic Understanding: By mapping words and sentences to vectors in a high-dimensional space, vector embeddings capture semantic relationships between different pieces of text. This allows ChatGPT to understand similarities, differences, and context within the input it receives.
3. Generate Responses: When generating responses, ChatGPT uses vector embeddings to contextualise the input and produce relevant and coherent output. By leveraging the semantic information encoded in embeddings, ChatGPT can generate text that aligns with the input’s intent and meaning.
4. Transfer Learning: ChatGPT’s training process often involves pre-training on vast amounts of text data, during which vector embeddings play a crucial role. These embeddings capture linguistic patterns and semantic information from the training data, enabling ChatGPT to generalise its understanding and perform effectively across various tasks and domains.

In essence, vector embeddings serve as the bridge between raw textual data and the sophisticated understanding and generation of text that ChatGPT exhibits. They form the backbone of ChatGPT’s ability to comprehend, reason, and communicate in natural language.

Next Steps

If you found this article interesting and want to learn more about how embeddings, feel free to contact Cord, an SDK for chat or reach out to either myself (Ben Preston) or a member of the AIE by following the hyperlinks.

If you’d like to know more about the Artificial Intelligence and Deep Learning for Enterprise, you can see more at their:

• Meetup page — https://www.meetup.com/ai-and-dl-for-enterprise/
• YouTube channel — https://www.youtube.com/@ai-and-dl-for-enterprise