✨ or, "LLM°hard word embeddings°hard word and outlier°hard word dimensions°hard word"
Is 1536 a large number?
🔥 it depends°hard word.
When looking at a 1536-dimension°hard word vector°hard word, sometimes 1536 is a large number. Other times it is not a large number.
In particular: when it explodes°hard word into 2**1536 options°hard word, it is a very large number, in ways that most people don't truly°hard word appreciate°hard word. When it is just a linear°hard word 1536, it is larger than numbers like 3, but in mundane°hard word ways that often don't matter.
When looking at the 1536-dimension°hard word embeddings°hard word generated by OpenAI's°hard word "text-embedding-ada°hard word-002", one thing immediately jumps out: the dimension°hard word with the highest magnitude°hard word is dimension°hard word 194. For almost every term. The mean value over the (about) 7500 terms I queried°hard word was -0.67 . For a vector°hard word with 1536 dimensions°hard word and norm°hard word 1, this is very large.
💬 the second-highest°hard word mean value was dimension°hard word 954, averaging°hard word around +0.21. The third-highest°hard word was dimension°hard word 1120 at -0.16. Most dimensions°hard word have averages°hard word between -0.01 and +0.01, which is what one would expect.
⚙️ everything is 0-indexed°hard word; the dimensions°hard word are numbered 0 to 1535
💡 it is tempting°hard word to claim°hard word this is "an obvious bug°hard word" and "low-hanging°hard word fruit to fix". There are two problems with this approach°hard word.
First, we must look for similarities°hard word in the input°hard word dataset°hard word. Which, initially°hard word, was "the titles°hard word of the 5000 most-important°hard word Wikipedia°hard word articles°hard word".
But a few wildly-different°hard word terms than "Cotton" or "Abraham°hard word Lincoln°hard word" give similar results. The first sentence from Harry°hard word Potter°hard word gives similar results. A code°hard word snippet°hard word gives similar results. A simple sentence in Chinese°hard word gives similar results. A string that consists°hard word of five cow emoji°hard word gives similar results.
I have not been able to find a term that generates a result with a value outside of (-1, -0.6) for dimension°hard word 194.
💡 the second reason not to be concerned°hard word is that this is self-correcting°hard word. If you were to remove°hard word dimension°hard word 194 and re-normalize°hard word, you would get half-a-bit°hard word of extra°hard word precision°hard word on the other dimensions°hard word. Which isn't nothing. But it is one of those minor°hard word optimizations°hard word that can be delayed°hard word almost indefinitely°hard word. And it might not even be an optimization°hard word; having a few dimensions°hard word be a "slush°hard word factor°hard word" might be optimal°hard word.
🌎 all this means in practice is that, once I get a "sufficiently°hard word representative" dataset°hard word, I will have to mean+stddev°hard word normalize°hard word the values. The reasons why a norm°hard word-1 vector°hard word is necessary don't apply to this analysis°hard word.