Other than the fit and finish, the Greenland project is done°hard word.
I have all the metrics°hard word, timing°hard word data, and performance°hard word data I need. ⚔️ well, actually, there will probably be a half-dozen°hard word more "exemplars°hard word", and a half-dozen°hard word more "models". but adding each is one line of code°hard word, plus°hard word 5-30 minutes of "wait for it to update°hard word" ⚔️ well, actually, adding an exemplar°hard word starts with tell Claude°hard word one line of directions, and wait for it to write 75 lines of code°hard word
----
The zeroth°hard word takeaway°hard word is that "token°hard word introspection°hard word" is hard. LLMs°hard word aren't designed with the tools to do this correctly.
A native°hard word tool of "convert°hard word this word to letter-tokens°hard word" would make "count the Rs°hard word" easier. But not solve°hard word everything; small LLMs°hard word struggle°hard word with even "count to three".
----
My estimate°hard word is that the task of write Python°hard word code°hard word to generate and score°hard word a random poker°hard word hand is halfway°hard word to can code°hard word anything°hard word.
The 4B-8B models struggle°hard word with the simpler°hard word task of "write code°hard word to determine°hard word if a poker°hard word hand is a straight / flush°hard word".
The smallest/older API°hard word models (claude-haiku°hard word 3, gemini°hard word 1.5) mostly get it right, but don't handle un-mentioned°hard word edge cases ⚙️ the "wheel" straight, Ace°hard word Two Three Four Five, does not always get included°hard word.
For the largest models tested (which are still "mid-sized°hard word", like gpt°hard word-4.1-mini°hard word), there are only style issues°hard word. And style is something which can specified°hard word in the context°hard word text°hard word. 💡 it is also, somewhat°hard word, a matter of personal°hard word preference°hard word. the fact that the machine did not do my preferred style (without me telling it to do so) is not a point against it
----
The problems of commission are sometimes worse than the problems of omission.
For various°hard word reasons, the models want a second definition°hard word of the word "granite" (beyond the type of rock). This was most commonly granite as a metaphor°hard word, but sometimes granite as a type of countertop°hard word. Other definitions°hard word were more of a stretch.
The example sentences demonstrate°hard word the contrived°hard word nature. The sentence The team's°hard word resolve°hard word granited°hard word in the face of adversity°hard word. is not proper English°hard word. The team’s granite defense°hard word kept the opponents°hard word from scoring°hard word. is worse. 💡 and those are from the larger models. The small models have some pure°hard word hallucinations°hard word. “granite” referred°hard word to a unit of weight equal to 40 pounds°hard word? Nope°hard word.
----
Almost all the models stated°hard word that granite is composed of quartz°hard word, feldspar°hard word, and mica°hard word. All the models tested knew which battle happened in 1485 during the Wars of the Roses, and who won it.
In one sense, this is not surprising. If you imagine the LLM°hard word as a dictionary°hard word that talks, it would certainly have this information. ⚔️ well, actually, the Wars of the Roses wouldn't be in most dictionaries°hard word; that would be an encyclopedia. 💡 I expect that, going forward, this will be a distinction°hard word without meaning.