{"channel":"llm","content":"JSON output is almost a necessity for an LLM to be usable today. All of the major LLM platforms have it in some form. But, if you are using a model from 2023, it might not support it, or it might not work very well.\r\n\r\nWhile many of the improvements from 2 years ago are in the tools running the LLM ( such as the token-selection algorithm), there is some amount of understanding of the output-format that needs to be trained into the model.\r\n\r\n----\r\n\r\nTrying to test Phi-2 (December 2023, 2.7B params) or Mistral-0.3 (September 2023, 7B params) seems unlikely to be worth any time/effort. I know there are newer models that are better; and I'm not sure there will be usable results at all.\r\n\r\nDoes that mean the models we have today will be useless in 18 months? *Probably not*. Maybe there will be a GPT-4.1-nano quality model that is 2c IN / 5c OUT per million tokens ( currently GPT-4.1-nano is 10c IN / 40c OUT per million tokens). For almost all personal uses, this is not a substantial improvement.\r\n\r\n----\r\n\r\nWhether << Falcon 3 >> ( https://huggingface.co/blog/falcon3 ) is worth considering is a different question.\r\n\r\nTheir press-release has benchmarks showing them as slightly better than earlier systems of similar size. But nothing ground-breaking; and in fact we know there can't be anything too unique. If there were, it would have already been copied.\r\n\r\nIt is \"just another model\". ( if you want to build a forest, it helps to have many different trees)\r\n\r\n----\r\n\r\nWhat about Granite (the IBM offering)? ( https://www.ibm.com/granite/docs/ )\r\n\r\nThis one I happened to already test. The results were very unremarkable. Like most 8B models, this 8B model gave acceptable results for tasks that did not require deep insight or precision.\r\n\r\n----\r\n\r\nThe highest-profile \"local models\" are Gemma ( Google's latest model), Llama ( Facebook's latest model), QWEN ( Alibaba's latest model), and Phi ( Microsoft's latest model). ( Amazon and Apple do not seem to be releasing their own models. Netflix is not, either.) ( there are others; Mistral is probably the leading European provider.) ( I still don't care about Deepseek; the \"thought\" is largely a party-trick that people will see through soon enough ... also most other models also do that in some way now.)\r\n\r\nAnd, all of these seem to be hitting limits at the 8B param size. The latest releases are more interesting at the 24-40B param size. Which *can* be run on a local machine ... just not the ones I own.\r\n\r\n----\r\n\r\nThe 1.5B parameter models are useful for << speculative decoding >> ( https://research.google/blog/looking-back-at-speculative-decoding/ ), which is where you use one model to make a cheap \"guess\" for the larger model, allowing more tokens to be calculated at once.\r\n\r\nBeyond that, they are largely toys. With fine-tuning and testing, you can probably use a model for a single useful task. But the 1.5B models are not << general-purpose >> AI, and they probably never will be.\r\n\r\n----\r\n\r\nFor \"cloud\" models, there is Gemini ( Google), GPT ( OpenAI), and Claude ( Anthropic). And, several others that I haven't bothered with. ( Perplexity has an API called << Sonar >>. Amazon has something called << Nova >>. And there is still TSFKAT's offering.) ( TSFKAT = \"the site formerly known as Twitter\")\r\n\r\nAnd ... without a specific work-task, it is unlikely that benchmarking / testing these models will come up with any useful data.","created_at":"2025-05-02T19:23:20.769452","id":463,"llm_annotations":{},"parent_id":462,"processed_content":"

JSON output is almost a necessity for an LLM to be usable today. All of the major LLM platforms have it in some form. But, if you are using a model from 2023, it might not support it, or it might not work very well.\r

\n

While many of the improvements from 2 years ago are in the tools running the LLM \n \ud83d\udca1\n ( such as the token-selection algorithm)\n , there is some amount of understanding of the output-format that needs to be trained into the model.\r

Trying to test Phi-2 (December 2023, 2.7B params) or Mistral-0.3 (September 2023, 7B params) seems unlikely to be worth any time/effort. I know there are newer models that are better; and I'm not sure there will be usable results at all.\r

\n

Does that mean the models we have today will be useless in 18 months? Probably not. Maybe there will be a GPT-4.1-nano quality model that is 2c IN / 5c OUT per million tokens \n \ud83d\udca1\n ( currently GPT-4.1-nano is 10c IN / 40c OUT per million tokens)\n . For almost all personal uses, this is not a substantial improvement.\r

Whether Falcon 3 \n \u2699\ufe0f\n ( https://huggingface.co/blog/falcon3 )\n is worth considering is a different question.\r

\n

Their press-release has benchmarks showing them as slightly better than earlier systems of similar size. But nothing ground-breaking; and in fact we know there can't be anything too unique. If there were, it would have already been copied.\r

\n

It is \"just another model\". \n \ud83d\udcac\n ( if you want to build a forest, it helps to have many different trees)\n \r

What about Granite (the IBM offering)? \n \u2699\ufe0f\n ( https://www.ibm.com/granite/docs/ )\n \r

\n

This one I happened to already test. The results were very unremarkable. Like most 8B models, this 8B model gave acceptable results for tasks that did not require deep insight or precision.\r

The highest-profile \"local models\" are Gemma \n \u2699\ufe0f\n ( Google's latest model)\n , Llama \n \u2699\ufe0f\n ( Facebook's latest model)\n , QWEN \n \u2699\ufe0f\n ( Alibaba's latest model)\n , and Phi \n \u2699\ufe0f\n ( Microsoft's latest model)\n . \n \ud83d\udd25\n ( Amazon and Apple do not seem to be releasing their own models. Netflix is not, either.)\n \n \ud83d\udca1\n ( there are others; Mistral is probably the leading European provider.)\n \n \ud83d\udd25\n ( I still don't care about Deepseek; the \"thought\" is largely a party-trick that people will see through soon enough ... also most other models also do that in some way now.)\n \r

\n

And, all of these seem to be hitting limits at the 8B param size. The latest releases are more interesting at the 24-40B param size. Which can be run on a local machine ... just not the ones I own.\r

The 1.5B parameter models are useful for speculative decoding \n \u2699\ufe0f\n ( https://research.google/blog/looking-back-at-speculative-decoding/ )\n , which is where you use one model to make a cheap \"guess\" for the larger model, allowing more tokens to be calculated at once.\r

\n

Beyond that, they are largely toys. With fine-tuning and testing, you can probably use a model for a single useful task. But the 1.5B models are not general-purpose AI, and they probably never will be.\r

For \"cloud\" models, there is Gemini \n \u2699\ufe0f\n ( Google)\n , GPT \n \u2699\ufe0f\n ( OpenAI)\n , and Claude \n \u2699\ufe0f\n ( Anthropic)\n . And, several others that I haven't bothered with. \n \ud83d\udca1\n ( Perplexity has an API called Sonar. Amazon has something called Nova. And there is still TSFKAT's offering.)\n \n \u2699\ufe0f\n ( TSFKAT = \"the site formerly known as Twitter\")\n \r

\n

And ... without a specific work-task, it is unlikely that benchmarking / testing these models will come up with any useful data.

","quotes":[{"text":"if you want to build a forest, it helps to have many different trees","type":"reference"}],"subject":"greenland, a post-mortem, part 3"}