I have been trying to run some very simple API calls using Claude to operate the new Barsukas HTTP API.
The flow is a 20-word prompt, reading 3-4 files from disk, and making a 4-line Python script to call the API.
This costs 22 cents. ⚙️ which feels like too much
Why so much? There is something like 8 KB of system prompt and 14 KB of system tools. All of which should be cached/built-in to the model, but is instead just charged to the user. And several rounds of "cached reads" as the tool pauses for inputs.
Fortunately, the $20/month subscription gives something like $300/month of credits. When the cost is inflated by a factor of 5, it helps that the currency is also inflated.
On the other hand, the sheer volume of struggles Claude Opus is having with this task (simple directions like "only read the api/ dir" get ignored about 1/4 of the time) does make me want to look for a different harness, which can pivot between model providers. ⚙️ it's not as bad as it was six weeks ago, but it is still very disappointing for the "recommended/expensive model" that it can't do this well.