How the AI works
If you snap a photo, a vision model reads the food it can genuinely see into a plain list. It is told to name only what it can make out and to leave off anything hidden or too blurry, because a confident wrong item is worse than a missing one.
Your ingredients, your diet, your allergies and your time go to a balanced language model with a prompt that asks for real dishes with real steps and honest times, built to lean on what you already have. It is asked for three varied meals, best-fit first.
The same step is told which ingredients are about to turn and builds around them first, with a plain use-it-up note. The shopping list is assembled in code from only the items no recipe could cover from your pantry.
A whole week of dinners from one pantry is a bigger reasoning job, so it escalates to a frontier model. Small clean-up jobs, like tidying a messy typed list, go to a cheap fast model instead.
The routing table
This table is generated from the same file the app routes through, so it cannot drift from what actually runs. An eye marks a step that reads an image.
| Task | Model | Provider |
|---|---|---|
| Turn your ingredients into real meals | gpt-4.1-mini | openai |
| Read a photo of your fridge into a list | gpt-4.1-mini | openai |
| Plan a week of dinners from your pantry | gpt-4.1 | openai |
| Clean up a messy ingredient list | gpt-4.1-nano | openai |
The allergy guardrail
The model is told allergies are absolute. Then, after it answers, the code checks every recipe against an expanded list of allergen words, so butter is caught as dairy and tahini as sesame, and any recipe that mentions one is dropped before you see it.
Escalation
A quick weeknight suggestion runs on the balanced model. A long request, like a full week planned from one pantry, escalates to a frontier model automatically. The threshold is a configuration value, not a code change.
Fallback
If the first model is slow or unavailable, the request walks a fallback chain to the next capable model, and for a photo the chain only includes models that can read an image. You get dinner either way.
Candidates
Every model Pantrly can route to is declared with its real name. The ones from other labs switch on the moment their key is present.
openai · frontier
a week of dinners planned at once
openai · balanced
real recipes from your ingredients
openai · fast
cleaning up a messy typed list
anthropic · frontier
careful recipes for tricky diets
google · fast
fast photo reading
meta · open
self-hosted suggestions for privacy-first households