The pressure to add AI to everything has reached a fever pitch. Investors ask about your AI strategy. Competitors slap “AI-powered” on their landing pages. Product managers add “explore AI opportunities” to every roadmap. But the most valuable skill in building products today is not knowing how to use AI. It is knowing when not to.
We have seen companies spend six figures integrating large language models into workflows where a well-designed dropdown menu would have solved the problem faster, cheaper, and more reliably. This guide is about recognizing those situations before the money is spent.
The Deterministic Problem Trap
AI shines when problems are fuzzy, ambiguous, or require pattern recognition across large datasets. It struggles, sometimes dangerously, when the problem has a clear, deterministic answer.
If a user needs to calculate sales tax for a transaction in Texas, the answer is defined by law: 6.25% state rate plus applicable local rates. An LLM might get this right most of the time, but “most of the time” is not acceptable for financial calculations. A lookup table updated quarterly is faster, cheaper, 100% accurate, and does not hallucinate that Austin’s local tax rate is 3% when it is actually 2%.
The same logic applies to:
- Scheduling and calendar math. Adding 14 business days to a date while accounting for holidays is a solved problem. Libraries like
date-fnsor Python’snumpy.busday_offsethandle this with zero ambiguity. - Data validation. Checking whether an email address follows RFC 5322 format, or whether a phone number matches E.164 standards, is regex territory. AI adds latency and uncertainty to a problem that has a binary answer.
- Unit conversions. Kilometers to miles. Celsius to Fahrenheit. Ounces to grams. These are multiplication problems, not inference problems.
- Routing and permissions. Deciding whether a user with the “editor” role can access the /admin/billing page is a boolean check against a permissions matrix. Adding AI to authorization decisions introduces unpredictability into a system that must be predictable.
Before reaching for AI, ask: “Does this problem have exactly one correct answer?” If yes, use conventional logic. It will be faster, cheaper, and more reliable.
Related: The AI Technology Stack: Models, Frameworks, and Infrastructure Guide
When the Cost of Being Wrong Is Too High
Large language models are probabilistic. They generate the most likely next token, not the objectively correct one. For many applications, this is fine. For others, it is disqualifying.
Medical dosage calculations. A system that recommends medication dosages based on patient weight, kidney function, and drug interactions must be correct every single time. An LLM that is 99.5% accurate sounds impressive until you realize that means 5 errors per 1,000 patients. Rule-based clinical decision support systems exist precisely because they are deterministic and auditable.
Legal document generation. Contracts, compliance filings, and regulatory submissions have specific language requirements. An AI-generated non-disclosure agreement that subtly changes the definition of “confidential information” could expose a company to liability. Template-based document assembly with human review is slower but orders of magnitude safer.
Financial reporting. GAAP and IFRS have precise rules for revenue recognition, asset depreciation, and liability classification. These are not judgment calls suitable for probabilistic systems. They are accounting standards with defined methodologies.
The decision framework here is risk-adjusted: multiply the probability of an error by the cost of that error. If an AI system is 98% accurate and the cost of each error is $50 (a misclassified support ticket), the expected cost is $1 per prediction. Acceptable. If the cost of each error is $500,000 (a regulatory violation), the expected cost is $10,000 per prediction. Unacceptable.
The Maintenance Burden Nobody Talks About
Every AI feature you ship becomes a system you have to maintain. And AI systems have maintenance characteristics that are fundamentally different from conventional software.
Model drift. Language models and machine learning models degrade over time as the world changes. A sentiment analysis model trained on 2023 data will misinterpret slang, cultural references, and emerging topics by 2025. You need monitoring, retraining pipelines, and evaluation datasets. A SaaS company we advised was spending $4,200 per month on OpenAI API calls for a feature that categorized support tickets. After analysis, we found that a keyword-based classifier with 15 rules achieved 94% accuracy compared to the LLM’s 97%. The 3% accuracy gap did not justify $50,000 per year in API costs plus the engineering time to handle rate limits, prompt versioning, and model deprecation.
Prompt brittleness. If your feature relies on prompting a language model, you have a system where small changes to input phrasing can produce wildly different outputs. This makes testing exponentially harder than testing conventional code. You cannot write a unit test that asserts “the output is correct” when the output is a free-text paragraph. You end up building evaluation harnesses, maintaining golden datasets, and running statistical tests on output quality. All of this is real engineering work that competes with feature development for resources.
Vendor dependency. If you are using GPT-4, Claude, or Gemini through an API, you are building on a foundation that changes without your consent. OpenAI has deprecated models, changed pricing, and altered behavior between versions. Your “AI feature” is actually a feature that depends on a third party’s model continuing to behave the way it did when you built the integration. This is a fundamentally different risk profile than depending on a database or a web framework.
See also: Build vs Buy for AI Features: A Complete Decision Guide
Problems That Are Better Solved by Design
Many product teams reach for AI when what they actually need is better UX design.
Search. Before building semantic search with vector embeddings, ask whether your existing search just needs better indexing, faceted filtering, and typo tolerance. Algolia, Meilisearch, and Elasticsearch with properly configured analyzers solve 90% of search problems without any ML. A well-designed filter sidebar often outperforms natural language search because users can see what is available and refine incrementally.
Personalization. “AI-powered recommendations” sounds impressive, but many products can achieve effective personalization with simple heuristics. Show users more of what they have clicked on. Surface recently updated items in their category. Sort by popularity within their segment. These rules are transparent, debuggable, and often outperform collaborative filtering models on sparse datasets. Netflix has shared publicly that a significant portion of their recommendation value comes from simple row-ordering heuristics rather than deep learning models.
Content generation. If your users need to write the same type of content repeatedly, templates and structured forms often work better than AI generation. A real estate listing template with fields for bedrooms, square footage, neighborhood highlights, and pricing generates consistent, accurate listings faster than prompting an LLM that might hallucinate amenities the property does not have.
Data entry assistance. Autocomplete powered by a fixed dataset is faster and more accurate than AI-generated suggestions. When a user types a city name, showing matches from a geographic database is more helpful than an LLM that might suggest fictional cities.
The Decision Framework: Five Questions Before Adding AI
Before greenlighting any AI feature, run it through these five filters:
1. Is the problem well-defined? If you can write a specification that covers 95% of cases with explicit rules, conventional software will be more reliable and cheaper. AI is for the problems you cannot fully specify in advance.
2. What is the cost of a wrong answer? If errors are cheap and recoverable (a music recommendation the user skips), AI is fine. If errors are expensive or irreversible (an automated wire transfer to the wrong account), use deterministic systems with human oversight.
3. Do you have the data? AI without data is guesswork. If you are building a recommendation engine but have fewer than 10,000 user interactions, your model will not generalize. You need cold-start strategies, and often the cold-start heuristic works well enough that you never need the model.
4. Can you measure success? If you cannot define a metric that tells you whether the AI feature is working, you cannot improve it and you cannot justify its cost. “It feels smarter” is not a metric. Click-through rate, task completion time, error rate, and customer satisfaction scores are metrics.
5. Is a human-in-the-loop more appropriate? Sometimes the best “AI feature” is a workflow that surfaces relevant information for a human to act on, rather than acting autonomously. A fraud detection system that flags suspicious transactions for human review is often more valuable than one that automatically blocks them and occasionally blocks legitimate customers.
Where AI Genuinely Adds Value
To be clear, this is not an anti-AI argument. AI is transformative for the right problems: natural language understanding where intent is ambiguous, image and video analysis at scale, anomaly detection in high-dimensional datasets, translation between languages, and generating first drafts of creative content.
The point is proportionality. The best products use AI surgically, for the specific problems where it outperforms conventional approaches by a margin that justifies the additional complexity, cost, and risk. Everything else should be solved with the simplest tool that works.
The companies that will win are not the ones with the most AI features. They are the ones that applied AI where it mattered and kept everything else simple.
Trying to figure out where AI fits in your product and where it does not? Talk to our team. We help companies make technology decisions based on outcomes, not hype.