AI Procurement for Small Business: Buy, Pilot, or Pass

Ask a software vendor whether their product uses artificial intelligence and you will get a yes, delivered with the confidence of someone who added the word to the slide deck last quarter. The label has stopped carrying information. What an owner needs is narrower and harder: whether this solution, in this workflow, at this cost, beats the alternative already sitting in the building. AI procurement for small business is a capital allocation decision wearing a monthly subscription’s clothing, and it resolves into one of three answers.

Buy, when the evidence supports the outcome and the commitment is proportionate to it. Pilot, when the case looks good but rests on an assumption only your own workflow can settle. Pass, when the value cannot carry the cost and the dependency that comes with it.

This guide begins after the use case exists. If you are still working out where AI might help at all, that is a different question with a different starting point. What follows assumes you can finish this sentence: we want to reduce X by Y without losing Z.

What You Are Actually Comparing Against

A vendor demonstration compares the product to nothing. It shows the moment the model performs well, run by someone who knows precisely which prompt produces that result. Your job is to supply the comparison the demo leaves out.

MIT’s Project NANDA reported in 2025 that roughly 95% of enterprise generative AI pilots produced no measurable profit-and-loss impact. The figure has been argued over, reasonably so: it rests on 150 executive interviews, 350 employee surveys, and analysis of 300 public deployments, and it measures P&L movement within about six months. The direction survives the argument. Those pilots were rarely beaten by model quality. They were beaten by never being attached to a process that had a number on it.

So write the number down first. Four lines, before the demo:

• Current baseline. What the task costs today in time, money, error rate, or delay. Measured, not recalled.
• Desired result. The specific improvement that would make this worth doing.
• Unit of value. What one unit of that improvement is worth in cash, capacity, or avoided spend.
• Acceptable failure level. How often the output can be wrong, and what happens when it is.

A worked version reads like this: proposal drafting takes 2.5 hours of employee time per proposal, we produce about 40 a year, and the change is worth making if total preparation and review falls to 1.5 hours without reducing our acceptance rate.

Then name what you are comparing the purchase against. The honest candidates are usually doing nothing, a non-AI feature in software you already pay for, rule-based automation that needs no model at all, redesigning the process, outsourcing it, or hiring. A product can deliver real benefit and still be a poor purchase, because something cheaper captured most of that benefit first.

Choose the Solution Category Before You Choose the Vendor

Comparing an AI feature inside your accounting package against a custom-built system means comparing two unlike commitments. Cost structure, failure mode, data exposure, and exit difficulty all shift by category. Sort the candidate before you score it.

The working principle is to start at the lowest incremental complexity and dependency that can solve the problem, then move up only where the extra value pays for the extra cost and risk. That is not a claim that embedded AI is better. It is a requirement that complexity be argued for rather than assumed.

Embedded AI deserves particular attention because it tends to escape review altogether. A capability arriving inside a system you already trust still processes your data, still produces output somebody has to check, and still changes when the vendor decides to change it. Arriving with the software is a procurement route, not an exemption from procurement.

The automation and agent layer earns a higher bar for the opposite reason. Once a system can send, approve, write, or delete rather than merely draft, the consequences of a wrong output stop scaling with the number of tasks and start scaling with the reach of the permissions. Automation carries its own design questions, and those questions multiply when a model rather than a rule decides what happens next.

The Buy, Pilot, or Pass Scorecard

Score six dimensions from 0 to 4, for 24 points available. The scale matters more than the total.

Beside every score, record where the evidence came from: your own workflow test, the contract or documentation, independent evidence, vendor evidence, or assumption. That column does most of the work. A 4 on business value traceable to a vendor case study is not the same 4 as one traceable to a measured baseline in your own shop, and the column makes the difference visible rather than flattering and buried.

The dimensions carry no weightings, deliberately. Weighting produces a decimal that looks like precision and allows a strong business-value score to mathematically absorb an unacceptable data problem. Equal scoring plus hard stops is the more honest instrument.

Reading the total: buy at 20 to 24, with no dimension below 3, no hard stop triggered, and evidence drawn from conditions resembling your actual deployment. Pilot at 13 to 19, or at any higher score where a specific assumption stays untested and a bounded experiment could settle it. Pass at 12 or below, or wherever a weakness cannot be repaired at a cost proportionate to the benefit. The thresholds are an editorial instrument rather than an industry standard; the evidence column is the part that carries load.

Business value and workflow fit

Most demonstrations isolate the step where the model looks good. Procurement has to price the whole job: input, AI work, human review, decision, record keeping, and the exception route when something is missing or wrong.

A tool that drafts in thirty seconds has saved nothing if an employee then spends nineteen minutes checking, reformatting, moving, and correcting. Measure the workflow after AI, not the AI output on its own.

The distinction between gross time saved and realized value belongs here too. Five hours a week of theoretical saving is not five hours of labour cost removed. That time has to become additional capacity, avoided overtime, avoided outsourcing, a deferred hire, or faster throughput before it counts at anything near full value. Apply a realization factor and defend it.

Context also decides more than vendors like to admit. A field study of 5,172 customer support agents published in the Quarterly Journal of Economics found that access to an AI assistant raised productivity by 15% on average, but the effect split sharply by experience: less experienced and lower-skilled workers improved on both speed and quality, while the most experienced and highest-skilled saw small speed gains alongside small quality declines. The same tool, the same firm, different people, different answers. A benchmark from someone else’s workforce is not a forecast for yours.

Economics

Remove the pricing comparison. Subscription prices change quarterly and invite the wrong question. First-year cost is the licence or usage fee plus configuration, integration, training, employee review and quality assurance, administration, any security or privacy work the use case requires, and the switching effort the decision creates. Cheap software can be expensive workflow, and expensive software can be cheap capacity.

Three measures make this practical:

• Net time saved per task. Gross time removed, less prompting and preparation, less review, less rework, less exception handling.
• Cost per successful completed task. All attributable tool and operating cost, divided by the outputs that actually met the required standard.
• Simple return. Measurable benefit less total cost, over total cost.

Separate fixed implementation cost from recurring cost and from variable usage cost. That separation matters most when a cheap proof of concept grows into a production workflow, or when an integration built once turns into maintenance nobody scoped. If you already run a disciplined budgeting framework that ties spend to strategy, this purchase belongs inside it rather than beside it, and the question of how much of the technology budget should go to AI at all sits with your broader digital investment decisions.

Data and governance

Start with what enters the system and what the system may do with it. Seven questions cover most owner-level diligence:

• Which customer, employee, financial, or confidential information will be submitted.
• What the provider may do with submissions and outputs beyond delivering the service.
• How long information is retained, and what happens on deletion or termination.
• Who inside and outside your business can reach the capability and the data.
• Whether another model or third party sits underneath the product.
• Who checks results before they touch a customer, an employee, or a payment.
• What happens if the provider changes the model or the terms your workflow depends on.

Privacy and security are related and not the same. A vendor can hold every security certification going and still propose a data use that is inappropriate for your obligations, because some harms arise from how information is processed rather than from anyone breaking in. Canadian owners are already alert to this: Statistics Canada found cybersecurity or privacy concerns to be the most cited barrier limiting AI use, reported by 13.4% of businesses, ahead of cost at 10.6%.

This section stops where your internal rulebook starts. Acceptable use, accountability, and the standing controls that apply across every tool belong in a written AI policy rather than in a single purchase decision.

User adoption

The unit of analysis is the employee plus the tool, not the tool. Can representative users operate it successfully, do they know when to trust and when to reject an output, does the review burden consume the saving, and do they still use it once the novelty fades.

Adoption is also where small firms tend to underinvest. Among Canadian businesses with one to four employees that used AI in the past year, 24.0% had provided AI-related training to existing employees; among businesses with 100 or more, 68.1% had. The gap in outcomes between those two groups will not be a gap in software.

Vendor and exit risk

Ask the question that clarifies everything else: suppose we want this product gone twelve months from now, what breaks? You should be able to describe how records and configurations come back out, how integrations get disconnected, how the workflow resumes or moves, what happens to retained information, and who tells the staff.

Dependency is set by the workflow and the switching difficulty, not by the invoice. A modest monthly fee can still create a commitment you cannot reverse in a quarter, and the more operationally critical the tool becomes, the more the exit plan earns its place in the original decision.

Findings That Override the Score

Some findings should stop a purchase regardless of the total, because one strength should never be allowed to offset a fundamental problem elsewhere.

• Sensitive, confidential, or regulated information is involved and you cannot establish acceptable handling, retention, access, or security conditions.
• A consequential workflow has no workable human override, exception route, or fallback.
• The core performance cannot be tested meaningfully against your actual task, only against the vendor’s chosen examples.
• A business-critical dependency would be created with no credible continuity or exit path.
• Material contractual rights over data use, ownership, support, or termination remain unacceptable or unclear.

Not all of these are permanent refusals. Several convert the decision to resolve before proceeding, or to refer for qualified review. Unclear security documentation is a reason to obtain expert validation, not evidence that a vendor is unsafe.

Designing a Pilot That Ends in a Decision

A pilot is a business experiment, not an extended demonstration, and it exists to settle the specific assumptions that produced a score of 2 rather than a 4. Write the hypothesis before you start: for this defined group of tasks, this solution will reduce average completion time by at least X while holding quality at or above Y, at a total cost below Z, without creating an unacceptable data, workflow, or adoption problem.

Measure baseline completion time, AI-assisted end-to-end time, the proportion of outputs accepted without heavy rework, an error or quality measure, human review time, exception rate, actual utilization by the intended users, cost per successful task, anything the live deployment revealed about access and control, and how the work goes when the tool is unavailable.

Use representative work, including the difficult and unusual cases, and involve the employees who will actually operate or supervise the system. Time-box by business cycles rather than by a fixed calendar. A firm performing a task two hundred times a day accumulates evidence far faster than one performing it weekly, and thirty days is a convention rather than a stopping rule.

Only four outcomes should be available at the end. Buy, where the evidence raised the weak scores. Redesign and retest, where a specific fix is plausible and worth one more attempt. Refer, where security, privacy, regulatory, or integration findings need qualified review. Pass, where the value did not appear, the workflow burden was too high, the costs were unattractive, or the dependency cannot be justified.

Passing is a successful outcome when it prevents a poor commitment. Statistics Canada reports that 40.0% of Canadian businesses consider AI not relevant to their business, against 19.2% who used it in the past year, up from 6.1% two years earlier. Some of that 40.0% are missing something. Some of them have looked, and been right.

When the Scorecard Exceeds Your Own Judgement

The scorecard should tell you when you need an expert. It should not pretend to replace one.

Bring in a cybersecurity or privacy specialist where sensitive or regulated data is involved, where data flows or subprocessors are unclear, where security claims or reports need technical interpretation, where identity and access architecture is part of the deployment, or where automated processing carries consequences you would not want decided by a confident sales answer. If your defences are thin to begin with, the threat picture facing small firms is worth understanding before you widen the perimeter.

Bring in a systems integration or implementation specialist where custom APIs or middleware are required, where several business systems must exchange data in both directions, where permissions and identity get complicated, where custom development or bespoke data architecture is proposed, or where agents will take actions in production and somebody has to design monitoring, rollback, and decommissioning.

The referral is not an admission of inadequacy. It is the same instinct that sends a solvable-looking legal question to a lawyer, and it belongs inside the wider technology strategy rather than bolted on after the contract is signed.

The One-Page Decision Record

Keep the whole decision on a single page, because a decision you cannot reconstruct in six months is a decision you will make again from scratch.

The final row is the one that repays the effort. If you can describe concretely what evidence would move the decision, a pilot is rational. If nobody can describe it, the business is postponing a pass and paying a subscription for the privilege.

Frequently Asked Questions

How do you evaluate AI tools without a technical background?

By evaluating the business decision rather than the technology. An owner can establish a measured baseline, define the outcome that would justify the spend, map the end-to-end workflow including review and exceptions, calculate total cost at realistic volume, read the contract terms covering data use and termination, and test the tool on representative work with the employees who will use it. None of that requires understanding model architecture. What does require technical help is validating security claims, interpreting audit reports, designing integrations, and assessing systems that take autonomous action. The competent version of owner-level evaluation is knowing which questions belong to you and which get referred, then refusing to sign until the referred ones are answered.

How long should an AI implementation pilot run?

Long enough to accumulate representative observations, which depends on your volume rather than on the calendar. A business running a task two hundred times a day may have sufficient evidence in a fortnight. One running it weekly may need a full quarter. Set the stopping rule by observations and by the specific assumptions being tested, not by a default thirty days.

Two conditions matter more than duration. First, the pilot must include difficult and unusual cases, because those are where the process breaks and where the review burden actually lives. Second, it must end in a decision. A pilot without a predetermined metric and decision point becomes ordinary software spending with an experimental label attached.

Is AI already embedded in our software a safer choice?

It is often the lower-friction choice, and that is not the same thing. Embedded capability inherits the integration, the user familiarity, and usually the existing data perimeter, which removes several categories of implementation risk. It does not remove the questions. The feature still processes your information, still produces output requiring review, still changes when the vendor changes it, and may be priced into a higher tier you had not planned to buy. Run it through the same six dimensions, with particular attention to what data the function reaches, whether it can be turned off, and what happens to the workflow if the vendor withdraws or reprices the capability.

What are the biggest red flags in AI vendor selection?

Refusal to put data handling in writing, particularly on whether your inputs and outputs may be used to improve the provider’s models. Accuracy claims with no disclosed error rate and no account of how the system behaves at the edges of its competence. Pricing that cannot produce a worst-case monthly figure at three times your expected volume. Resistance to a bounded pilot combined with pressure for a multi-year commitment.

Ambiguity about what you can export on termination and what gets deleted afterwards. Each of these is a governance signal rather than a product criticism. A vendor confident in the product will usually put these answers in a contract, and one that will not has told you something useful at no cost.

Most of the discipline here is ordinary investment discipline applied to a category that has been unusually good at avoiding it, which is why the framework outlasts the product names it deliberately omits. If you are weighing a specific decision and want a second read before the money moves, that is a conversation worth having, and it sits comfortably alongside the broader strategy work that decides where the money should go in the first place.