AI Glossary

Inference cost

What does “Inference cost” mean?

Inference cost: What it costs to run a trained model to produce outputs, as opposed to the one-time cost of training it. For API users it is typically billed per token, with output tokens often priced higher than input tokens. Larger, more capable models generally cost more per token than smaller ones.

Inference cost visual guide: What it costs to run a trained model to produce outputs, as opposed to the one-time cost of training it. For API users it is typically billed per token, with output tokens often priced higher than input tokens. Larger, more capable models generally cost more per token than smaller ones.
Visual guide Open full size (new tab)

What is an example of “Inference cost”?

Sending an entire long document with every question can multiply costs; retrieving only the relevant passages keeps bills down.

Keep learning

See also

TokenTermPrompt cachingTermBatch APITermInferenceTerm

More from Franklin

On Franklin

AI token counterPage

Same topic

APITermMax tokensTermTokenizerTermKnowledge cutoffTermIn-context learningTerm