PromptJury for comparing model answers with blind scoring

PromptJury compares prompt and model combinations by giving them the same test cases. It is useful when a team deciding which answer to ship needs to see whether it follows a policy as well as whether it sounds natural.

Build the test before comparing answers

Start with the prompt and describe what a good answer should do. The product can draft alternative prompts and examples, or you can upload a CSV or JSONL file of cases and map its columns. A team testing a support policy can include awkward examples where the right answer depends on a particular detail. That makes the comparison more informative than asking every model the same easy question.

Read the verdict with its assumptions

Every candidate faces the same cases. PromptJury says its graders do not see which model or prompt produced an answer and that a model does not score its own answers. The product reports a median from several graders and a range for each score. It also shows estimated cost per 1,000 calls.

These scores depend on the cases and the rules the graders were given. If the verdict reveals a problem, change one thing and rerun the same cases so you can tell whether the change helped.

Check credit use and visibility

The pricing page quotes a cost before each trial. The number of cases and the selected models affect that estimate. New accounts get an initial credit allowance for lower-cost models. The pricing page also says free verdicts expose the title, models and scores, while keeping prompts and examples hidden. A team should check verdict visibility before it starts an evaluation whose results should stay private.