The reasoning behind Apprentice’s safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.
This page explains a capability that is coming soon. Today Apprentice ships the prompt-optimization SDK. The shadow router, canary rollout, and automatic takeover described here are in development. The design is settled; the public flow is not live yet.
Replacing a frontier model with a smaller one is where most cost-cutting attempts quietly lose quality. Apprentice’s answer is to never let the smaller model take traffic on faith. Every shift is gated by your own evaluation, and the frontier model is always one step behind, ready to take back over.
The problem with a naive swap
A 4B model is cheap to serve, but a blind swap trades a known-good model for an unknown one. Quality regressions on real traffic are easy to miss until a customer finds them. The cost saving is real; the risk is that you cannot see the regression until it has already shipped.
The gate
The smaller model only earns traffic by passing a gate measured on gold rows only, never on model-generated data. This is the same honesty rule as the data tiers: the model is never graded against another model’s output. If it does not clear your threshold on human-verified rows, your frontier model keeps serving and nothing changes for your users.You set the thresholds. They are yours to configure per task, because what “good enough” means is your call, not ours.
Why a canary, not a switch
Even a model that passes the gate goes live gradually, not all at once. A small share of live traffic moves to the smaller model first, measured against the frontier model it is replacing. Only when it holds up does its share grow. This is the standard safe-rollout pattern: contain the blast radius of any surprise to a fraction of requests.
Why fallbacks count against the saving
When the smaller model is unsure and the request falls back to the frontier model, that fallback is counted as a failed replacement in the savings math. Hiding fallbacks would inflate the reported saving and quietly reintroduce the cost you were trying to cut. Counting them keeps the number honest: the saving you see is the saving you got.
Why rollback is always one click
The frontier model stays on hot standby for the life of the takeover. If anything looks wrong, traffic reverts in under a second, with an audit-log entry on every promotion and rollback. The takeover is reversible by design, so adopting it is never a one-way door.
The bigger picture
The throughline is that cost reduction must not cost you quality, and the only way to prove that is to measure against verified truth before and during every shift. Prompt optimization, which ships today, is the first half of this: it improves the prompt against the same verified dataset. Model replacement is the second half, and it reuses the same gate.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Apprentice","description":"Apprentice documentation: cut frontier-LLM cost on repeatable tasks. Optimize prompts on a verified dataset with the runapprentice Python SDK.","colors":{"primary":"#0B7A57","light":"#1FA97B","dark":"#0A6A4C"},"favicon":"https://mintcdn.com/apprentice/Kt2pVQJsrB80LQvt/favicon.png?fit=max\u0026auto=format\u0026n=Kt2pVQJsrB80LQvt\u0026q=85\u0026s=3929ba6e11af37ac7e77936a1d7ffbf3","appearance":{"default":"dark"},"background":{"color":{"light":"#F7F3EA","dark":"#0A0A0B"}},"navbar":{"links":[{"href":"https://github.com/singhabhishekkk/apprentice-benchmark","label":"Benchmark"}],"primary":{"type":"button","label":"runapprentice.com","href":"https://runapprentice.com"}},"footer":{"socials":{"github":"https://github.com/singhabhishekkk","website":"https://runapprentice.com"}},"seo":{"metatags":{"og:title":"Apprentice docs: cut LLM cost without losing quality","og:description":"Optimize prompts on a verified dataset with the runapprentice Python SDK. Quickstart, how-to guides, and the full SDK reference.","og:image":"https://runapprentice.com/opengraph-image","twitter:card":"summary_large_image","twitter:image":"https://runapprentice.com/opengraph-image"},"indexing":"all"},"contextual":{"options":["copy","view","chatgpt","claude","cursor","vscode","mcp"],"display":"header"},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"apprentice","hasAudioNarration":false,"actualSubdomain":"apprentice","customDomains":["docs.runapprentice.com"],"gitSource":{"type":"github","owner":"singhabhishekkk","repo":"apprentice","deployBranch":"main","contentDirectory":"apprentice-docs","isPrivate":true,"mountPath":"","label":""},"entitlements":{"AI_CHAT":{"status":"DISABLED"}},"buildId":"6af6fabbe90c1f4f013ecc87","clientVersion":"0.0.4270","searchFilterCounts":[{"language":"default","tag":"How-to guides","count":6},{"language":"default","tag":"Concepts","count":3},{"language":"default","tag":"Get started","count":2},{"language":"default","tag":"Reference","count":1}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"site-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[]}
{"page":{"pagePath":"concepts/eval-gated-replacement","slug":"/concepts/eval-gated-replacement","pageMetadata":{"title":"Why replacement is eval-gated","description":"The reasoning behind Apprentice's safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.","href":"/concepts/eval-gated-replacement"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"The reasoning behind Apprentice’s safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,p:`p`,strong:`strong`,..._provideComponents(),...props.components},{Heading,Note,OptimizedImage}=_components;if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);return _jsxs(_Fragment,{children:[_jsx(Note,{children:_jsxs(_components.p,{children:[`This page explains a capability that is `,_jsx(_components.strong,{children:`coming soon`}),`. Today Apprentice ships the prompt-optimization SDK. The shadow router, canary rollout, and automatic takeover described here are in development. The design is settled; the public flow is not live yet.`]})}),`\n`,_jsx(_components.p,{children:`Replacing a frontier model with a smaller one is where most cost-cutting attempts quietly lose quality. Apprentice’s answer is to never let the smaller model take traffic on faith. Every shift is gated by your own evaluation, and the frontier model is always one step behind, ready to take back over.`}),`\n`,_jsx(OptimizedImage,{className:`mint-hidden dark:mint-block`,src:`https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=1fb9a916ff93b1a67bf9a89c19564a73`,alt:`The challenger is scored by the eval gate on held-out gold; only a pass promotes it to production, and shadow traffic is capped`,\"data-og-width\":`760`,width:`760`,\"data-og-height\":`330`,height:`330`,\"data-path\":`images/diagrams/gate-dark.svg`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=280\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=f753d0ac78d5266237f41ea8aacc94f2 280w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=560\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=2351be6bd92bc81483c4628ae3c67fca 560w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=840\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=00bc4bd7decb75e5fab73de79697c7f6 840w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=1100\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=86b8d2caaf3611c1ceea3d899e0b2723 1100w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=1650\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=bc8963a96d6e6b4747888679e67e3359 1650w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-dark.svg?w=2500\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=1a7ca537942c1b68ef92e9cd98f7491f 2500w`}),`\n`,_jsx(OptimizedImage,{className:`mint-block dark:mint-hidden`,src:`https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=2f29f3e53decd3eaaab389493f5f0990`,alt:`The challenger is scored by the eval gate on held-out gold; only a pass promotes it to production, and shadow traffic is capped`,\"data-og-width\":`760`,width:`760`,\"data-og-height\":`330`,height:`330`,\"data-path\":`images/diagrams/gate-light.svg`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=280\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=a17e47357c9b8bda71c10ba3c032d56e 280w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=560\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=f4599b96b4b5984074503e6763db2921 560w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=840\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=10483840b4bb0fa64cc192c5d6d6ebe5 840w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=1100\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=3b094f565113219deb9fc4719e387f43 1100w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=1650\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=9a47cc5c756eb59dbacdfc47d73d11b0 1650w, https://mintcdn.com/apprentice/YtAzRJYvgDVx46XU/images/diagrams/gate-light.svg?w=2500\u0026fit=max\u0026auto=format\u0026n=YtAzRJYvgDVx46XU\u0026q=85\u0026s=7542c5b03b75adb3d2f599e9a4613e9d 2500w`}),`\n`,_jsx(Heading,{level:`2`,id:`the-problem-with-a-naive-swap`,children:`The problem with a naive swap`}),`\n`,_jsx(_components.p,{children:`A 4B model is cheap to serve, but a blind swap trades a known-good model for an unknown one. Quality regressions on real traffic are easy to miss until a customer finds them. The cost saving is real; the risk is that you cannot see the regression until it has already shipped.`}),`\n`,_jsx(Heading,{level:`2`,id:`the-gate`,children:`The gate`}),`\n`,_jsxs(_components.p,{children:[`The smaller model only earns traffic by passing a gate measured on `,_jsx(_components.strong,{children:`gold rows only`}),`, never on model-generated data. This is the same honesty rule as the `,_jsx(_components.a,{href:`/concepts/data-tiers`,children:`data tiers`}),`: the model is never graded against another model’s output. If it does not clear your threshold on human-verified rows, your frontier model keeps serving and nothing changes for your users.`]}),`\n`,_jsx(_components.p,{children:`You set the thresholds. They are yours to configure per task, because what “good enough” means is your call, not ours.`}),`\n`,_jsx(Heading,{level:`2`,id:`why-a-canary-not-a-switch`,children:`Why a canary, not a switch`}),`\n`,_jsx(_components.p,{children:`Even a model that passes the gate goes live gradually, not all at once. A small share of live traffic moves to the smaller model first, measured against the frontier model it is replacing. Only when it holds up does its share grow. This is the standard safe-rollout pattern: contain the blast radius of any surprise to a fraction of requests.`}),`\n`,_jsx(Heading,{level:`2`,id:`why-fallbacks-count-against-the-saving`,children:`Why fallbacks count against the saving`}),`\n`,_jsx(_components.p,{children:`When the smaller model is unsure and the request falls back to the frontier model, that fallback is counted as a failed replacement in the savings math. Hiding fallbacks would inflate the reported saving and quietly reintroduce the cost you were trying to cut. Counting them keeps the number honest: the saving you see is the saving you got.`}),`\n`,_jsx(Heading,{level:`2`,id:`why-rollback-is-always-one-click`,children:`Why rollback is always one click`}),`\n`,_jsx(_components.p,{children:`The frontier model stays on hot standby for the life of the takeover. If anything looks wrong, traffic reverts in under a second, with an audit-log entry on every promotion and rollback. The takeover is reversible by design, so adopting it is never a one-way door.`}),`\n`,_jsx(Heading,{level:`2`,id:`the-bigger-picture`,children:`The bigger picture`}),`\n`,_jsx(_components.p,{children:`The throughline is that cost reduction must not cost you quality, and the only way to prove that is to measure against verified truth before and during every shift. Prompt optimization, which ships today, is the first half of this: it improves the prompt against the same verified dataset. Model replacement is the second half, and it reuses the same gate.`})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Why replacement is eval-gated","description":"The reasoning behind Apprentice's safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.","href":"/concepts/eval-gated-replacement"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"The problem with a naive swap","slug":"the-problem-with-a-naive-swap","depth":2,"children":[]},{"title":"The gate","slug":"the-gate","depth":2,"children":[]},{"title":"Why a canary, not a switch","slug":"why-a-canary-not-a-switch","depth":2,"children":[]},{"title":"Why fallbacks count against the saving","slug":"why-fallbacks-count-against-the-saving","depth":2,"children":[]},{"title":"Why rollback is always one click","slug":"why-rollback-is-always-one-click","depth":2,"children":[]},{"title":"The bigger picture","slug":"the-bigger-picture","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"dynamicCss":".mint-block {\n display: block\n}\n.mint-hidden {\n display: none\n}\n.dark\\:mint-block:is(.dark *) {\n display: block\n}\n.dark\\:mint-hidden:is(.dark *) {\n display: none\n}","subdomain":"apprentice","siteName":"Apprentice","title":"Apprentice docs: cut LLM cost without losing quality","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-07-12T00:02:19.538Z","originalFileLocation":"concepts/eval-gated-replacement.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"dark\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"The reasoning behind Apprentice\u0026#x27;s safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.\">\u003cmeta data-mint-head name=\"application-name\" content=\"Apprentice\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Apprentice docs: cut LLM cost without losing quality\">\u003cmeta data-mint-head property=\"og:description\" content=\"Optimize prompts on a verified dataset with the runapprentice Python SDK. Quickstart, how-to guides, and the full SDK reference.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Apprentice\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.runapprentice.com/concepts/eval-gated-replacement\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://runapprentice.com/opengraph-image\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Apprentice docs: cut LLM cost without losing quality\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Optimize prompts on a verified dataset with the runapprentice Python SDK. Quickstart, how-to guides, and the full SDK reference.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://runapprentice.com/opengraph-image\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/apprentice/ieH8VPn9rNiKveV1/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Apprentice\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#0B7A57\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.runapprentice.com/concepts/eval-gated-replacement\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/concepts/eval-gated-replacement.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.runapprentice.com/#organization\",\"name\":\"Apprentice\",\"url\":\"https://docs.runapprentice.com\"},{\"@type\":\"WebSite\",\"@id\":\"https://docs.runapprentice.com#website\",\"name\":\"Apprentice\",\"url\":\"https://docs.runapprentice.com\",\"publisher\":{\"@id\":\"https://docs.runapprentice.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement#webpage\",\"url\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement\",\"name\":\"Why replacement is eval-gated\",\"description\":\"The reasoning behind Apprentice's safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.\",\"dateModified\":\"2026-07-12T00:02:19.538Z\",\"isPartOf\":{\"@id\":\"https://docs.runapprentice.com#website\"},\"breadcrumb\":{\"@id\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Concepts\",\"item\":\"https://docs.runapprentice.com/concepts/using-the-console\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Why replacement is eval-gated\",\"item\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement#article\",\"headline\":\"Why replacement is eval-gated\",\"name\":\"Why replacement is eval-gated\",\"description\":\"The reasoning behind Apprentice's safety model: how a smaller model takes over from a frontier model only when your evals prove no quality regression.\",\"url\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement\",\"mainEntityOfPage\":{\"@id\":\"https://docs.runapprentice.com/concepts/eval-gated-replacement#webpage\"},\"image\":\"https://runapprentice.com/opengraph-image\",\"dateModified\":\"2026-07-12T00:02:19.538Z\",\"publisher\":{\"@id\":\"https://docs.runapprentice.com/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.runapprentice.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Apprentice\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Apprentice\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/concepts/eval-gated-replacement.md","siteName":"Apprentice","nav":{"global":null,"groups":[{"group":"Get started","pages":[{"title":"Apprentice","href":"/index"},{"title":"Quickstart","href":"/quickstart"}]},{"group":"How-to guides","pages":[{"title":"JSON extraction, end to end","href":"/how-to/json-extraction"},{"title":"Capture from an OpenAI app","href":"/how-to/capture-openai"},{"title":"Capture from a LangChain app","href":"/how-to/capture-langchain"},{"title":"Use Apprentice inside Claude Code, Codex, and Copilot CLI","sidebarTitle":"Use inside Claude Code and Codex","href":"/how-to/agent-skill"},{"title":"MLX fine-tuning on Apple silicon","sidebarTitle":"Train and serve on Mac","href":"/how-to/deploy-mlx"},{"title":"Deploy in your Kubernetes cluster","href":"/how-to/deploy-kubernetes"}]},{"group":"Concepts","pages":[{"title":"Using the console","href":"/concepts/using-the-console"},{"title":"Data tiers: gold, silver, raw","description":"What gold, silver, and raw rows mean in Apprentice, and which tiers count for optimization versus eval gates.","href":"/concepts/data-tiers"},{"title":"Why replacement is eval-gated","href":"/concepts/eval-gated-replacement"}]},{"group":"Reference","pages":[{"title":"Python SDK reference","description":"The runapprentice client's capture, feedback, dataset, and optimization methods: parameters, return values, and what each raises.","href":"/reference/python-sdk"}]}]},"basePath":"","title":"Apprentice docs: cut LLM cost without losing quality","deploymentId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","buildId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","dynamics":["a74be98ba7"],"chunks":["zoom-image-lightbox-BQ4tbj_9.js","lightbox-handle-r17bVUz3.js"]}