feat(evaluations): run generation-only evaluations from code - #39
feat(evaluations): run generation-only evaluations from code#39donei003 wants to merge 7 commits into
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| api_client = LDApiClient( | ||
| api_token=token, | ||
| base_uri=base_uri or _env("LD_BASE_URI") or DEFAULT_BASE_URI, | ||
| transport=transport, | ||
| ) |
There was a problem hiding this comment.
🟡 Custom LaunchDarkly host setting for flag delivery is reused for management API calls, sending requests to the wrong server
The evaluations client picks up the same host setting already used for flag delivery (_env("LD_BASE_URI") at packages/client/src/launchdarkly_ai_server/evaluations/module.py:74) even though the two point at different LaunchDarkly services, so anyone who configured a streaming/relay host will have their evaluation requests sent to a server that cannot answer them.
Impact: Users with a relay proxy or staging streaming endpoint configured get failing or misdirected evaluation API calls instead of reaching the LaunchDarkly management API.
Env var collision between SDK polling base URI and /api/v2 base URI
packages/client/src/launchdarkly_ai_server/lifecycle.py:175 already consumes LD_BASE_URI as the SDK polling/streaming base URI (documented in packages/client/README.md:36 as "Override the LaunchDarkly polling base URI"). The evaluations module reuses the same variable but appends /api/v2/... (packages/client/src/launchdarkly_ai_server/evaluations/api.py:93), which targets the public management API (default https://app.launchdarkly.com). A user pointing LD_BASE_URI at e.g. a relay proxy or https://sdk.launchdarkly.com would silently redirect all evaluations API traffic there. A distinct variable (e.g. LD_API_BASE_URI) avoids the conflict.
| api_client = LDApiClient( | |
| api_token=token, | |
| base_uri=base_uri or _env("LD_BASE_URI") or DEFAULT_BASE_URI, | |
| transport=transport, | |
| ) | |
| api_client = LDApiClient( | |
| api_token=token, | |
| base_uri=base_uri or _env("LD_API_BASE_URI") or DEFAULT_BASE_URI, | |
| transport=transport, | |
| ) |
Was this helpful? React with 👍 or 👎 to provide feedback.
8151552 to
9ffa481
Compare
donei003
changed the title
donei003
changed the title
2 participants
Intent
Update existing draft PR #39 in place by moving into its independently reviewable diff the generation-result publishing/ingestion feature-flag gate currently contributed by stacked draft PR #44. Preserve the captain-selected current PR 44 HEAD contract exactly: use the single canonical flag key/constant enable-batch-ingest-in-evals-from-code; when LD_SDK_KEY is configured, only a variation that is strictly true enables generation-result ingest/publish, while false/default, malformed values, or evaluation errors safely skip ingest/publish; when no SDK key is configured and the gate cannot be evaluated, retain PR 39 existing generation-result ingestion behavior. Integrate the gate naturally with PR 39 generation-only evaluation publishing, with executable no-network coverage for enabled, disabled/default, no-SDK, malformed, evaluation-error-safe, and publishing/no-publishing behavior. Do not move PR 44 judge/scorer functionality or unrelated changes. Avoid duplicate constants, evaluations, tests, or documentation, preserve PR 39 existing generation-only SDK evaluation behavior outside the selected gate contract, and document the behavior accurately. Keep PR 39 draft, update its existing remote branch without opening a new PR, never force-push, and ensure the stacked outcome leaves the gate in PR 39 independent responsibility rather than PR 44.
What Changed
launchdarkly_ai_server.evaluationsmodule —init_evaluations(), anLDApiClientfor the management API, andEvaluationsModule.run()which resolves tools, fetches the dataset, creates the evaluation + client-source run, invokes the handler once per row, ingests generations, and returns LaunchDarkly's stored verdict.enable-batch-ingest-in-evals-from-codeflag: whenLD_SDK_KEYis set, only a strictlytruevariation publishes results, while false/default, malformed, or failed evaluations skip publish; without an SDK key the gate is unevaluable and ingest runs unconditionally.init_evaluationsand result types from the package barrel and documented the new entry point, env vars, and gate contract in the client andlaunchdarkly-ai-pythonREADMEs andagents.md, alongside no-network tests covering the runner, flag gate, and end-to-end run.Risk Assessment
✅ Low: The flag gate is a small, well-bounded change: a single canonical flag constant, fail-safe evaluation (identity check on True; broad exception catch returning False), and one call site that only runs when an SDK key is configured; executable no-network tests exercise enabled, disabled, malformed, evaluation-error, no-SDK, and publishing/no-publishing paths.
Testing
Targeted
packages/client/tests/test_evaluation_flags.pyandpackages/client/tests/test_evaluations_run.pyunderuv run pytest: 16/16 pass in 0.06s with aSequencedTransportmock that raises on any unexpected HTTP call, so all scenarios are strictly no-network. The parametrized ingest test proves theenable-batch-ingest-in-evals-from-codegate governs the/generation-resultsPOST (enabled → POST; disabled/malformed/eval-error → no POST), and the existing order-preservation test withLD_SDK_KEYunset confirms the pre-PR-39 ingest behavior is preserved when no SDK key is present. Flag-helper unit tests directly cover the malformed and evaluation-error safety branches. This is behavior-level evidence of the intent working end-to-end for CLI callers; no worktree artifacts were created and no source files modified. Coverage is sufficient; no findings.Evidence: gate-test-output
16 passed in 0.06s — flags helper + run() gate scenarios (enabled, disabled-default, malformed, evaluation-error, no-SDK, publishing/no-publishing) all exercise the public interface with SequencedTransport (no network).Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
uv run pytest packages/client/tests/test_evaluation_flags.py packages/client/tests/test_evaluations_run.py -v(16 passed)test_batch_ingest_flag_controls_generation_result_publishing[enabled|disabled-default|malformed|evaluation-error]— asserts/generation-resultsPOST presence/absence withLD_SDK_KEYsettest_run_calls_private_operations_in_order_and_returns_server_verdict— no-SDK path:init_client.assert_not_awaited()and existing ingest POST still firestest_enabled_flag_enables_generation_result_batch_ingest— asserts canonical flag key and project context are used with defaultFalsetest_disabled_flag_disables_generation_result_batch_ingest,test_malformed_flag_disables_generation_result_batch_ingest[None|1|'true'|{}],test_flag_evaluation_error_disables_generation_result_batch_ingest✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.