[camera_android_camerax] Update agent evals#12285
Conversation
There was a problem hiding this comment.
I think this is no longer used. remove
| DisplayOrientedMeteringPointFactory(CameraInfo cameraInfo, double width, double height); | ||
| } | ||
|
|
||
| // dummy eval modification |
There was a problem hiding this comment.
This file should not be checked in. This is leaked state from an eval.
0d0aa4d to
68f87b7
Compare
There was a problem hiding this comment.
Code Review
This pull request adds the run-evals skill, including its evaluation configurations, templates, and scripts, and introduces a test suite in packages/camera/camera_android_camerax to verify the structural consistency of evaluation files. Feedback on these changes highlights issues with hardcoded repository paths in the skill documentation, a missing reference to code_quality_rubric.json in the evaluation configuration, and the use of Directory.current.path in tests, which may cause failures when executed from the repository root.
| void main() { | ||
| group('Evals structure consistency', () { | ||
| test('all evals.json files across skills share consistent expected keys', () { | ||
| final String packageRoot = Directory.current.path; |
There was a problem hiding this comment.
Using Directory.current.path directly as the package root can cause test failures or silent skips when tests are run from the repository root (which is common in monorepos like flutter/packages or during CI runs). We should dynamically resolve the package root by checking if the .agents directory exists in the current working directory, and if not, fallback to the relative path of the package.
| final String packageRoot = Directory.current.path; | |
| final String packageRoot = Directory(p.join(Directory.current.path, '.agents')).existsSync() | |
| ? Directory.current.path | |
| : p.join(Directory.current.path, 'packages', 'camera', 'camera_android_camerax'); |
| }); | ||
|
|
||
| test('all rubric JSON files in evals/ share consistent structure and keys', () { | ||
| final String packageRoot = Directory.current.path; |
There was a problem hiding this comment.
Using Directory.current.path directly as the package root can cause this test to be silently skipped when run from the repository root because evals/ won't be found. Let's dynamically resolve the package root here as well.
| final String packageRoot = Directory.current.path; | |
| final String packageRoot = Directory(p.join(Directory.current.path, '.agents')).existsSync() | |
| ? Directory.current.path | |
| : p.join(Directory.current.path, 'packages', 'camera', 'camera_android_camerax'); |
| 1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals. | ||
| 2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`. |
There was a problem hiding this comment.
The paths tool/dart_skills_lint/evals/README.md and tool/dart_skills_lint/evals/ are hardcoded to the dart_skills_lint repository structure. In this repository (camera_android_camerax), these files and directories do not exist under tool/dart_skills_lint/. We should make these instructions generic or relative to the package root so that the agent can successfully locate the evaluations and rubrics.
| 1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals. | |
| 2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`. | |
| 1. **Read Framework**: Read `evals/README.md` (if present in the package root) for understanding the difference between per-skill evals and cross-skill evals. | |
| 2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in the package's `evals/` directory. |
| @@ -0,0 +1,62 @@ | |||
| { | |||
| "repo_criteria": [ | |||
| "evals/code_quality_rubric.json" | |||
There was a problem hiding this comment.
The rubric file evals/code_quality_rubric.json referenced here is not included in the repository or under third_party/skill-repos/agent-plugins/. This will cause the evaluation to fail when the agent attempts to resolve the referenced rubric. Please ensure that code_quality_rubric.json is added to the repository or remove/update this reference.
Add the concept of evals for skills to camera_android_camerax this is part of project-oneshot.
This is basically the equivalent of llm unit tests.
The idea and harness comes from dart_skills_lint see flutter/agent-plugins#189
The infra for running evals can and should be replaced by a tool that is designed for this use case when it is available.
In the meantime, the valuable parts are the evals.jsons files. They document the expectations for the skills under test in this repo.
Agent authored description
Updates agent evals for check-readiness and run-evals, including fixing missing license blocks and enforcing relative paths.