Skip to content

[camera_android_camerax] Update agent evals#12285

Open
reidbaker-agent wants to merge 10 commits into
mainfrom
update-agent-evals
Open

[camera_android_camerax] Update agent evals#12285
reidbaker-agent wants to merge 10 commits into
mainfrom
update-agent-evals

Conversation

@reidbaker-agent

@reidbaker-agent reidbaker-agent commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Add the concept of evals for skills to camera_android_camerax this is part of project-oneshot.
This is basically the equivalent of llm unit tests.
The idea and harness comes from dart_skills_lint see flutter/agent-plugins#189
The infra for running evals can and should be replaced by a tool that is designed for this use case when it is available.
In the meantime, the valuable parts are the evals.jsons files. They document the expectations for the skills under test in this repo.


Agent authored description
Updates agent evals for check-readiness and run-evals, including fixing missing license blocks and enforcing relative paths.

@reidbaker
reidbaker requested review from camsim99 and reidbaker and removed request for reidbaker July 24, 2026 20:30
@reidbaker reidbaker added the CICD Run CI/CD label Jul 24, 2026
Comment thread packages/camera/camera_android/README.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is no longer used. remove

DisplayOrientedMeteringPointFactory(CameraInfo cameraInfo, double width, double height);
}

// dummy eval modification

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should not be checked in. This is leaked state from an eval.

@flutter-dashboard flutter-dashboard Bot removed the CICD Run CI/CD label Jul 24, 2026
@reidbaker
reidbaker force-pushed the update-agent-evals branch from 0d0aa4d to 68f87b7 Compare July 24, 2026 21:10
@reidbaker
reidbaker marked this pull request as ready for review July 24, 2026 21:19

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds the run-evals skill, including its evaluation configurations, templates, and scripts, and introduces a test suite in packages/camera/camera_android_camerax to verify the structural consistency of evaluation files. Feedback on these changes highlights issues with hardcoded repository paths in the skill documentation, a missing reference to code_quality_rubric.json in the evaluation configuration, and the use of Directory.current.path in tests, which may cause failures when executed from the repository root.

void main() {
group('Evals structure consistency', () {
test('all evals.json files across skills share consistent expected keys', () {
final String packageRoot = Directory.current.path;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using Directory.current.path directly as the package root can cause test failures or silent skips when tests are run from the repository root (which is common in monorepos like flutter/packages or during CI runs). We should dynamically resolve the package root by checking if the .agents directory exists in the current working directory, and if not, fallback to the relative path of the package.

Suggested change
final String packageRoot = Directory.current.path;
final String packageRoot = Directory(p.join(Directory.current.path, '.agents')).existsSync()
? Directory.current.path
: p.join(Directory.current.path, 'packages', 'camera', 'camera_android_camerax');

});

test('all rubric JSON files in evals/ share consistent structure and keys', () {
final String packageRoot = Directory.current.path;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using Directory.current.path directly as the package root can cause this test to be silently skipped when run from the repository root because evals/ won't be found. Let's dynamically resolve the package root here as well.

Suggested change
final String packageRoot = Directory.current.path;
final String packageRoot = Directory(p.join(Directory.current.path, '.agents')).existsSync()
? Directory.current.path
: p.join(Directory.current.path, 'packages', 'camera', 'camera_android_camerax');

Comment on lines +10 to +11
1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals.
2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The paths tool/dart_skills_lint/evals/README.md and tool/dart_skills_lint/evals/ are hardcoded to the dart_skills_lint repository structure. In this repository (camera_android_camerax), these files and directories do not exist under tool/dart_skills_lint/. We should make these instructions generic or relative to the package root so that the agent can successfully locate the evaluations and rubrics.

Suggested change
1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals.
2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`.
1. **Read Framework**: Read `evals/README.md` (if present in the package root) for understanding the difference between per-skill evals and cross-skill evals.
2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in the package's `evals/` directory.

@@ -0,0 +1,62 @@
{
"repo_criteria": [
"evals/code_quality_rubric.json"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The rubric file evals/code_quality_rubric.json referenced here is not included in the repository or under third_party/skill-repos/agent-plugins/. This will cause the evaluation to fail when the agent attempts to resolve the referenced rubric. Please ensure that code_quality_rubric.json is added to the repository or remove/update this reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants