charlie-evals/outline/issues/9
Scores are averages across provided metric entries.
LLM-judged text
- GPT: While Claude's PR had better narrative and slightly higher implementation polish, Charlie's strong testability and comparable quality make his the more complete and reliable contribution.
- Gemini: Although Claude's PR has a slightly higher average quality score and better description, its testability is critically low (average 0.1044 vs. Charlie's 0.7420). Claude's lack of testing for new behavior, especially for core scenarios like relative-to-absolute conversion, leaves the integrity of the feature completely unverified. In contrast, Charlie's PR adds meaningful unit tests, even if they have some limitations, making his contribution far more reliable and verifiable.
Charlie
- Testability: The PR adds meaningful unit tests for the new absolutizeImageUrls utility, verifying key behaviors like converting root-relative image URLs, preserving absolute URLs, and ignoring non-image links. However, coverage is limited by missing edge cases (e.g., protocol-relative URLs, titles, and empty input) and lacks integration tests for the downstream usage in copyDocumentAsMarkdown.
- Description: The PR title is clear and the description correctly explains the problem and high-level solution, aligning with the actual code changes. However, the body contains placeholder text, lacks concrete details like function names and file paths, and includes a broken verification section—reducing clarity, coverage, and long-term usefulness.
- Overall quality: The change is focused, minimal, and idiomatic: it introduces a well-scoped helper to convert root-relative image URLs to absolute ones and integrates it cleanly into the “Copy as Markdown” action. While the implementation is correct and tests cover key cases, minor nits include regex brittleness, an unused parameter, and edge cases involving alt text or env.URL handling.
Claude
- Testability: No tests were added for the new baseUrl behavior in markdown serialization, despite introducing logic to convert relative image URLs to absolute URLs via Image.toMarkdown and Document.toMarkdown. Core scenarios—including relative-to-absolute conversion, trailing slash handling, absent or absolute URLs, and propagation of baseUrl—are all untested, leaving presence, depth, and integrity entirely lacking.
- Description: The title is clear and the description accurately explains the problem and high-level fix: converting relative image paths in Copy as Markdown to absolute URLs using a new optional baseUrl passed from team.url. While the test plan is relevant and the framing is sound, most summaries omit details like the serializer API change, the restriction to leading-slash paths, and the fact that only specific markdown generation paths are affected.
- Overall quality: The change is focused, minimal, and idiomatic: it introduces an optional baseUrl to the Markdown serializer, uses it in the Image node to convert root-relative image paths to absolute URLs, and wires it into Document.toMarkdown for the “Copy as Markdown” action via team.url. While the implementation is correct and consistent with existing patterns, a minor limitation is that other Markdown serializers (like ClipboardTextSerializer) don't yet support baseUrl, so some copy paths still yield relative URLs—acceptable given the stated scope.