1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
Would you know of decent sources on creating (personal mostly) benchmarks? Doing something like that currently for processing PDFs and it's a bit cumbersome, but that's work I guess.
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it
Evals and benchmarks lol. Make the eval to see what the AI should be doing, benchmark to see what the whole landscape is doing. Refine and repeat and that’ll make it better
There are two ways I see to do this:
1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
Would you know of decent sources on creating (personal mostly) benchmarks? Doing something like that currently for processing PDFs and it's a bit cumbersome, but that's work I guess.
This article should is a good source for getting started in the space https://every.to/also-true-for-humans/how-to-create-your-own...
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it
Thank you, will give it a read.
Evals and benchmarks lol. Make the eval to see what the AI should be doing, benchmark to see what the whole landscape is doing. Refine and repeat and that’ll make it better