There's low-hanging fruit IMHO by testing using semantic naming vocabulary with semantic error vocabulary.
Example macro: assert_approx_eq_with_relative_error!(a, b, err). It's a semantic name that explains more about what it does, and the macro also provides a test failure message that guides the developer, or an AI, toward fixes and documentation.
What I learned is writing semantic testing can help many more kinds of higher-value testing. As a specific example described by the article, I use enumeration testing for proving the UK National Health Service Number validator in Rust is accurate i.e. the test literally checks every number, then I can write tests such as assert_valid_nhs_number("111 111 1111").
I'm not 100% certain, but I think "functional" here is used in more of the math sense: given a specific input, a specific output is obtained deterministically. A performance test doesn't meet this criterion, for example, because the run time of a program (or whatever) depends on a lot of factors, and any set of runs of the same test are most likely to generate distinct results.
Hmm, in my circles "non-functional" in this context (as in NFRs or "non-functional requirements") refers to things that don't relate strictly to the core functionality of a given feature or system. Performance (as in user-perceived latency) falls into this bucket (though I've always felt that performance is not just a feature, but the most important feature /digression).
I find that the biggest problem with all these classifications is that at the end of the day, developers will write whatever works for them in the moment, and not care if it fits your theoretical box.
I read your paper. It's quite interesting! "a large number of tests the developers actually write are using test-automation frameworks and infrastructure in ways the framework authors may not have intended".
I do think it'll be really interesting to use current gen LLMs to look through pre-AI githubs and do a richer profile of the kinds of tests that were hand crafted for high quality projects.
My experience with agents writing tests fully autonomously is that they are mostly ... a waste of their time and mine: they don't really catch the sort of regressions that modern agents are likely to cause.
I've heard much of the commercial value prop of SQLite is their test / regression suite.
> at the end of the day, developers will write whatever works for them in the moment, and not care if it fits your theoretical box.
That's description versus prescription: we should still be able to describe those "whatever works" tests, in a non-judgemental/loaded way, that others will interpret in the way it was intended.
I've worked on teams which, if shown the exact same test, would describe it with completely different terms (e.g. "functional unit test" versus "behaviour-driven integration test", etc.); and who, if asked to write a particular type of test (e.g. "check the DB code with a few integration tests"), would interpret that in completely different ways (say, one team might interpret that as "test how it actually works against a real RDBMS", whilst the other might interpret that as "test that the Query class instantiates Result instances correctly", or whatever).
It would be nice to have terminology which has a particular meaning, and for that to be stable over time. I've actually written about this before:
That is not limited to tests. People either do refactoring or they don’t. I saw some people come around on a trunk based development project but that was still incremental.
I loved this article. I find that the most important part of engineering projects is defining a shared vocabulary for discussing and aligning with other parties.
The vocabulary proposed that clear delineates across the various concerns you're testing against is particularly helpful.
This was also my first thought when reading OP. It is not restricted to testing even. The whole development process has this distinction (e.g. functional vs non-functional requirements) and it hurts more than it helps.
I think this is an important distinction that is also real when thinking about verification: verifying function can be boolean or lets say limited to a simple set while verifying non functional properties means mapping to a more continous interval and so it is a qualitative decision.
I think the problem with testing vocabulary is that the field is just too continuous. We have all sorts of situations in the world where there is no sharp line, yet we have terminology for various chunks of the possibility space even so because there are still some sort of clump in the space that we can identify. For example, there isn't a true sharp line between any of the primary colors, but we do not generally have a problem labelling things as "red" or "green" or whatever even so. There is a certain personality type frequently encountered online that this bothers no end, who can't really deal with the idea of words that have fuzzy boundaries around them and struggle with the concept of a "fuzzy line", but most of us get on even so.
However, when there is no natural clumping, there isn't even a place to put the fuzzy boundary necessarily. I think a lot of testing has this problem. I've disliked "unit" versus "integration" terminology for at least a decade now, because to me there's just a continuum. Yes, obviously, tests can test more or less of the code base at a time, and can draw in fewer or more dependencies in the process, and arguably there is an important distinction between 0 other dependencies included and 1, but beyond that I've thought for a long time there aren't any other distinctions worth putting a label on.
There is value to tests that mock out all the other dependencies. There is value to writing stubs. There is value to bringing up those dependencies and mocking or stubbing out their dependencies. There is value to having a full staging environment and running acceptance tests by hand, and that staging environment itself can have any number of mocks or stubs in it itself. And there is no rule that says you rigidly must make this decision based on the exact number of dependencies away something is from your code, to the extent that that is even a well-defined concept, nor are "mock" and "stub" themselves some sort of bright shining line as one can mix and match them freely, nor is that the complete list of all possibilities. The result is a fairly smooth continuum of possible tests. In my opinion, any attempt to put labels on very particular bits of it raises the risk of overprivileging certain parts of the space just because they get a name and creating the streetlight effect on the named bits and causing the others to be missed.
Even "performance testing" versus "correctness testing" gets awfully fuzzy; again, there is arguably some value in saying this test has performance concerns and this other test is just related to correctness, but even "performance testing" still has to worry about a lot of other things too and I don't find it anywhere near a binary distinction. Anyone who has spent any time with performance testing has certainly learned the risk of worrying about "performance" without also worrying about all the other things we test for at the same time, or to put it another way, I can make my performance testing look really good if I let the code it is testing be incorrect.
It is unreasonable to expect a universal nomenclature catalog of such a smooth space. We can't even as a species completely agree on color delineations in the spectrum, which is a simpler problem in most ways.
Time and time again I have had to halt discussion about functional and integration tests because nobody knew what the other people meant by integration and functional.
This tends to be along application versus service developer lines but even then it’s unclear whether functional means process or module level testing and integration means IPC testing, or whether integration testing means module level testing and functional means IPC.
I just checked and it looks like by 2018 if not earlier Fowler had given up on the distinction and used alternately integration vs E2E or service vs UI, citing Mike Cohn.
Even without the Tower of Babel thwarting conversations, testing is hard. I often say that to give people permission to fuck up in tests from time to time, and/or to impress upon them that you have to respect the tests the way we respect guns - they hurt you the moment you take them for granted.
And every time a coworker disagrees with the assertion that tests are hard, it’s someone whose tests I have beef with. Everything is easy if you don’t care about outcomes. Of course I have beef with -everyone’s- tests including my own. Which is why I know they’re hard, because even when you know what you’re supposed to do and try to do it, you don’t get it completely right.
So I keep an eye out for articles trying to rewrite how we are doing testing, because there are some pretty big impedance mismatches between the practice of testing and the goals of testing. And I don’t know if that’s because it’s a sort of Halting Problem, “only” NP-hard, or we haven’t found the correct viewpoint where everything makes sense.
For instance, I think by and large architectural and mechanical diagrams have given way to 3D modeling, because while I can read 3D projections in my sleep, a lot of people squeaked by in that class. We lack an equivalent representation in code testing. Code coverage is still another isomorphic projection that you can’t turn around until you see the object. 2D of code and .5D of coverage and branch execution.
I dont think this will last much longer. Looking around, i see most engineers not being interested in anything other than AI related stuff. Certainly true for the junior ones
There's low-hanging fruit IMHO by testing using semantic naming vocabulary with semantic error vocabulary.
Example macro: assert_approx_eq_with_relative_error!(a, b, err). It's a semantic name that explains more about what it does, and the macro also provides a test failure message that guides the developer, or an AI, toward fixes and documentation.
I wrote the Rust crate Assertables to help with this. See https://crates.io/crates/assertables.
What I learned is writing semantic testing can help many more kinds of higher-value testing. As a specific example described by the article, I use enumeration testing for proving the UK National Health Service Number validator in Rust is accurate i.e. the test literally checks every number, then I can write tests such as assert_valid_nhs_number("111 111 1111").
I would like "non-functional testing" to be called "cross-functional testing".
I'm not 100% certain, but I think "functional" here is used in more of the math sense: given a specific input, a specific output is obtained deterministically. A performance test doesn't meet this criterion, for example, because the run time of a program (or whatever) depends on a lot of factors, and any set of runs of the same test are most likely to generate distinct results.
Technically correct - totally wrong when explaining to non tech, non math people.
Hmm, in my circles "non-functional" in this context (as in NFRs or "non-functional requirements") refers to things that don't relate strictly to the core functionality of a given feature or system. Performance (as in user-perceived latency) falls into this bucket (though I've always felt that performance is not just a feature, but the most important feature /digression).
I find that the biggest problem with all these classifications is that at the end of the day, developers will write whatever works for them in the moment, and not care if it fits your theoretical box.
I did a paper on it once, to see what tests devs actually write: https://ieeexplore.ieee.org/document/11081450
I read your paper. It's quite interesting! "a large number of tests the developers actually write are using test-automation frameworks and infrastructure in ways the framework authors may not have intended".
I do think it'll be really interesting to use current gen LLMs to look through pre-AI githubs and do a richer profile of the kinds of tests that were hand crafted for high quality projects.
My experience with agents writing tests fully autonomously is that they are mostly ... a waste of their time and mine: they don't really catch the sort of regressions that modern agents are likely to cause.
I've heard much of the commercial value prop of SQLite is their test / regression suite.
> at the end of the day, developers will write whatever works for them in the moment, and not care if it fits your theoretical box.
That's description versus prescription: we should still be able to describe those "whatever works" tests, in a non-judgemental/loaded way, that others will interpret in the way it was intended.
I've worked on teams which, if shown the exact same test, would describe it with completely different terms (e.g. "functional unit test" versus "behaviour-driven integration test", etc.); and who, if asked to write a particular type of test (e.g. "check the DB code with a few integration tests"), would interpret that in completely different ways (say, one team might interpret that as "test how it actually works against a real RDBMS", whilst the other might interpret that as "test that the Query class instantiates Result instances correctly", or whatever).
It would be nice to have terminology which has a particular meaning, and for that to be stable over time. I've actually written about this before:
http://www.chriswarbo.net/blog/2017-11-10-unit_testing_termi...
http://www.chriswarbo.net/blog/2020-07-07-more_testing_termi...
That is not limited to tests. People either do refactoring or they don’t. I saw some people come around on a trunk based development project but that was still incremental.
I loved this article. I find that the most important part of engineering projects is defining a shared vocabulary for discussing and aligning with other parties.
The vocabulary proposed that clear delineates across the various concerns you're testing against is particularly helpful.
I think the alleged dictum between functional and non-functional is the biggest conceptual, theoretical, or even philosophical issue.
This was also my first thought when reading OP. It is not restricted to testing even. The whole development process has this distinction (e.g. functional vs non-functional requirements) and it hurts more than it helps.
I think this is an important distinction that is also real when thinking about verification: verifying function can be boolean or lets say limited to a simple set while verifying non functional properties means mapping to a more continous interval and so it is a qualitative decision.
I think the problem with testing vocabulary is that the field is just too continuous. We have all sorts of situations in the world where there is no sharp line, yet we have terminology for various chunks of the possibility space even so because there are still some sort of clump in the space that we can identify. For example, there isn't a true sharp line between any of the primary colors, but we do not generally have a problem labelling things as "red" or "green" or whatever even so. There is a certain personality type frequently encountered online that this bothers no end, who can't really deal with the idea of words that have fuzzy boundaries around them and struggle with the concept of a "fuzzy line", but most of us get on even so.
However, when there is no natural clumping, there isn't even a place to put the fuzzy boundary necessarily. I think a lot of testing has this problem. I've disliked "unit" versus "integration" terminology for at least a decade now, because to me there's just a continuum. Yes, obviously, tests can test more or less of the code base at a time, and can draw in fewer or more dependencies in the process, and arguably there is an important distinction between 0 other dependencies included and 1, but beyond that I've thought for a long time there aren't any other distinctions worth putting a label on.
There is value to tests that mock out all the other dependencies. There is value to writing stubs. There is value to bringing up those dependencies and mocking or stubbing out their dependencies. There is value to having a full staging environment and running acceptance tests by hand, and that staging environment itself can have any number of mocks or stubs in it itself. And there is no rule that says you rigidly must make this decision based on the exact number of dependencies away something is from your code, to the extent that that is even a well-defined concept, nor are "mock" and "stub" themselves some sort of bright shining line as one can mix and match them freely, nor is that the complete list of all possibilities. The result is a fairly smooth continuum of possible tests. In my opinion, any attempt to put labels on very particular bits of it raises the risk of overprivileging certain parts of the space just because they get a name and creating the streetlight effect on the named bits and causing the others to be missed.
Even "performance testing" versus "correctness testing" gets awfully fuzzy; again, there is arguably some value in saying this test has performance concerns and this other test is just related to correctness, but even "performance testing" still has to worry about a lot of other things too and I don't find it anywhere near a binary distinction. Anyone who has spent any time with performance testing has certainly learned the risk of worrying about "performance" without also worrying about all the other things we test for at the same time, or to put it another way, I can make my performance testing look really good if I let the code it is testing be incorrect.
It is unreasonable to expect a universal nomenclature catalog of such a smooth space. We can't even as a species completely agree on color delineations in the spectrum, which is a simpler problem in most ways.
Time and time again I have had to halt discussion about functional and integration tests because nobody knew what the other people meant by integration and functional.
This tends to be along application versus service developer lines but even then it’s unclear whether functional means process or module level testing and integration means IPC testing, or whether integration testing means module level testing and functional means IPC.
I just checked and it looks like by 2018 if not earlier Fowler had given up on the distinction and used alternately integration vs E2E or service vs UI, citing Mike Cohn.
Even without the Tower of Babel thwarting conversations, testing is hard. I often say that to give people permission to fuck up in tests from time to time, and/or to impress upon them that you have to respect the tests the way we respect guns - they hurt you the moment you take them for granted.
And every time a coworker disagrees with the assertion that tests are hard, it’s someone whose tests I have beef with. Everything is easy if you don’t care about outcomes. Of course I have beef with -everyone’s- tests including my own. Which is why I know they’re hard, because even when you know what you’re supposed to do and try to do it, you don’t get it completely right.
So I keep an eye out for articles trying to rewrite how we are doing testing, because there are some pretty big impedance mismatches between the practice of testing and the goals of testing. And I don’t know if that’s because it’s a sort of Halting Problem, “only” NP-hard, or we haven’t found the correct viewpoint where everything makes sense.
For instance, I think by and large architectural and mechanical diagrams have given way to 3D modeling, because while I can read 3D projections in my sleep, a lot of people squeaked by in that class. We lack an equivalent representation in code testing. Code coverage is still another isomorphic projection that you can’t turn around until you see the object. 2D of code and .5D of coverage and branch execution.
Always enjoyable for us aged s/w engineers to see the new generations relearn the fundamentals.
I dont think this will last much longer. Looking around, i see most engineers not being interested in anything other than AI related stuff. Certainly true for the junior ones
AI opens up whole new ways of validating that systems behave correctly or to find issues.
Maybe or maybe now we can do proper testing.
Most of the testing techniques are baed on groups/set and graph theories while test data is based on sampling.
So now we really have the chance to model verification and validation even better
I recall finding different definitions in every book about testing I read.
I think most of the confusion came from people suddenly deciding that a unit test can't test less than a whole object, in OOP.
Mad Men meme:
"Testing, but applied to terminology across publications."
does vocabulary matter anymore as coding agents can make interpretations?
great infographic