I'm curious - why does it sound like a fallacy to you? As far as I am aware, detection models are rarely looking at the underlying meaning, and most claim to scan sentence by sentence and calculate scores on that basis - which is what the note is about. in fact many humanizers rewrite the text so that even humans have difficulty making sense of it, and yet many detectors will give a 100% human result on those humanized texts.
about your note - it's a purely mathematical eq. if some text is 100% AI, how it is shuffled/sliced (get AI to slice/re-order sentences so that it still makes sense; if you want to control for human input) should not change the score significantly.
and - Yes ofcourse, to humans reading - it would make a dramatic difference. But when tools claim 'we give you a sentence level breakdown', it implies a degree of isolated analysis. So without breaking sentence units, it would seem fair to play around to check how much a score changes. For instance, I tried a 75 word piece on a popular detector, the last 25 words came back as AI and first 50 as assisted (roughly). When I clipped the 25 and put them at the front of the paragraph, they came back as human. Not sure if you get what I mean?
Word substitution does change meaning, but here's how I see it - if you change 2/100 words. but the score moves 25%, it sounds like a highly sensitive model, which could also be interpreted as a result based on weak signals. The point here is simply that gimmicks like percentages should be taken as gimmicks, unless you can find more robust ways to substantiate the scores.
"if it is truly accurate measure of writing as is, it would give consistent % scores on the same text, no matter how it's sliced"
Sounds to me like a fallacy, even different punctuation can change the meaning of a sentence/paragraph. Let alone reordering sentences randomly.
I'm curious - why does it sound like a fallacy to you? As far as I am aware, detection models are rarely looking at the underlying meaning, and most claim to scan sentence by sentence and calculate scores on that basis - which is what the note is about. in fact many humanizers rewrite the text so that even humans have difficulty making sense of it, and yet many detectors will give a 100% human result on those humanized texts.
about your note - it's a purely mathematical eq. if some text is 100% AI, how it is shuffled/sliced (get AI to slice/re-order sentences so that it still makes sense; if you want to control for human input) should not change the score significantly.
and - Yes ofcourse, to humans reading - it would make a dramatic difference. But when tools claim 'we give you a sentence level breakdown', it implies a degree of isolated analysis. So without breaking sentence units, it would seem fair to play around to check how much a score changes. For instance, I tried a 75 word piece on a popular detector, the last 25 words came back as AI and first 50 as assisted (roughly). When I clipped the 25 and put them at the front of the paragraph, they came back as human. Not sure if you get what I mean?
Word substitution does change meaning, but here's how I see it - if you change 2/100 words. but the score moves 25%, it sounds like a highly sensitive model, which could also be interpreted as a result based on weak signals. The point here is simply that gimmicks like percentages should be taken as gimmicks, unless you can find more robust ways to substantiate the scores.