Built a shared spreadsheet ranking every AI tool my department tested this year
our department spent the last three pd sessions actually testing tools instead of just talking about them abstractly, and i built a shared spreadsheet tracking accuracy on known samples, false positive rate on our own past student work, cost per license, and ease of classroom integration for six different tools including turnitin, gptzero, originality.ai, grammarly and quillbot. originality.ai came out ahead on raw accuracy in our small test but was the most expensive per seat by a wide margin. turnitin scored lower on ai specific accuracy but weve already got it for traditional plagiarism so the marginal cost of enabling the ai module was basically zero. grammarly and quillbot arent really detection tools at all so we excluded them from the accuracy comparison and just noted them as writing support tools worth having separate policy conversations about. happy to share the spreadsheet template if anyone wants to run a similar exercise at their own school, structured comparison beat every individual opinion we had going in.
122 Replies
Join the discussion.
Log In to Reply
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.
@Kevin Thompson will post a link in a follow up once i get admin sign off on sharing our specific sample essays alongside the blank template, might need to swap those for generic samples first for privacy reasons.
appreciate the rigor here but still worth flagging that even the best scoring tool in your test wasnt anywhere close to perfect. structured comparison is great for choosing between imperfect options, it doesnt make any of them reliable enough for high stakes decisions on its own.
marginal cost basically zero for turnitin's ai module is probably why so many boards default to it even with worse accuracy. budget beats accuracy every time in these decisions.
this is exactly the kind of methodology more departments need, fascinating that the cost accuracy tradeoff was so stark between originality.ai and turnitin specifically. would add one suggestion for the template, track false negative rate separately from false positive rate rather than a single combined accuracy score, since the two errors have very different consequences for students and the decision about which tool to trust often depends on which type of error your school is more worried about.
this is fantastic, please share the template. we've had the same abstract arguing without any actual side by side testing at my school for two years running now.