https://tilvids.com/videos/watch/ffc61832-ffee-4493-baec-acbc653dfc96 Someone at work asks: "Could AI handle this for us?" You've seen the demo where it writes a perfect answer in seconds, and the threads where engineers say they were let down. And you can't tell what AI can actually do reliably, or how much of it is hype. I think "how good is AI?" hides three smaller questions: What kind of task it is, What the model was given to work with, What happens to its output afterwards., In this video, I use one example (an imagined pilot where AI drafts customer support replies in five seconds, yet a ticket still takes about as long as before) to show how to judge AI on your own task, end to end, and end with five checks to try on one task your team actually does. Full post and transcript: https://aiforthecurious.ai/videos/judge-ai-on-your-own-workflow Chapters 0:00 Could AI handle this for us? 0:59 The support pilot: 5-second drafts, same 8-minute ticket 1:50 Question 1: what kind of task is it? 3:17 Question 2: what did the model have to work with? 4:10 Question 3: what happens after the output? 5:18 AI as an amplifier (DORA) 6:27 Felt faster, measured slower (METR) 8:24 How to measure it on your own task 10:02 Did it save the work, or just move it? Sources Ask HN: How is AI-assisted coding going for you professionally? (Hacker News, 2026): https://news.ycombinator.com/item?id=47388646, DORA, "State of AI-assisted Software Development" (Google Cloud, 2025): https://research.google/pubs/dora-2025-state-of-ai-assisted-software-development-report/, METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (2025): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/, METR, "We are Changing our Developer Productivity Experiment Design" (2026): https://metr.org/blog/2026-02-24-uplift-update/, Cui, Demirer, Jaffe, Musolff, Peng and Salz, "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers" (Management Science): https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535, Next time someone shows you how fast an AI step got, what will you ask about the rest of the work? Tell me in the comments. Sun, 25 Oct 2026 16:36:33 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://tilvids.com https://tilvids.com/lazy-static/avatars/a8b2efbf-2a14-445c-a432-b4ebcf28cb7d.png https://tilvids.com/videos/watch/ffc61832-ffee-4493-baec-acbc653dfc96 All rights reserved, unless otherwise specified in the terms specified at https://tilvids.com/about and potential licenses granted by each content's rightholder.