Claude Docs sorted 400 customer comments into ten themes, and all ten tallies were right
Seventy-two customers asked for dark mode. I wrote those answers myself, so I knew that number before Claude saw the file. Claude Docs is Anthropic's new document editor inside Claude. You attach files and ask, and Claude writes a doc you can edit. From four hundred survey answers, Claude Docs wrote a report with a count for every theme, and all ten counts match mine. The company and its customers are made up. Pinnockly sells a task board for small teams. Every quarter its product manager asks one open question, what's the one thing you'd change. Someone has to read, tag and count four hundred answers before Thursday's roadmap review.
Free-text survey answers become numbers through tagging, counting and quoting
The method is older than the software. Researchers call it coding. You give each answer a short tag for what it's about, you count the tags, and you pull a quote for each one. Anthropic publishes the method for product teams as a checklist on GitHub. The part on open-ended survey answers has five lines, and the middle three are the whole method. Code each response with themes. Count frequency of themes across responses. Pull representative quotes for each theme. My request to Claude Docs asked for those same three steps in plain words.
Coding gives each comment one or more tags, even for words it leaves out
Take three answers from the file. The first answer says, export to CSV keeps timing out. That answer gets the tag export. A CSV is a spreadsheet saved as plain text. The next answer comes from someone who loves the product, but exports fail half the time and the mobile app is slow. That answer gets two tags, export and mobile. That person also asks for exports twice, and still counts once for export. The third answer says, can't get my data out. The word export is missing. A reader tags it export anyway, because getting your data out is what an export does. That match takes judgment, and a word search skips it.
How you match and merge the tags decides the order of the themes
Tags that mean the same thing merge into a theme. The themes rank by people, and each person counts once per theme. Two choices can move that ranking. The first choice is how you match. Twenty-eight of the sixty-four export answers skip the word export. Search the file for that word and you find thirty-seven answers, and export falls from second place to sixth. The second choice is what you merge. Forty-two people said the price is too high. Eighteen people were billed the wrong amount. Put both groups in one pricing bucket, and that bucket holds sixty people and jumps above mobile, at fifty-eight. The two groups need different fixes. The totals have a trap too. Fifty-two people raised two or three themes, so the theme counts add up past the three hundred and forty-three people who answered.
The nine customers who lost work sit last in a ranking by count
Nine people described losing work. One says a teammate's bulk move wiped the due dates on sixty tasks. Another answer opens with dark mode, and then says sync overwrote the board and the person lost a week of changes. That answer needs both tags. Tag it dark mode alone, and the lost work theme drops to eight people. Six of the nine are workspace admins, the people who run the account for their team. In a list sorted by count, lost work comes tenth of ten.
The survey goes in with a 69-word request and one menu choice
In Claude, attach the file, open the Output menu under the message box, and pick Document. Claude Docs is in beta, an early release, on the Pro, Max, Team and Enterprise plans. My request asked for a themes report for Thursday's roadmap review. It told Claude to tag every answer with one or more themes, and to count each theme by people, with each person counted once. It asked Claude to say what the counts are out of. It also asked for one word-for-word quote per theme, with the response number beside it.
The report’s ten counts equal the planted ones, row for row
The report came back in two minutes and ten seconds. Claude wrote a tag for each of the four hundred answers, then counted people per theme. On three hundred and ninety-nine answers, Claude's tags match mine exactly. The one difference is an answer that says not sure. Claude called it off-topic, and I called it no change. Claude tagged all twenty-eight export answers that skip the word. Claude kept the forty-two price complaints apart from the eighteen billing errors, and the report says why. The report counts each person once and says so. The report states its base too, three hundred and forty-three people who named a change, out of four hundred. The report also raises the two small themes. On satisfaction, out of five, the people with billing errors averaged two point two. The people who lost work averaged two point three. Everyone who answered averaged three point six.
Claude ranks the themes by count, and the product manager ranks them by impact
Here is where I'd stop and read it myself. Anthropic's checklist puts themes with high frequency and high impact first. The report can measure frequency. Impact is the column only the product manager can fill. My call is that lost work goes first on Thursday, because losing work is the complaint that ends a subscription. Export goes to the review as a candidate. Dark mode gets a follow-up question in the next survey, desktop or mobile, because two answers say the mobile app already has it. Billing errors go to the finance team this week. Then export the report as a Word file, a PDF or Markdown, which is a plain text file, or send a copy to Google Docs or Notion. Claude Docs has no version history yet, so save the copy you present.
Two by-hand checks catch a wrong count or a misquoted answer before Thursday
Two checks stay with you, and they take about ten minutes by my estimate. Claude sends your survey file back with a themes column added. Filter that column for each of the top three themes, and count the rows. The recount gives seventy-two, sixty-four and fifty-eight, the same as the report. Then search the survey for each quote, and read the response number beside it. All ten quotes are whole answers, word for word, from the row the report cites. This run matched, and the next survey gets the same two checks.
The themes report needs about 40 minutes with Claude Docs, against about two hours by hand
15
67
15
quote · 15
20
15
20
By hand I estimate two hours and twelve minutes. Most of that is reading and tagging four hundred answers, then merging, counting and writing it up. With Claude Docs the estimate is thirty-nine minutes. The draft took two minutes and ten seconds, and that part is measured. Twenty of the thirty-nine minutes go to checking Claude's work. Fifteen go to deciding what matters, the same fifteen minutes it takes by hand.
9 people who lost work go ahead of 72 who asked for dark mode
Claude Docs did the reading, and all ten counts held up. The counts are what the roadmap review repeats, so the counts are what you check, and the order is yours to set. Which theme from your last survey would you recount first?
