The hardest survey field is often “Anything else you would like to share.” Open responses mix a broken button, an entire workflow and a moment of frustration. Asking AI to “summarise user feedback” can create neat conclusions with no trail back to evidence. Preserve wording, define the questions first, then use AI for extraction and candidate themes.
Use ChatGPT or Claude only on redacted responses. It should not calculate representative percentages, infer intent or invent demand.
Redact and retain context
Remove names, email, phones, order IDs and company names from the working copy. Use response_id and retain source, date, user role and related rating. “Too complicated” means different things for a first-time user and a daily administrator.
Define research questions before keywords
Ask what task users stop at, what information is unclear, or who would use the product again. Keep task, blocker, verbatim evidence and follow-up need for every response. “Price” can mean expensive, unclear or undiscoverable; a word cloud cannot tell you which.
From these redacted open responses, extract only explicit information: response_id, user task, feedback type, observed blocker, verbatim evidence, sentiment, follow-up question and confidence. Use “unknown” when needed. Do not estimate satisfaction, market size, intent or frequency, and do not merge responses.
Confirm themes and their boundaries
AI may propose candidate themes, but similar wording is not enough to merge replies. Review task, condition and consequence. A theme card should hold its evidence, sample count, conditions, current hypothesis and next validation step.
Be honest about counts
If 8 of 40 valid responses mention import problems, report exactly that. Do not claim 20% of all users are affected: open text is rarely a random sample. Counts are signals, not proof. Compare rating questions and open responses without forcing an explanation.
Turn a theme into a small validation
Choose one next action: improve documentation, revise wording, reproduce a bug, run usability tests, ask follow-up questions or defer with a reason. Define hypothesis, evidence needed, owner and completion standard. Traceable evidence and explicit sample boundaries turn AI from a generic summariser into a useful research assistant.