Your research data may be invalid before the analysis

The most expensive error in research isn’t a bad chart. It happens before the analysis, when the data was never valid to begin with. A survey filled in by a bot, a panel that stopped representing anyone, a tool that counts a success as a failure. In each case the numbers look clean, but the sample underneath is broken. The research methods topic index collects the material behind this post.

Why the costliest error happens before the analysis

A finding only holds when the person behind it was real, willing, and able to answer. If any of the three fails, the analysis carries the flaw with it. A flawless test with the wrong person still leads to the wrong priority. That is how a data problem becomes a product decision, and why selection criteria sit at the base of the whole chain.

Automated answers in surveys and the signs

The NN/g article on survey bots calls a “survey bot” an automated program, or a person with an automation tool, that fills a questionnaire with fraudulent answers. The range runs from random choice to open text that reads as plausible and was written by AI. Many of them clear an attention check, and a modest incentive is enough to attract the traffic. Any survey sitting in an open channel is exposed.

Watch for these signs:

Completion far faster than the questionnaire size would allow.
Open text that stays long, generic, uniform in length, and too polished.
The same IP inside a short window, with the caveat that a shared network creates legitimate duplicates.
A duplicated email, or an address that follows a strange sequence.
A block of answers that falls out of line when the question order is randomized.

One sign on its own proves nothing. Two or three converging signs are usually enough to flag a response. Bots also arrive in batches and improve fast, so a signal that worked in an earlier wave may miss the one arriving today.

The internal participant panel

A panel kept in house fails for three reasons. The base is static, so it ages and nobody renews it, which forces extra screening. Sampling bias creeps in, because people who join a panel already like the company and may praise too much. The same people also learn the product too well to stand in for a first-time user. The panel then drifts from the business reality it was built to reflect, since it froze the market at the moment of its construction.

The fix is clear ownership. Someone has to answer for keeping the data fresh, reviewing the response rate, and renewing the segments, with rotation and occasional recruitment from outside the panel.

The tool that imposes bad method

The research tool itself can have method errors built in. The archive gives examples. One platform, in an interaction test, allows a single success URL and doesn’t randomize the task, so a real success gets counted as a failure. An analysis tool lets you mark text but not video, which fails when the critical moment happens without narration. A product labels everything a discussion guide and blurs an interview with a usability test.

The risk grows in the AI era. Some tools write the task in the site’s own terms and tell the person where to click. That primes the answer and returns a biased result that still looks like solid research. A clean funnel analysis built on that task inherits the same distortion.

Synthetic users and digital twins

The archive reviews three third-party studies. A digital twin replicates human answers well at the individual and group level, which makes it useful for filling a missing response with relatively high accuracy. The simplest method, enriching the prompt with interview data, seems to give the best result. A synthetic user is less convincing: it captures the trend but misses the magnitude and the variability.

The same review of simulation studies reports 78% accuracy when filling a missing response and 67% on a new question, with a population-level correlation of 0.98 for the filling. One study reviewed there ran an AI-conducted interview with 1,052 interviewees, a marketing survey gathered 605 respondents, and a method comparison put accuracy at 0.85 against 0.70 and 0.71. The same review also found a smaller standard deviation in synthetic respondents.

The review warns that the bias depends on the demographic group and the task. It also states that the result doesn’t replace human research, and it raises consent and the loss of agency. Treat a twin as a proxy for a gap you can name. It doesn’t stand in for the user. AI research synthesis covers how to fold that proxy back into a real study.

Selection criteria set before the collection opens

Rigorous selection is the base of external validity. The selection criteria article lists three ways recruitment goes wrong. A poor-fit candidate never matches the profile. A “professional tester” joins everything for the money and stops standing in for an ordinary user. A bad-faith participant lies during screening or uses AI to pass it fast.

The cost shows up in two ways. When the session reveals the problem, you pay the incentive and the time. When it stays hidden, the imprecise data enters the finding and drives the wrong priority. That is the more dangerous case, because nobody sees the loss.

Demographic data, on its own, describes no one. A slice of age, income, and marital status can gather four very different people. Past behavior predicts future behavior better than any profile field. The archive builds around inclusion, exclusion, and diversity, and uses a recruitment matrix to balance the sample instead of a rigid quota. A small five-participant usability test still needs that discipline at the door.

A checklist before you trust the data

Confirm who could have answered and how the invitation was sent.
Check the completion time against the length of the questionnaire.
Read the open text for uniform length and generic phrasing.
Compare IPs, emails, and answer blocks for convergence.
Name the owner of the panel and the renewal date.
Read the tool defaults for a hidden method rule.
Ask what a synthetic fill is standing in for.
Write inclusion, exclusion, and diversity criteria before the first session.

Where the sources disagree

The sharpest conflict sits between the two AI articles. The survey bot article treats a non-human respondent as contamination to remove. The simulation review treats an AI-generated respondent as a legitimate way to fill a missing answer. Same mechanism, opposite value. The distinction both articles hold without stating it is unsolicited fraud versus a deliberate, traceable proxy.
The panel: one article says a well kept panel yields quality participants, and the other warns that a professional tester stops representing an ordinary user, which is exactly the population of a panel.
The framing of AI: the simulation review is fairly optimistic about AI imitating a human, while the tool article treats AI inside a product as a risk of failed research presented with confidence at scale. They don’t contradict each other, because one reviews academic work and the other criticizes a commercial tool, but the text has to pick a frame instead of pasting both side by side.
Inside the survey bot article: bots improve fast, which weakens any fixed screening list, and the same archive proposes a stable method of selection. The article’s answer is to demand a converging signal rather than one fixed tell.