AI Citation Tracking: Why Repeated Measurements Matter

AI citation tracking from one answer can misclassify a real opportunity or create a false impression of progress. Three public accounts describe changing sources and inconsistent inclusion. A useful measurement plan records repeated answers, collection conditions and missing results before comparing periods.

The lesson

Measure how often a brand or URL appears across repeated comparable answers, rather than treating one answer as a fixed ranking.

What people reported

These public accounts describe different setups. Read each reported outcome with its design limits; repeated descriptions of the same campaign are not independent replications.

u/-amphisbaena: Repeated queries returned changing sources

Reddit user u/-amphisbaena reports tracking 20 queries across ChatGPT, Gemini and Perplexity for nine days. They noticed different cited sources between runs and described differences across tools. Logged-in sessions and changes to the collection setup limit interpretation; the post does not prove a time-of-day mechanism or a general platform stability ranking. Read the original report: u/-amphisbaena — Reddit

The account reports repeated observations, but it also describes collection conditions that changed. Login state or the retrieval method may affect what can be observed. Preserve platform, account state, location where known and whether web search ran; otherwise, apparent temporal change can be partly a collection artifact.

u/EmbarrassedBuddy9743: A single pass misclassified content opportunities

In an r/aeo comment, u/EmbarrassedBuddy9743 says repeated work revealed prompts switching between apparently empty citation opportunities and strong incumbent sources. They describe running each prompt at least five times and assigning a majority category. This is a reported practical correction, not evidence that five runs are statistically sufficient for every question. Read the original report: u/EmbarrassedBuddy9743 — Reddit

Repeating a prompt can reveal that a one-off absence was unstable. The commenter's preferred number of runs is a workflow choice, not a validated sample size for every platform. Decide how much uncertainty is acceptable for the decision you need to make, and avoid cherry-picking the answer containing your brand.

Dana Billingsley: Initial inclusion did not always last

Dana Billingsley comments on Kim Huynh’s experiment that she has seen content appear quickly in AI answers and then disappear, while other sources were reused. She provides no sample counts or measurement schedule. Her qualitative experience is consistent with treating first inclusion as provisional, while leaving the reasons for retention unresolved. Read the original report: Kim Huynh — LinkedIn

Initial inclusion is not the same as persistence. Follow an identical prompt set over scheduled dates, retaining both citations and absences. This comment is an attributed observation, not a documented estimate of citation survival or a reason to assume that every initial citation will disappear.

What the experiences have in common

The shared problem is a changing observation being treated as a fixed property of a website. A team can find its brand once, celebrate an improvement and then conclude that a later absence means a penalty. Without repeated comparable measurements, either conclusion can be premature.

Create separate fields for a brand mention, an explicit recommendation and a linked source. A response can contain one without the others. Also separate a new conversation from an ongoing conversation, since the user’s preceding question changes the task. The measurement should describe the surface actually tested rather than implying it represents every customer conversation.

What these reports cannot establish

These sources are self-reports, including two Reddit pseudonyms and a short LinkedIn comment. No common prompt set or verified dataset is available. Changing retrieval results, model behavior, geography, account context and collection errors are possible explanations. A correlation with run time does not establish that time caused the change.

Define the unit of measurement before counting citations

Count presence at most once per eligible response when estimating citation inclusion. Repeated links in one answer should not inflate the rate. Decide whether you count a brand mention, your domain or a particular URL; these outcomes answer different questions. Report retrieval failures separately from successful answers without a citation.

A Meikai repeated-prompt analysis examines how additional runs affect its own visibility measurements. It is vendor research, not a universal sampling rule. Use a pilot to observe variability in your prompt set, then choose runs and dates that fit the precision and cost your decision requires.

A test you can run: proposed protocol

Use the following protocol as a starting design. Choose one outcome and a practical review window before making changes, and retain the original observations so a disappointing result remains reportable.

1. Fix a question set covering real buyer intents. Store exact wording, language, country context, product surface, visible model label and account state.
2. Run each question repeatedly in comparable fresh sessions over at least two weeks, using permitted manual access or authorized collection. Keep follow-up conversations as a separate cohort.
3. Save the answer, cited URLs, timestamp and any retrieval failure. Count errors separately instead of recording them as brand absence.
4. Calculate mention and citation frequency per question, then summarize across questions with the same intent. Show the number of valid observations.
5. Before crediting an intervention, compare its movement with unchanged questions and the baseline range. Repeat an apparent win before using it in a forecast.

Conclusion

These reports support repeated, documented observations rather than decisions from screenshots. They do not establish one ideal number of runs or prove that every change reflects a content intervention. Prompt and collection differences can resemble a visibility gain or loss.

Begin with a small fixed prompt set and record complete answers over several dates. Report inclusion counts, eligible responses and collection failures together. Investigate only changes that remain meaningful under consistent conditions, then connect them to referrals or reader outcomes before expanding the monitoring workload.