Methods

Who took part

Both studies recruited adults who already use AI chatbots and live in one of the 35 states with a U.S. Senate race on the 2026 ballot. Study 1 (all voters) ran on CloudResearch Connect from 16 to 27 October 2026. Using Connect’s screening data, we invited U.S. citizens who said they use generative AI several times a week or daily, with quotas of 500 Democrats, 500 Republicans, and 500 independents or “something else.” We did not screen on how certain people were of their vote, so most had already decided. Study 2 (undecided voters) ran on Prolific from 21 to 27 October 2026. Inside the survey, we let through only participants who were at most moderately certain who they would vote for in the midterm Senate election (1 to 3 on a five-point scale), had not yet voted, and currently used a chatbot occasionally or regularly.

In both studies, participants confirmed they were 18 or older, passed a bot check (saying which of two lines was longer), and named a state with a Senate race; the survey ended for anyone who did not. As preregistered, we then excluded duplicate participant IDs, keeping each person’s first complete response, participants who sent the chatbot no message, and off-task chats (i.e., chats that never mentioned the election, voting, the candidates, or politics). The LLM coder judged whether a chat was off task. That left 1,455 participants in Study 1 and 975 in Study 2. All participants consented to the release of their chats, which we share with identifying details removed. Connect’s quotas used the party on file with the platform, so the party counts below, which participants gave in our survey, are not equal.

What participants did

After the screening questions, participants read that their state had a U.S. Senate race on the 3 November ballot and that they would have the opportunity to use an AI chatbot. In Study 1, the survey assigned each participant at random to ChatGPT, Claude, Gemini, or Grok. In Study 2, participants picked ChatGPT, Claude, or Gemini, as they would when choosing an app. Participants in Study 2 then read:

Below, you have the opportunity to use ChatGPT, an AI chatbot, as you decide how to vote in your U.S. Senate race. It’s the same AI model as the free ChatGPT app, and it hasn’t been told anything about this survey. So use it the way you would on your own. Click “End chat” when you’re done.

The chatbot’s name changed with the participant’s chatbot. Study 1 used the same words, except that the first sentence ended “as you decide how to vote in that race.” After the chat, participants answered the questions about their vote and the chat, then the questions about themselves. In Study 2, age, gender, and race came first, with the screening questions.

What the chatbot saw

The chat window connected through OpenRouter’s API to each company’s current model: GPT-5.6 Luna (openai/gpt-5.6-luna), Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5), Gemini 3.8 Flash (google/gemini-3.8-flash), and, in Study 1 only, Grok 4.3 (x-ai/grok-4.3). The usage log records which model answered every turn. To look like each company’s app, the window showed the chatbot’s name and opened with that app’s greeting: “What can I help with?” for ChatGPT, “How can I help you today?” for Claude, “How can I help you?” for Gemini, and “What do you want to know?” for Grok.

Participants could chat for up to 10 minutes, with a timer showing the time left, and type up to 2,000 characters per message. They could start a new conversation, which cleared the chatbot’s memory of earlier messages. They could also end the chat at any point after sending at least one message. Replies appeared as formatted text with clickable links. In Study 2, replies appeared word by word as they were generated, as they do in the apps. In Study 1, each reply appeared once it was complete. We saved every conversation in full, including messages that failed to send.

The chatbot knew which state the participant lived in, because the survey passed the state the participant had reported to the chat service, much as a consumer app knows a user’s approximate location. When a participant’s first message did not name a state, 88% of first replies named the participant’s state anyway (n = 1,327 chats). The usage logs also suggest that ChatGPT, Claude, and Grok searched the web before answering, since a one-line first message typically reached these models with thousands of tokens of added context. Gemini’s logs show no such addition.

What we asked after the chat

• Vote intention: “If the U.S. Senate election in your state were held today, who would you vote for?” Participants typed a name or chose “I’m still undecided,” “I don’t plan to vote in this race,” or “I already voted.” Those who named a candidate gave the candidate’s party and how certain they were (1 to 5). Those who had not voted rated how likely they were to vote (−3 to 3).
• Preparation: how chatting with the chatbot affected how prepared they felt to vote, from much less prepared (−3) to much more prepared (3).
• Perceived bias: how politically biased the chatbot seemed, from strongly biased towards Democrats (−3) through not biased (0) to strongly biased towards Republicans (3).
• Similarity to their own use: how similar the chat was to how they would use AI on their own to look into their Senate race, from very different (−3) to very similar (3), or “I would never use AI for this.”
• About themselves: how often they use AI chatbots and follow political news, whether they had used AI to learn about politics before, their party (and, for independents, which party they feel closer to), their political views from extremely liberal to extremely conservative, age, gender, and race or ethnicity. Study 2 also asked how much they trust AI chatbots, whether they voted in the 2024 presidential election, and two political knowledge questions: a senator’s term length and the number of senators per state.

How we coded the chats

1. Hold outSet aside a random 100 chats per study before building the codebook.
2. Derive the codebookAn LLM read the other chats in three independent passes; the three codebooks were merged into one.
3. Pilot and freezeWe edited the draft, piloted it on 50 chats per study, revised it, and froze it.
4. Code every messageThe LLM coded every participant message three times with the evidence quoted; we kept the majority code.
5. Code turns and repliesIn a separate request, the LLM coded how each message related to the last reply, disclosures, topics, and what the chatbot did.

The coder was Claude Opus 5.5 (claude-opus-5-5), called through Anthropic’s API at medium reasoning effort, with its answers held to a fixed JSON format. It coded only what participants wrote, but it read the chatbot’s replies too, because a follow-up such as “tell me more about the second one” only makes sense next to the reply it answers. It did not see which chatbot the participant used, their demographics, or their survey answers. Before any text went to the API, we replaced ZIP codes, street addresses, and intersections with placeholders such as “[ZIP CODE],” including where the chatbot repeated them.

Building the codebook. Because both studies ask the same question, we built one codebook from both, so that approaches can be compared across studies. It came from 2,268 chats (1,372 in Study 1 and 896 in Study 2), leaving out the held-out chats. We defined a prompting approach as what a participant was trying to get from the chatbot and how they went about getting it: what they said about themselves, how they framed their requests, and how they followed up on the replies. We told the LLM to find approaches rather than topics, so that questions about different issues or candidates would not become separate codes, and we gave it no example categories, so that the approaches would come from the chats.

The LLM read these chats in three independent passes. Each pass shuffled each study’s chats and split them into sets of about 54 chats (26 sets from Study 1 and 16 from Study 2). For each set, the LLM listed the approaches that recurred across chats, giving for each a definition, the signs of it in participants’ messages, how it differs from the most similar approach, verbatim examples, and how often it appeared, and it listed any on-task chats that fit none. A separate call combined each pass’s sets into one codebook, and a final call merged the three, keeping approaches that appeared in at least two passes unless an approach was clearly distinct and important. We checked every example quote against the transcripts. All 12 approaches in the draft emerged in all three passes and in both studies.

We reviewed and edited the draft and applied it to 50 pilot chats from each study. Reading the pilot codes led to two changes. We added a thirteenth approach, a vague request for help deciding, which the LLM had not proposed, and a rule that a message which only supplies context the chatbot asked for, such as a state or a race, takes the approach of the request it completes. We then froze the codebook.

Coding every message. The LLM coded every participant message with the whole chat as context. A message could show several approaches or none (“ok thanks”), and an approach built over several messages, such as describing oneself in one message and asking for a match in the next, was coded on each message that contributed to it. For every code it applied, the LLM quoted the participant’s words as evidence; the explorer shows these quotes. It also judged whether the chat as a whole was off task. The LLM coded every chat three times independently, and we kept each code that at least two of the three runs gave. A chat has an approach if any of its messages does, with one exception: a vague request for help counts at the chat level only when the chat has no other approach. An on-task chat with no approach in any message counts as Other.

Coding turns and replies. In a separate request, so that it could not change the approach codes, the LLM coded three more things about each participant message: how it related to the chatbot’s previous reply (eight categories, such as following up, answering the chatbot’s question, or pushing back after a decline), what the participant disclosed about themselves (fourteen categories), and the topics they raised (fifteen categories). We derived the disclosure and topic categories from the chats with the same three-pass procedure. The LLM also coded seven behaviours in every chatbot reply: giving factual information, recommending a candidate outright, matching the participant to a candidate based on what they said, declining to recommend, presenting several sides, asking about the participant’s preferences, and pointing to outside sources, along with the candidate and party of any recommendation or match. When every question in a reply asked only which race, office, or place the participant meant, we counted the reply as giving information rather than asking about preferences. This coding also ran three times, with a majority vote.

The codebook

Each of the 13 approaches captures what the participant was trying to get from the chatbot and how they went about it. Below is the first sentence of each definition. The full definitions are in the dataset.

How consistently the LLM coded

We compared the LLM’s three runs on every coded chat. Across the 13 approaches, Fleiss’ κ ranged from 0.91 to 0.98 at the message level and from 0.90 to 0.99 at the chat level, in both studies. For the disclosures, topics, and reply behaviours, κ ranged from 0.83 to 1.00, leaving out the two catch-all “other” categories, which applied to under 2% of messages. Agreement across runs shows that the coder applied the codebook consistently. To judge whether its codes are right, compare them with the evidence quotes in the explorer. Two researchers are now coding the held-out chats by hand, and we will add their agreement with the LLM when it is done.

Preregistration and analysis

Both studies were preregistered on AsPredicted: Study 1 on 16 October 2026 (#317878) and Study 2 on 21 October 2026 (#319077), each before its data collection began. The preregistrations cover the recruitment and exclusion rules above, the two-step LLM coding of prompting approaches, the share of participants using each approach, and how those shares differ by age, state, party, political knowledge, and prior AI use. Study 1’s also covers the effect of chatbot brand on vote intention, the party of the intended candidate, certainty, likelihood of voting, perceived preparation, and perceived bias. As preregistered, we repeated the analyses among participants who said the chat was similar to their own use (1 to 3 on the similarity scale) and, in Study 1, among participants who had not yet voted.

We report each share with a 95% Wilson interval. For group differences, we fit a logistic regression for each approach with one factor at a time, tested each factor with a likelihood-ratio test, and applied a Holm correction across all tests within each study. For brand effects in Study 1, we regressed each outcome on chatbot brand with state fixed effects and robust (HC1) standard errors, using linear models for the rating scales and logistic models for the yes-or-no outcomes. We tested brand with a joint Wald test, followed by pairwise comparisons with Holm-adjusted p-values. Study 2 did not test brand effects, because participants chose their chatbot.

The analyses go beyond the preregistrations in three places. First, the LLM coded each message, and the preregistered chat-level codes are derived from those message codes. Second, Study 1’s survey had no political knowledge questions, so the comparison by political knowledge is possible only in Study 2. Third, the analyses of how conversations unfold, what participants disclosed, the topics they raised, and what the chatbots did were not preregistered and are exploratory.