AI testing: an AI model checks and uses your site

Has an AI model look at the page as a tester would (CheckWithAi), or use the site with a purpose, e.g. fill in forms with valid data and go through a checkout (ActWithAi), and logs the problems it finds. Needs an API key: --plugin name=Ai apiKey=... (for all of them).

This is Elsetest's Ai plugin. It runs as 2 instances, each with an alias and options of its own (below). Like every plugin, it gets its turn at random: in each step of a test, one plugin instance is picked, weighted by its frequency. Raise it to check more often.

Getting started

Its two instances do different things. CheckWithAi looks at the window as a tester would and reports what looks wrong. ActWithAi uses the site with a goal of its own: it fills in a form with valid data, searches, or goes through a checkout, to open the parts of the site that random clicks rarely reach, and lets the other plugins loose there. Both are off until you give an API key, and one entry with "name": "Ai" gives it to both.

The easiest way is setup: it asks which AI service you use, your key and the model, and tries them on your site before it writes the config file:

elsetest --setup --url staging.example.com

Or set it yourself, in a config file:

{
  "plugins": [
    {
      "name": "Ai",
      "provider": "anthropic",
      "apiKey": "sk-ant-...",
      "model": "claude-sonnet-5-5"
    }
  ]
}

or on the command line: --plugin name=Ai apiKey=sk-ant-... (Anthropic and its default model).

AI services

Elsetest works with Anthropic's API and any API compatible with OpenAI's, including a model on your own machine. These are the services setup offers, and the options each needs besides apiKey and model:

Anthropic (Claude)

provider=anthropic

Get a key. Model e.g. claude-opus-5-5. claude-sonnet-5-5 and claude-haiku-5-5 cost less.

OpenAI

provider=openai

Get a key. Model e.g. gpt-5-mini.

Azure OpenAI

provider=openai
baseUrl=https://<resource>.openai.azure.com/openai/v1

Needs a key. The model is the name of a deployment in your Azure OpenAI resource, e.g. gpt-5-mini.

Google Gemini

provider=openai
baseUrl=https://generativelanguage.googleapis.com/v1beta/openai

Get a key. Model e.g. gemini-flash-latest.

OpenRouter

provider=openai
baseUrl=https://openrouter.ai/api/v1

Get a key.

Ollama, on this machine

provider=openai
baseUrl=http://localhost:11434/v1

No key needed. Pick a model that takes images and doesn't think (e.g. qwen3-vl:8b-instruct), with a context of at least 16k tokens (PARAMETER num_ctx 16384 in a Modelfile).

LM Studio, on this machine

provider=openai
baseUrl=http://localhost:1234/v1

No key needed. Pick a model that takes images and doesn't think, loaded with a context of at least 16k tokens.

Another API compatible with OpenAI's

provider=openai
baseUrl=<the API's base URL>

A key if the API needs one.

The model needs to understand images, since it's sent a screenshot of the window. You don't need to set locate: setup chooses it from the model's name, and the default works with any model.

What it costs, and what is sent

Your AI service bills each call to the model on your account with them; Elsetest's testing time doesn't change. maxCalls limits the calls per run (default 50), and the log has each call's tokens and time and the run's total. Each call sends a screenshot of the window and the page's text to the service; without a key, nothing is sent. A model on your own machine (Ollama, LM Studio) keeps it all there.

Options

Set them for one instance with --plugin alias=<alias> key=value, or for every instance of the plugin with --plugin name=Ai key=value. In a config file, each --plugin is an object in "plugins", e.g.:

Most sites need only apiKey and model, and for some services provider and baseUrl (above). The others tune how the model works.

A selector says which elements of the page an option is about, e.g. #chat-widget or .cookie-banner button: see Selectors.

`frequency`

--plugin alias=CheckWithAi frequency=<level>

How often the plugin runs compared to the others.

One of off, lowest, low, normal, high, highest. Default: normal. The default instances set their own (below).

`task`

--plugin alias=CheckWithAi task=<task>

check: look at the window and report problems. act: use the site the way a user would, with a goal, for a few steps in a row, and report problems.

One of check, act. Default: check.

`provider`

--plugin alias=CheckWithAi provider=<provider>

anthropic for Anthropic's Claude models, or openai for any API compatible with OpenAI's: OpenAI, Google Gemini, OpenRouter, Mistral, Groq, or a model on your own machine with Ollama or LM Studio (set baseUrl).

One of anthropic, openai. Default: anthropic.

`apiKey`

--plugin alias=CheckWithAi apiKey=<key>

The API key from your account with the provider. Without one, the plugin is off (except with provider openai and a baseUrl, for a local model). The page's screenshot and text are sent to the provider.

`model`

--plugin alias=CheckWithAi model=<model>

The model to use. With provider anthropic, the default is claude-opus-5-5; claude-sonnet-5-5 and claude-haiku-5-5 cost less. With provider openai, it must be set.

`baseUrl`

--plugin alias=CheckWithAi baseUrl=<url>

The API's base URL, for another provider than OpenAI with provider openai (e.g. https://generativelanguage.googleapis.com/v1beta/openai, https://openrouter.ai/api/v1, http://localhost:11434/v1 for Ollama), or a proxy. With Ollama, give the model a context of at least 16k tokens (Ollama's default, 4096, is too small: PARAMETER num_ctx 16384 in a Modelfile, or OLLAMA_CONTEXT_LENGTH for the server) and pick one without thinking (e.g. qwen3-vl:8b-instruct rather than qwen3-vl:8b), since thinking makes it slow and it may give no answer.

`reasoning`

--plugin alias=CheckWithAi reasoning=<level>

How much the model thinks before answering. Less is faster and costs less, more may judge better. Each provider's own levels differ, so these are translated (lowest is OpenAI's minimal, normal its medium, highest its xhigh); a level the model doesn't take is lowered to the next one it does, or left to the model.

One of lowest, low, normal, high, highest. Default: the model's own.

`check`

--plugin alias=CheckWithAi check=true|false

With task act: also report the problems the model sees as it uses the site, especially those only it can tell, knowing what it meant to happen (valid input rejected, something saved that isn't shown afterwards). Off, it only acts and task check reports problems, since a model reasoning little (reasoning lowest, ActWithAi's default) reports more guesses and repeats.

Default: false.

`screenshot`

--plugin alias=CheckWithAi screenshot=true|false

Send a screenshot of the window along with the page's text. Without it the model sees only the text, which costs less and finds fewer problems.

Default: true.

`locate`

--plugin alias=CheckWithAi locate=<how>

How the model points at what to act on and what has a problem. element: by its number in a list of the window's elements, sent along with the screenshot. marks: with task act, as element, and the links, buttons and form fields are also marked with their numbers in the screenshot, which helps models that mix up which number is which element; with task check, it's element, since marks would cover what it looks for. pixels: by x and y in the screenshot's pixels, for models trained for computer use that point so (Claude, Fara); thousandths: by x and y from 0 to 1000 of its width and height (Qwen3-VL, Gemini). With pixels or thousandths, only the screenshot is sent, at most 1280 pixels wide, which makes each call much shorter. computer: with task act and provider openai, through OpenAI's computer-use tool in its Responses API (e.g. gpt-5.4-mini, also on Azure), the way those models are trained to use a computer; with task check, it's pixels.

One of element, marks, pixels, thousandths, computer. Default: element.

`maxCalls`

--plugin alias=CheckWithAi maxCalls=<number>

The most times per test run the model is asked, to limit what the run costs. With task check, the page is only looked at again when it has changed.

Default: 50.

`pause`

--plugin alias=CheckWithAi pause=<probability>

With task act: the probability that a flow pauses after a step, so that random use (favouring a dialog the flow has opened) tests where it has got, before the flow goes on at its next turn. 0 never pauses.

Default: 0.3.

`maxSteps`

--plugin alias=CheckWithAi maxSteps=<number>

With task act: the most steps (each one call to the model) in a row for one goal. With locate computer, whose steps do less, 3 times as many.

Default: 6.

`ignoreSelector`

--plugin alias=CheckWithAi ignoreSelector=<selector>

With task act: don't act on elements matching this CSS selector, or inside them; replaces the global ignoreSelector (not the data-elsetest-ignore attribute, which always applies). The site can also mark the elements in its HTML with the attribute data-elsetest-ai-ignore.

Default: the global ignoreSelector.

`ignoreText`

--plugin alias=CheckWithAi ignoreText=<regex>

With task act: don't act on links, buttons and other controls whose text or label matches this regular expression (not case-sensitive), e.g. "log ?out|delete"; replaces the global ignoreText.

Default: the global ignoreText.

Default instances

Alias	The options it sets
`CheckWithAi`	`task=check` `frequency=lowest`
`ActWithAi`	`task=act` `frequency=low` `reasoning=lowest`

Other plugins

Accessibility · BrokenText · Click · Cluster · History · Login · MutationCluster · Overlap · Scroll · SelectOption · Spelling · Type · Validate

This page is generated from the help texts of Elsetest 0.9.8. The same help is in the program: elsetest --help.