Generation API

Scripts can generate text to use for whatever purpose they need. This is done through the api.v1.generate function. Currently generation is only supported with models that use the OpenAI-like completions endpoint.

Text Generation

Basic Generation

To generate text, you can call the api.v1.generate function with an array of messages that form the prompt for the generation. Each message should be an object with a role property (either “user”, “assistant”, or “system”) and a content property containing the text of the message.

The parameters object can also include generation settings such as model, temperature, max_tokens, and others to control the behavior of the generation. model is required and must be set to a valid model ID.

Example:

let response = await api.v1.generate(
  [
    { role: 'system', content: 'You are a not-very-helpful assistant.' },
    { role: 'user', content: 'Where am I?' }
  ],
  {
    model: 'glm-4-6',
  }
);
api.v1.log('Generated text: ' + response.choices[0].text);

Generation Parameters

The list of available generation parameters includes:

model (string, required)
temperature (number)
max_tokens (number)
top_p (number)
top_k (number)
frequency_penalty (number)
presence_penalty (number)
stop (array of strings)

Streaming Generation

All requests to api.v1.generate will stream the response by default. These streamed values can be accessed by providing a callback function as the third parameter to api.v1.generate. This callback will be called multiple times as new data is received from the generation endpoint.

Example:

await api.v1.generate(
  [
    { role: 'user', content: 'aaaa, nslx, bjal, abcd, or izti?' }
  ],
  {
    model: 'glm-4-6',
  },
  (choices, final) => {
    for (let choice of choices) {
      api.v1.log('Received text chunk: ' + choice.text);
    }
    if (final) {
      api.v1.log('Generation complete.');
    }
  }
);

Blocking and Background Generation

By default, calls to api.v1.generate are considered ‘blocking’ and will prevent interaction with the editor and user-initiated generation until the generation is complete. To allow user interaction during generation, you can set the fourth parameter, behaviour, to 'background'.

Generation Limit

Scripts have a limit on the maximum number of generations they can perform without user interaction. The current limit is 5 generations and is reset whenever the user interacts with the scripts ui or performs a generation in the editor. If a script exceeds this limit, an error will be thrown when attempting to generate.

Editor Generate

Scripts can also trigger text generation directly into the editor using the api.v1.editor.generate function. This functions as if the user had clicked the “Send” button in the editor, generating text based on the current document content and appending the result to the document.