Gemini

Installation

npm i llamaindex @llamaindex/google

pnpm add llamaindex @llamaindex/google

yarn add llamaindex @llamaindex/google

bun add llamaindex @llamaindex/google

Usage

import { gemini, GEMINI_MODEL } from "@llamaindex/google";
import { Settings } from "llamaindex";

Settings.llm = gemini({
  model: GEMINI_MODEL.GEMINI_2_0_FLASH,
});

Usage with Vertex AI

To use Gemini via Vertex AI, you can specify the vertex configuration:

import { gemini, GEMINI_MODEL } from "@llamaindex/google";

const llm = gemini({
  model: GEMINI_MODEL.GEMINI_2_0_FLASH,
  vertex: {
    project: "your-cloud-project",    // required for Vertex AI
    location: "us-central1",          // required for Vertex AI
  },
});

To authenticate for local development:

gcloud auth application-default login

To authenticate for production you'll have to use a service account. googleAuthOptions has credentials which might be useful for you.

Multimodal Usage

Gemini supports multimodal inputs including text, images, audio, and video:

import { gemini, GEMINI_MODEL } from "@llamaindex/google";
import fs from "fs";

const llm = gemini({ model: GEMINI_MODEL.GEMINI_2_0_FLASH });

const result = await llm.chat({
  messages: [
    {
      role: "user",
      content: [
        {
          type: "text",
          text: "What's in this image?",
        },
        {
          type: "image",
          data: fs.readFileSync("./image.jpg").toString("base64"),
          mimeType: "image/jpeg",
        },
      ],
    },
  ],
});

Tool Calling

Gemini supports function calling with tools:

import { gemini, GEMINI_MODEL } from "@llamaindex/google";
import { tool } from "llamaindex";
import { z } from "zod";

const llm = gemini({ model: GEMINI_MODEL.GEMINI_2_0_FLASH });

const result = await llm.chat({
  messages: [
    {
      content: "What's the weather in Tokyo?",
      role: "user",
    },
  ],
  tools: [
    tool({
      name: "weather",
      description: "Get the weather",
      parameters: z.object({
        location: z.string().describe("The location to get the weather for"),
      }),
      execute: ({ location }) => {
        return `The weather in ${location} is sunny and hot`;
      },
    }),
  ],
});

Live API (Real-time Conversations)

For real-time audio/video conversations using Gemini Live API.

The Live API is running directly in the frontend. That's why you have to generate an ephemeral key first on the server side and pass it to the frontend.

To use the Live API, make sure to pass apiVersion: "v1alpha" to the httpOptions.

import { gemini, GEMINI_MODEL } from "@llamaindex/google";

// Server-side: Generate ephemeral key
const serverLlm = gemini({
  model: GEMINI_MODEL.GEMINI_2_0_FLASH_LIVE,
  httpOptions: { apiVersion: "v1alpha" },
});
const ephemeralKey = await serverLlm.live.getEphemeralKey();

// Client-side: Use ephemeral key for Live API
const llm = gemini({
  apiKey: ephemeralKey,
  model: GEMINI_MODEL.GEMINI_2_0_FLASH_LIVE,
  voiceName: "Zephyr",
  httpOptions: { apiVersion: "v1alpha" },
});

const session = await llm.live.connect();

Load and index documents

For this example, we will use a single document. In a real-world scenario, you would have multiple documents to index.

import { Document, VectorStoreIndex } from "llamaindex";

const document = new Document({ text: essay, id_: "essay" });

const index = await VectorStoreIndex.fromDocuments([document]);

Query

const queryEngine = index.asQueryEngine();

const query = "What is the meaning of life?";

const results = await queryEngine.query({
  query,
});

Full Example

import { gemini, GEMINI_MODEL } from "@llamaindex/google";
import { Document, VectorStoreIndex, Settings } from "llamaindex";

Settings.llm = gemini({
  model: GEMINI_MODEL.GEMINI_2_0_FLASH,
});

async function main() {
  const document = new Document({ text: essay, id_: "essay" });

  // Load and index documents
  const index = await VectorStoreIndex.fromDocuments([document]);

  // Create a query engine
  const queryEngine = index.asQueryEngine();

  const query = "What is the meaning of life?";

  // Query
  const response = await queryEngine.query({
    query,
  });

  // Log the response
  console.log(response.response);
}

API Reference

Gemini

Gemini

On this page