Document and Nodes

We offer readers for different file formats.

import { CSVReader } from '@llamaindex/readers/csv'
import { PDFReader } from '@llamaindex/readers/pdf'
import { JSONReader } from '@llamaindex/readers/json'
import { MarkdownReader } from '@llamaindex/readers/markdown'
import { HTMLReader } from '@llamaindex/readers/html'
// you can find more readers in the documentation

SimpleDirectoryReader

SimpleDirectoryReader is the simplest way to load data from local files into LlamaIndex.

import { SimpleDirectoryReader } from "@llamaindex/readers/directory";
 
const reader = new SimpleDirectoryReader()
const 
const documents: Document<Metadata>[]documents = await reader.loadData("./data")
 
 
const 
const texts: string[]texts = documents.map(doc => doc.getText())
 

Tips when using in non-Node.js environments

When using @llamaindex/readers in a non-Node.js environment (such as Vercel Edge, Cloudflare Workers, etc.) Some classes are not exported from top-level entry file.

The reason is that some classes are only compatible with Node.js runtime, (e.g. PDFReader) which uses Node.js specific APIs (like fs, child_process, crypto).

If you need any of those classes, you have to import them instead directly through their file path in the package.

As the PDFReader is not working with the Edge runtime, here's how to use the SimpleDirectoryReader with the LlamaParseReader to load PDFs:

import { SimpleDirectoryReader } from "@llamaindex/readers/directory";
import { LlamaParseReader } from "@llamaindex/cloud";
 
export const DATA_DIR = "./data";
 
export async function getDocuments() {
  const reader = new SimpleDirectoryReader();
  // Load PDFs using LlamaParseReader
  return await reader.loadData({
    directoryPath: DATA_DIR,
    fileExtToReader: {
      pdf: new LlamaParseReader({ resultType: "markdown" }),
    },
  });
}

Note: Reader classes have to be added explicitly to the fileExtToReader map in the Edge version of the SimpleDirectoryReader.

You'll find a complete example with LlamaIndexTS here: https://github.com/run-llama/create_llama_projects/tree/main/nextjs-edge-llamaparse

Load file natively using Node.js Customization Hooks

We have a helper utility to allow you to import a file in Node.js script.

node --import @llamaindex/readers/node ./script.js

import csv from './path/to/data.csv';
 
const text = csv.getText()