JSONReaderOptions

Defined in: packages/readers/src/json.ts:10

Properties

streamingThreshold?

optional streamingThreshold: number

Defined in: packages/readers/src/json.ts:18

The threshold for using streaming mode. Give the approximate size of the JSON data in MB. Estimates character length by calculating: "(streamingThreshold * 1024 * 1024) / 2" and comparing against string.length Streaming mode avoids memory issues when parsing large JSON data. Set "undefined" to disable streaming or "0" to always use streaming.

Default

50 MB

ensureAscii?

optional ensureAscii: boolean

Defined in: packages/readers/src/json.ts:26

Whether to ensure only ASCII characters. Converts non-ASCII characters to their unicode escape sequence.

Default

false

isJsonLines?

optional isJsonLines: boolean

Defined in: packages/readers/src/json.ts:35

Whether the JSON is in JSON Lines format. Split into lines, remove empty lines, parse each line as JSON. Note: Uses a custom streaming parser, most likely less robust than json-ext

Default

false

cleanJson?

optional cleanJson: boolean

Defined in: packages/readers/src/json.ts:43

Whether to clean the JSON by filtering out structural characters ({}, [], and ,). If set to false, it will just parse the JSON, not removing structural characters.

Default

true

levelsBack?

optional levelsBack: number

Defined in: packages/readers/src/json.ts:51

Specifies how many levels up the JSON structure to include in the output. cleanJson will be ignored. If set to 0, all levels are included. If undefined, parses the entire JSON and treats each line as an embedding.

Default

undefined

collapseLength?

optional collapseLength: number

Defined in: packages/readers/src/json.ts:59

The maximum length of JSON string representation to be collapsed into a single line. Only applicable when levelsBack is set.

Default

undefined

logger?

optional logger: Logger

Defined in: packages/readers/src/json.ts:65

A placeholder for a custom logging function.

Default

consoleLogger