diff --git a/web/TEST_RESULTS.md b/web/TEST_RESULTS.md new file mode 100644 index 0000000000000000000000000000000000000000..1e475760a88b3e0918562da41cdf8c495331a806 --- /dev/null +++ b/web/TEST_RESULTS.md @@ -0,0 +1,93 @@ +# Test results — 2026-07-15 + +All suites run with `npm test` (chains all nine). Every suite exits 0. +Hardware for GPU numbers: NVIDIA via WebGPU, DP4A int8 dot path, exact-gated +against the verified units at init. + +| suite | what it proves | result | +|---|---|---| +| `test_core.js` | float trainer converges, replicas bit-identical | PASS — loss 33.71 → 0.000000, replica diff 0.000e+0 | +| `test_verified.js` | training THROUGH the int8 units converges, replicas bit-identical | PASS — loss 300.6 → 1.41, replica diff 0.000e+0 | +| `test_ieee.js` | the JS epilogue mirror is IEEE-754 spec-correct, not merely agreeable | PASS — 500k+ checks, 0 disagreements; rejects the old round-once mirror on 34% of inputs | +| `test_gates.js` | the exact kernel gate rejects real bugs, accepts the real kernel | PASS — 5/5 injected bugs rejected, clean kernel accepted | +| `test_metamorphic.js` | correctness properties that need no reference implementation | PASS — 6/6 properties hold; catches stride/swap/acc= mutants | +| `test_corpus.js` | mutation-scores the oracles with an external bug taxonomy | PASS — properties 2/4 (2/2 loop, 0/2 math), differential 4/4, control clean | +| `test_optimizer.js` | DaisyAdam beats SGD through the units, deterministic replicas | PASS — 1.59 vs 1.95, replica diff 0.000e+0 | +| `test_transformer.js` | the transformer LM trains through the units end to end | PASS — loss 4.75 → 1.26 (baseline 4.56), replica diff 0.000e+0 | +| `test_unit_backward.js` | int8 STE gradients do not damage convergence | PASS — units/float loss ratio 1.007 | + +## The IEEE-754 oracle (`test_ieee.js`) + +Built from the binary32 definition in exact BigInt arithmetic — no +`Math.fround` anywhere in the oracle, so neither side was tuned to the other. + +``` +i32ToF32Spec matches Math.fround on 200000 random int32 (incl. |s| > 2^24) +mulF32Spec matches the correctly-rounded product on 300000 draws (subnormal..overflow) +Verified.epi vs the oracle: 200000 random triples, 0 disagreements +tie-to-even ladder around 2^24: 378 cases, 0 disagreements +bgemmJS outputs rebuilt from the raw int32 accumulator via the oracle: 0 disagreements +the oracle REJECTS the round-once mirror shipped before: 68314/200000 inputs (34.16%) +``` + +That last line is the teeth: an oracle that never disagrees with anything +proves nothing. This one rejects the exact bug the old `1e-6` tolerance hid. + +## Oracle mutation scores (`test_corpus.js`) + +Bugs ported from an external taxonomy +([dipankarsarkar/gpuemu-corpus](https://huggingface.co/datasets/dipankarsarkar/gpuemu-corpus)) +so the bug list has a different author than the checks. + +| bug | lives in | properties | differential | +|---|---|---|---| +| `acc=` instead of `acc+=` | loop | CAUGHT (sensitivity) | CAUGHT | +| missing bounds guard (mult-of-8) | loop | CAUGHT (nonTriviality) | CAUGHT | +| dropped constant factor (2×) | math | missed | CAUGHT | +| wrong leaky-ReLU alpha | math | missed | CAUGHT | + +The split is the finding: properties are a **loop-bug detector** (2/2), blind +to structure-preserving value bugs (0/2). Those are owned by the differential +gate against the spec-correct mirror. Each check class has an owner. + +## Backward rework: bit-identity + GPU wall clock + +The backward was reworked for dispatch efficiency: independent GEMMs +overlapped, the QKV weight-gradient and dln1in trios fused into single +batched (batch=3) GEMMs, and the `g.emb` operand quantized column-wise in one +pass instead of transpose-then-quantize. **None of this may change a bit** — +block scales are per-row/per-column per batch element, so fusion is exact, +and every fused sum keeps the original per-add f32 rounding schedule. + +Bit-identity, old backward vs new (CPU mirrors, Node): + +``` +char-96, float backward: loss + all 20480 gradient floats bit-identical +char-96, unit backward: loss + all 20480 gradient floats bit-identical +Spikewhale 16k, float backward: loss + all 545792 gradient floats bit-identical +Spikewhale 16k, unit backward: loss + all 545792 gradient floats bit-identical +``` + +GPU (c=32 t=32 b=8 layers=2 heads=2, 16512-token vocab, 15 measured steps, +gradient FNV hash compared old vs new on-device): + +| config | ms/step | grad hash | +|---|---|---| +| old backward, float | 286 | `7ff11308` | +| old backward, units | 466 | `62596547` | +| new backward, float | 286 | `7ff11308` (identical) | +| new backward, units | 346 | `62596547` (identical) | + +- float path: unchanged speed, unchanged bits — this is what ships enabled. +- unit-backward path (dormant, `cfg.unitBackward`): cost drops **1.63× → 1.21×** + vs float. Because the rework is bit-identical, the convergence curves from + the unit-backward experiment stand unchanged; only the wall-clock exchange + rate moved. At equal wall clock float still wins (~1.5% at the 200-step + horizon), so `unitBackward` stays off by default. + +One bug was caught during the rework, by the bit-identity check itself: the +fused q+k+v sum initially ran in f64 and rounded once, where the old code +rounded to f32 after each add — a last-ulp fork that would have split +replicas. Fixed by matching the rounding schedule (`Math.fround` per add). +Same lesson as the epilogue mirror: match the rounding schedule, not just +the values. diff --git a/web/node_modules/.package-lock.json b/web/node_modules/.package-lock.json new file mode 100644 index 0000000000000000000000000000000000000000..11868836a3f8ebcea7a0af9023cd86698b292bb5 --- /dev/null +++ b/web/node_modules/.package-lock.json @@ -0,0 +1,35 @@ +{ + "name": "daisychain-web", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "node_modules/hyparquet": { + "version": "1.26.2", + "resolved": "https://registry.npmjs.org/hyparquet/-/hyparquet-1.26.2.tgz", + "integrity": "sha512-6qjyK7R2tZ4vnYP1J8NpcCeDPK1UIYk3xh5XgtQUnUM1iwkYbvI6Mz3xtmpMd6AfCCeoUoJVbEq29k5GReZRWA==", + "license": "MIT" + }, + "node_modules/ws": { + "version": "8.21.0", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz", + "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + } + } +} diff --git a/web/node_modules/hyparquet/LICENSE b/web/node_modules/hyparquet/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..5523c3692d69ffe7c6d7cb7bd90184ac7507cc85 --- /dev/null +++ b/web/node_modules/hyparquet/LICENSE @@ -0,0 +1,7 @@ +The MIT License (MIT) + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/web/node_modules/hyparquet/README.md b/web/node_modules/hyparquet/README.md new file mode 100644 index 0000000000000000000000000000000000000000..068e4d2cbadf11778cfcf3bc4a49d5dca55643f5 --- /dev/null +++ b/web/node_modules/hyparquet/README.md @@ -0,0 +1,248 @@ +# hyparquet + +![hyparquet parakeet](hyparquet.jpg) + +[![npm](https://img.shields.io/npm/v/hyparquet)](https://www.npmjs.com/package/hyparquet) +[![minzipped](https://img.shields.io/bundlephobia/minzip/hyparquet)](https://www.npmjs.com/package/hyparquet) +[![workflow status](https://github.com/hyparam/hyparquet/actions/workflows/ci.yml/badge.svg)](https://github.com/hyparam/hyparquet/actions) +[![mit license](https://img.shields.io/badge/License-MIT-orange.svg)](https://opensource.org/licenses/MIT) +![coverage](https://img.shields.io/badge/Coverage-92-darkred) +[![dependencies](https://img.shields.io/badge/Dependencies-0-blueviolet)](https://www.npmjs.com/package/hyparquet?activeTab=dependencies) + +Dependency free since 2023! + +## What is hyparquet? + +**Hyparquet** is a JavaScript library for parsing [Apache Parquet](https://parquet.apache.org) files in the browser. Apache Parquet is a popular columnar storage format that is widely used in data engineering, data science, and machine learning applications for storing large datasets. Hyparquet is designed to read parquet files efficiently over http, so that parquet files in cloud storage can be queried directly from the browser without needing a server. + +> Part of **[HypStack](https://hypstack.ai/)**, an open-source stack for AI observability. + + - Works in browsers and node.js + - Pure JavaScript, no dependencies + - Supports all parquet types, encodings, and compression codecs + - Minimizes data fetching using HTTP range requests + - Includes TypeScript definitions + +## Parquet Viewer + +**Try hyparquet online**: Drag and drop your parquet file onto [hyperparam.app](https://hyperparam.app) to view it directly in your browser. This service is powered by hyparquet's in-browser capabilities. + +[![hyperparam parquet viewer](./hyperparam.png)](https://hyperparam.app/) + +## Quick Start + +### Browser Example + +In the browser use `asyncBufferFromUrl` to wrap a url for reading asynchronously over the network. +It is recommended that you filter by row and column to limit fetch size: + +```javascript +const { asyncBufferFromUrl, parquetReadObjects } = await import('https://cdn.jsdelivr.net/npm/hyparquet/src/hyparquet.min.js') + +const url = 'https://hyperparam-public.s3.amazonaws.com/bunnies.parquet' +const file = await asyncBufferFromUrl({ url }) // wrap url for async fetching +const data = await parquetReadObjects({ + file, + columns: ['Breed Name', 'Lifespan'], + rowStart: 10, + rowEnd: 20, +}) +``` + +### Node.js Example + +To read the contents of a local parquet file in a node.js environment use `asyncBufferFromFile`: + +```javascript +const { asyncBufferFromFile, parquetReadObjects } = await import('hyparquet') + +const file = await asyncBufferFromFile('example.parquet') +const data = await parquetReadObjects({ file }) +``` + +Note: hyparquet is published as an ES module, so dynamic `import()` may be required for old versions of node. + +## Parquet Writing + +To create parquet files from javascript, check out the [hyparquet-writer](https://github.com/hyparam/hyparquet-writer) package. + +## Advanced Usage + +### Reading Metadata + +You can read just the metadata, including schema and data statistics using the `parquetMetadataAsync` function. This is useful for getting the schema, number of rows, and column names without reading the entire file. + +```javascript +import { parquetMetadataAsync, parquetSchema } from 'hyparquet' + +const file = await asyncBufferFromUrl({ url }) +const metadata = await parquetMetadataAsync(file) +// Get total number of rows (convert bigint to number) +const numRows = Number(metadata.num_rows) +// Get nested table schema +const schema = parquetSchema(metadata) +// Get top-level column header names +const columnNames = schema.children.map(e => e.element.name) +``` + +### AsyncBuffer + +Hyparquet requires an argument `file` of type `AsyncBuffer`. An `AsyncBuffer` is similar to a js `ArrayBuffer` but the `slice` method can return async `Promise`. This makes it a useful way to represent a remote file. + +```typescript +type Awaitable = T | Promise +interface AsyncBuffer { + byteLength: number + slice(start: number, end?: number): Awaitable +} +``` + +In most cases, you should probably use `asyncBufferFromUrl` or `asyncBufferFromFile` to create an `AsyncBuffer` for hyparquet. + +#### asyncBufferFromUrl + +If you want to read a parquet file remotely over http, use `asyncBufferFromUrl` to wrap an http url as an `AsyncBuffer` using http range requests. + + - Pass `requestInit` option to provide additional fetch headers for authentication (optional) + - Pass `byteLength` if you know the file size to save a round trip HEAD request (optional) + +```typescript +const url = 'https://s3.hyperparam.app/wiki_en.parquet' +const requestInit = { headers: { Authorization: 'Bearer my_token' } } // auth header +const byteLength = 415958713 // optional +const file: AsyncBuffer = await asyncBufferFromUrl({ url, requestInit, byteLength }) +const data = await parquetReadObjects({ file }) +``` + +#### asyncBufferFromFile + +If you are in a node.js environment, use `asyncBufferFromFile` to wrap a local file as an `AsyncBuffer`: + +```typescript +import { asyncBufferFromFile, parquetReadObjects } from 'hyparquet' + +const file: AsyncBuffer = await asyncBufferFromFile('example.parquet') +const data = await parquetReadObjects({ file }) +``` + +#### ArrayBuffer + +You can provide an `ArrayBuffer` anywhere that an `AsyncBuffer` is expected. This is useful if you already have the entire parquet file in memory. + +### parquetRead vs parquetReadObjects + +#### parquetReadObjects + +`parquetReadObjects` is a convenience wrapper around `parquetRead` that returns the complete rows as `Promise[]>`. This is the simplest way to read parquet files. + +```typescript +parquetReadObjects({ file }): Promise[]> +``` + +#### parquetRead + +`parquetRead` is the "base" function for reading parquet files. +It returns a `Promise` that resolves when the file has been read or rejected if an error occurs. +Data is returned via `onComplete` or `onChunk` or `onPage` callbacks passed as arguments. + +The reason for this design is that parquet is a column-oriented format, and returning data in row-oriented format requires transposing the column data. This is an expensive operation in javascript. If you don't pass in an `onComplete` argument to `parquetRead`, hyparquet will skip this transpose step and save memory. + +### Chunk Streaming + +The `onChunk` callback returns column-oriented data as it is ready. `onChunk` will always return top-level columns, including structs, assembled as a single column. This may require waiting for multiple sub-columns to all load before assembly can occur. + +The `onPage` callback returns column-oriented page data as it is ready. `onPage` will NOT assemble struct columns and will always return individual sub-column data. Note that `onPage` _will_ assemble nested lists. + +In some cases, `onPage` can return data sooner than `onChunk`. + +```typescript +interface ColumnData { + columnName: string + columnData: ArrayLike + rowStart: number + rowEnd: number +} +await parquetRead({ + file, + onChunk(chunk: ColumnData) { + console.log('chunk', chunk) + }, + onPage(chunk: ColumnData) { + console.log('page', chunk) + }, +}) +``` + +### Returned row format + +By default, the `onComplete` function returns an **array** of values for each row: `[value]`. If you would prefer each row to be an **object**: `{ columnName: value }`, set the option `rowFormat` to `'object'`. + +```javascript +import { parquetRead } from 'hyparquet' + +await parquetRead({ + file, + rowFormat: 'object', + onComplete: data => console.log(data), +}) +``` + +The `parquetReadObjects` function defaults to `rowFormat: 'object'`. + +### Binary columns + +Hyparquet defaults to decoding binary columns as utf8 text strings. A parquet `BYTE_ARRAY` column may contain arbitrary binary data or utf8 encoded text data. In theory, a column should be annotated as [LogicalType](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) STRING if it contains utf8 text. But in practice, many parquet files omit this annotation. Hyparquet's default decoding behavior can be disabled by setting the `utf8` option to `false`. The `utf8` option only affects `BYTE_ARRAY` columns _without_ an annotation. + +If Hyparquet detects a [GeoParquet](https://geoparquet.org/) file, any geospatial column will be marked with the GEOMETRY or GEOGRAPHY logical type and decoded to GeoJSON geometries. Set the `geoparquet` option to `false` to disable this behavior. + +## Compression + +By default, hyparquet supports uncompressed and snappy-compressed parquet files. +To support the full range of parquet compression codecs (gzip, brotli, zstd, etc), use the [hyparquet-compressors](https://github.com/hyparam/hyparquet-compressors) package. + +```javascript +import { parquetReadObjects } from 'hyparquet' +import { compressors } from 'hyparquet-compressors' + +const data = await parquetReadObjects({ file, compressors }) +``` + +| Codec | hyparquet | with hyparquet-compressors | +|---------------|-----------|----------------------------| +| Uncompressed | ✅ | ✅ | +| Snappy | ✅ | ✅ | +| GZip | ❌ | ✅ | +| LZO | ❌ | ✅ | +| Brotli | ❌ | ✅ | +| LZ4 | ❌ | ✅ | +| ZSTD | ❌ | ✅ | +| LZ4_RAW | ❌ | ✅ | + +## References + + - https://github.com/apache/parquet-format + - https://github.com/apache/parquet-testing + - https://github.com/apache/thrift + - https://github.com/apache/arrow + - https://github.com/dask/fastparquet + - https://github.com/duckdb/duckdb + - https://github.com/google/snappy + - https://github.com/hyparam/hightable + - https://github.com/hyparam/hysnappy + - https://github.com/hyparam/hyparquet-compressors + - https://github.com/ironSource/parquetjs + - https://github.com/zhipeng-jia/snappyjs + +Sample project that shows how to build a parquet viewer using hyparquet, react, and [HighTable](https://github.com/hyparam/hightable): + + - Hyparquet Demo: [https://hyparam.github.io/demos/hyparquet/](https://hyparam.github.io/demos/hyparquet/) + - Hyparquet Demo Source Code: [https://github.com/hyparam/demos/tree/master/hyparquet](https://github.com/hyparam/demos/tree/master/hyparquet) + + + +## Contributions + +Contributions are welcome! +If you have suggestions, bug reports, or feature requests, please open an issue or submit a pull request. + +Hyparquet development is supported by an open-source grant from Hugging Face :hugs: diff --git a/web/node_modules/hyparquet/package.json b/web/node_modules/hyparquet/package.json new file mode 100644 index 0000000000000000000000000000000000000000..644f8ef31bb6056d214c2c3adebe1211fc70a8b1 --- /dev/null +++ b/web/node_modules/hyparquet/package.json @@ -0,0 +1,68 @@ +{ + "name": "hyparquet", + "version": "1.26.2", + "description": "Parquet file parser for JavaScript", + "author": "Hyperparam", + "homepage": "https://hyperparam.app", + "keywords": [ + "ai", + "data", + "dataset", + "hyperparam", + "hyparquet", + "geoparquet", + "llm", + "ml", + "parquet", + "parquetjs", + "parser", + "snappy", + "thrift" + ], + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/hyparam/hyparquet.git" + }, + "files": [ + "src", + "types" + ], + "type": "module", + "types": "types/index.d.ts", + "main": "src/index.js", + "exports": { + ".": { + "browser": { + "types": "./types/index.d.ts", + "import": "./src/index.js" + }, + "default": { + "types": "./types/node.d.ts", + "import": "./src/node.js" + } + }, + "./src/*.js": { + "types": "./types/*.d.ts", + "import": "./src/*.js" + } + }, + "sideEffects": false, + "scripts": { + "build:types": "tsc -p ./tsconfig.build.json", + "coverage": "vitest run --coverage --coverage.include=src", + "lint": "eslint", + "lint:fix": "eslint --fix", + "prepare": "npm run build:types", + "test": "vitest run" + }, + "devDependencies": { + "@types/node": "26.0.1", + "@vitest/coverage-v8": "4.1.9", + "eslint": "9.39.4", + "eslint-plugin-jsdoc": "63.0.10", + "hyparquet-compressors": "1.1.1", + "typescript": "6.0.3", + "vitest": "4.1.9" + } +} diff --git a/web/node_modules/hyparquet/src/assemble.js b/web/node_modules/hyparquet/src/assemble.js new file mode 100644 index 0000000000000000000000000000000000000000..7ca6014024c18e52a22233a8e2efb8052c3a74a7 --- /dev/null +++ b/web/node_modules/hyparquet/src/assemble.js @@ -0,0 +1,257 @@ +import { getMaxDefinitionLevel, isListLike, isMapLike } from './schema.js' +import { decodeVariantColumn } from './variant.js' + +/** + * Reconstructs a complex nested structure from flat arrays of values and + * definition and repetition levels, according to Dremel encoding. + * + * @param {any[]} output + * @param {number[] | undefined} definitionLevels + * @param {number[]} repetitionLevels + * @param {DecodedArray} values + * @param {SchemaTree[]} schemaPath + * @returns {DecodedArray} + */ +export function assembleLists(output, definitionLevels, repetitionLevels, values, schemaPath) { + const maxDefinitionLevel = getMaxDefinitionLevel(schemaPath) + // If no def/rep levels, synthesize def levels at max + if (!definitionLevels?.length && !repetitionLevels.length) { + if (!maxDefinitionLevel || !values.length) return values + definitionLevels = new Array(values.length).fill(maxDefinitionLevel) + } + const n = definitionLevels?.length || repetitionLevels.length + const repetitionPath = schemaPath.map(({ element }) => element.repetition_type) + let valueIndex = 0 + + // Track state of nested structures + const containerStack = [output] + let currentContainer = output + let currentDepth = 0 // schema depth + let currentDefLevel = 0 // list depth + let currentRepLevel = 0 + + if (repetitionLevels[0]) { + // continue previous row + while (currentDepth < repetitionPath.length - 2 && currentRepLevel < repetitionLevels[0]) { + currentDepth++ + if (repetitionPath[currentDepth] !== 'REQUIRED') { + // go into last list + currentContainer = currentContainer.at(-1) + containerStack.push(currentContainer) + currentDefLevel++ + } + if (repetitionPath[currentDepth] === 'REPEATED') currentRepLevel++ + } + } + + for (let i = 0; i < n; i++) { + // assert(currentDefLevel === containerStack.length - 1) + const def = definitionLevels?.length ? definitionLevels[i] : maxDefinitionLevel + const rep = repetitionLevels[i] + + // Pop up to start of rep level + while (currentDepth && (rep < currentRepLevel || repetitionPath[currentDepth] !== 'REPEATED')) { + if (repetitionPath[currentDepth] !== 'REQUIRED') { + containerStack.pop() + currentDefLevel-- + } + if (repetitionPath[currentDepth] === 'REPEATED') currentRepLevel-- + currentDepth-- + } + // @ts-expect-error won't be empty + currentContainer = containerStack.at(-1) + + // Go deeper to end of definition level + while ( + (currentDepth < repetitionPath.length - 2 || repetitionPath[currentDepth + 1] === 'REPEATED') && + (currentDefLevel < def || repetitionPath[currentDepth + 1] === 'REQUIRED') + ) { + currentDepth++ + if (repetitionPath[currentDepth] !== 'REQUIRED') { + /** @type {any[]} */ + const newList = [] + currentContainer.push(newList) + currentContainer = newList + containerStack.push(newList) + currentDefLevel++ + } + if (repetitionPath[currentDepth] === 'REPEATED') currentRepLevel++ + } + + // Add value or null based on definition level + if (def === maxDefinitionLevel) { + // assert(currentDepth === maxDefinitionLevel || currentDepth === repetitionPath.length - 2) + currentContainer.push(values[valueIndex++]) + } else if (currentDepth === repetitionPath.length - 2) { + currentContainer.push(null) + } else { + currentContainer.push([]) + } + } + + // Handle edge cases for empty inputs or single-level data + if (!output.length) { + // return max definition level of nested lists + for (let i = 0; i < maxDefinitionLevel; i++) { + /** @type {any[]} */ + const newList = [] + currentContainer.push(newList) + currentContainer = newList + } + } + + return output +} + +/** + * Assemble a nested structure from subcolumn data. + * + * @param {Map} subcolumnData + * @param {SchemaTree} schema top-level schema element + * @param {ParquetParsers} parsers + * @param {number} [depth] depth of nested structure + */ +export function assembleNested(subcolumnData, schema, parsers, depth = 0) { + const path = schema.path.join('.') + const optional = schema.element.repetition_type === 'OPTIONAL' + const nextDepth = optional ? depth + 1 : depth + + if (isListLike(schema)) { + let sublist = schema.children[0] + let subDepth = nextDepth + if (sublist.children.length === 1) { + sublist = sublist.children[0] + subDepth++ + } + assembleNested(subcolumnData, sublist, parsers, subDepth) + + const subcolumn = sublist.path.join('.') + const values = subcolumnData.get(subcolumn) + if (!values) throw new Error('parquet list column missing values') + if (optional) flattenAtDepth(values, depth) + subcolumnData.set(path, values) + subcolumnData.delete(subcolumn) + return + } + + if (isMapLike(schema)) { + const mapName = schema.children[0].element.name + + // Assemble keys and values + assembleNested(subcolumnData, schema.children[0].children[0], parsers, nextDepth + 1) + assembleNested(subcolumnData, schema.children[0].children[1], parsers, nextDepth + 1) + + const keys = subcolumnData.get(`${path}.${mapName}.key`) + const values = subcolumnData.get(`${path}.${mapName}.value`) + + if (!keys) throw new Error('parquet map column missing keys') + if (!values) throw new Error('parquet map column missing values') + if (keys.length !== values.length) { + throw new Error('parquet map column key/value length mismatch') + } + + const out = assembleMaps(keys, values, nextDepth) + if (optional) flattenAtDepth(out, depth) + + subcolumnData.delete(`${path}.${mapName}.key`) + subcolumnData.delete(`${path}.${mapName}.value`) + subcolumnData.set(path, out) + return + } + + // Struct-like column + if (schema.children.length) { + // construct a meta struct and then invert + const invertDepth = schema.element.repetition_type === 'REQUIRED' ? depth : depth + 1 + /** @type {Record} */ + const struct = {} + for (const child of schema.children) { + assembleNested(subcolumnData, child, parsers, invertDepth) + const childData = subcolumnData.get(child.path.join('.')) + if (!childData) throw new Error('parquet struct missing child data') + struct[child.element.name] = childData + } + // remove children + for (const child of schema.children) { + subcolumnData.delete(child.path.join('.')) + } + + // invert struct by depth + let inverted = invertStruct(struct, invertDepth) + if (schema.element.logical_type?.type === 'VARIANT') { + inverted = decodeVariantColumn(inverted, parsers) + } + if (optional) flattenAtDepth(inverted, depth) + subcolumnData.set(path, inverted) + } +} + +/** + * @import {DecodedArray, ParquetParsers, SchemaTree} from '../src/types.js' + * @param {DecodedArray} arr + * @param {number} depth + */ +function flattenAtDepth(arr, depth) { + for (let i = 0; i < arr.length; i++) { + if (depth) { + flattenAtDepth(arr[i], depth - 1) + } else { + arr[i] = arr[i][0] + } + } +} + +/** + * @param {DecodedArray} keys + * @param {DecodedArray} values + * @param {number} depth + * @returns {any[]} + */ +function assembleMaps(keys, values, depth) { + const out = [] + for (let i = 0; i < keys.length; i++) { + if (depth) { + out.push(assembleMaps(keys[i], values[i], depth - 1)) // go deeper + } else { + if (keys[i]) { + /** @type {Record} */ + const obj = {} + for (let j = 0; j < keys[i].length; j++) { + const value = values[i][j] + obj[keys[i][j]] = value === undefined ? null : value + } + out.push(obj) + } else { + out.push(undefined) + } + } + } + return out +} + +/** + * Invert a struct-like object by depth. + * + * @param {Record} struct + * @param {number} depth + * @returns {any[]} + */ +function invertStruct(struct, depth) { + const keys = Object.keys(struct) + const length = struct[keys[0]]?.length + const out = [] + for (let i = 0; i < length; i++) { + /** @type {Record} */ + const obj = {} + for (const key of keys) { + if (struct[key].length !== length) throw new Error('parquet struct parsing error') + obj[key] = struct[key][i] + } + if (depth) { + out.push(invertStruct(obj, depth - 1)) // deeper + } else { + out.push(obj) + } + } + return out +} diff --git a/web/node_modules/hyparquet/src/bloom.js b/web/node_modules/hyparquet/src/bloom.js new file mode 100644 index 0000000000000000000000000000000000000000..4d56a41638699a72ab3422e8e96a85c2480b3552 --- /dev/null +++ b/web/node_modules/hyparquet/src/bloom.js @@ -0,0 +1,224 @@ +// Split Block Bloom Filter (https://github.com/apache/parquet-format/blob/master/BloomFilter.md) +// A bloom filter is a sequence of 32-byte blocks. Each block holds 8 little-endian uint32 words. +// Insertion sets one bit per word, chosen by salting the low 32 bits of an xxhash64. +// Membership requires all 8 bits to be set; misses are exact, hits are probabilistic. + +import { deserializeTCompactProtocol } from './thrift.js' +import { xxhash64 } from './xxhash.js' + +/** + * @import {BloomFilter, DataReader, ParquetQueryFilter, SchemaElement} from '../src/types.js' + */ + +const textEncoder = new TextEncoder() + +const SALT = new Uint32Array([ + 0x47b6137b, 0x44974d91, 0x8824ad5b, 0xa2b7289d, + 0x705495c7, 0x2df1424b, 0x9efc4947, 0x5c6bfb31, +]) + +/** + * Map the high 32 bits of a hash to a block index in [0, numBlocks). + * + * @param {bigint} hash + * @param {number} numBlocks + * @returns {number} + */ +function blockIndex(hash, numBlocks) { + return Number((hash >> 32n) * BigInt(numBlocks) >> 32n) +} + +/** + * Per-block mask: 8 uint32 words, each with a single bit set at position `(low32 * SALT[i]) >> 27`. + * + * @param {bigint} hash + * @returns {Uint32Array} + */ +function blockMask(hash) { + const m = new Uint32Array(8) + const low = Number(hash & 0xffffffffn) | 0 + for (let i = 0; i < 8; i++) { + m[i] = 1 << (Math.imul(low, SALT[i]) >>> 27) + } + return m +} + +/** + * Insert a hash into a Split Block Bloom Filter. + * + * @param {Uint32Array} blocks bloom filter words (8 * numBlocks long) + * @param {bigint} hash 64-bit xxhash of the parquet-plain-encoded value + */ +export function sbbfInsert(blocks, hash) { + const offset = blockIndex(hash, blocks.length >> 3) << 3 + const m = blockMask(hash) + for (let i = 0; i < 8; i++) { + blocks[offset + i] |= m[i] + } +} + +/** + * Test whether a hash might be present in a Split Block Bloom Filter. + * False positives are possible; false negatives are not. + * + * @param {Uint32Array} blocks bloom filter words (8 * numBlocks long) + * @param {bigint} hash 64-bit xxhash of the parquet-plain-encoded value + * @returns {boolean} + */ +export function sbbfContains(blocks, hash) { + const offset = blockIndex(hash, blocks.length >> 3) << 3 + const m = blockMask(hash) + for (let i = 0; i < 8; i++) { + if ((blocks[offset + i] & m[i]) === 0) return false + } + return true +} + +/** + * Parse a Split Block Bloom Filter from a reader positioned at the BloomFilterHeader. + * Returns undefined when the header advertises an unsupported algorithm, hash, or + * compression — callers should treat that as "cannot use this bloom filter." + * + * @param {DataReader} reader + * @returns {BloomFilter | undefined} + */ +export function readBloomFilter(reader) { + const header = deserializeTCompactProtocol(reader) + const numBytes = header.field_1 + if (typeof numBytes !== 'number' || numBytes <= 0 || numBytes % 32 !== 0) return undefined + // BloomFilterAlgorithm / Hash / Compression are unions with a single supported variant each. + if (!header.field_2?.field_1) return undefined // algorithm must be BLOCK + if (!header.field_3?.field_1) return undefined // hash must be XXHASH + if (!header.field_4?.field_1) return undefined // compression must be UNCOMPRESSED + + const { view, offset } = reader + if (offset + numBytes > view.byteLength) { + throw new Error(`parquet bloom filter truncated: need ${numBytes} bytes, have ${view.byteLength - offset}`) + } + // Reader offset is not 4-aligned in general, and we want endian-portable reads. + const blocks = new Uint32Array(numBytes >> 2) + for (let i = 0; i < blocks.length; i++) { + blocks[i] = view.getUint32(offset + i * 4, true) + } + reader.offset = offset + numBytes + return { numBytes, blocks } +} + +/** + * Hash a JS filter value as its parquet PLAIN-encoded bytes, suitable for a + * bloom filter lookup. Returns undefined when the column's parser is lossy or + * ambiguous (DATE, TIMESTAMP_*, DECIMAL, JSON, BSON, INT96, FLOAT16, UUID, + * GEOMETRY, GEOGRAPHY, INTERVAL) or when the JS value type doesn't match the + * column. Callers must treat undefined as "bloom filter cannot help." + * + * @param {any} value + * @param {SchemaElement} element + * @returns {bigint | undefined} + */ +export function hashParquetValue(value, element) { + if (value === null || value === undefined) return undefined + const { type, converted_type, logical_type } = element + + if (type === 'BOOLEAN') { + if (typeof value !== 'boolean') return undefined + return xxhash64(new Uint8Array([value ? 1 : 0])) + } + + if (type === 'FLOAT') { + if (typeof value !== 'number') return undefined + const buf = new ArrayBuffer(4) + new DataView(buf).setFloat32(0, value, true) + return xxhash64(new Uint8Array(buf)) + } + + if (type === 'DOUBLE') { + if (typeof value !== 'number') return undefined + const buf = new ArrayBuffer(8) + new DataView(buf).setFloat64(0, value, true) + return xxhash64(new Uint8Array(buf)) + } + + if (type === 'INT32') { + if (converted_type === 'DATE' || converted_type === 'DECIMAL' || converted_type === 'TIME_MILLIS') return undefined + if (logical_type?.type === 'DATE' || logical_type?.type === 'TIME' || logical_type?.type === 'DECIMAL') return undefined + if (typeof value !== 'number' || !Number.isInteger(value)) return undefined + const buf = new ArrayBuffer(4) + new DataView(buf).setInt32(0, value | 0, true) + return xxhash64(new Uint8Array(buf)) + } + + if (type === 'INT64') { + if (converted_type === 'TIMESTAMP_MILLIS' || converted_type === 'TIMESTAMP_MICROS') return undefined + if (converted_type === 'TIME_MICROS' || converted_type === 'DECIMAL') return undefined + if (logical_type?.type === 'TIMESTAMP' || logical_type?.type === 'TIME' || logical_type?.type === 'DECIMAL') return undefined + let bigValue + if (typeof value === 'bigint') bigValue = value + else if (typeof value === 'number' && Number.isSafeInteger(value)) bigValue = BigInt(value) + else return undefined + const buf = new ArrayBuffer(8) + new DataView(buf).setBigUint64(0, BigInt.asUintN(64, bigValue), true) + return xxhash64(new Uint8Array(buf)) + } + + if (type === 'BYTE_ARRAY') { + if (converted_type === 'JSON' || converted_type === 'BSON' || converted_type === 'DECIMAL') return undefined + if (logical_type?.type === 'JSON' || logical_type?.type === 'BSON' || logical_type?.type === 'VARIANT') return undefined + if (logical_type?.type === 'GEOMETRY' || logical_type?.type === 'GEOGRAPHY') return undefined + if (typeof value === 'string') return xxhash64(textEncoder.encode(value)) + if (value instanceof Uint8Array) return xxhash64(value) + return undefined + } + + if (type === 'FIXED_LEN_BYTE_ARRAY') { + if (converted_type === 'DECIMAL' || converted_type === 'INTERVAL') return undefined + if (logical_type?.type === 'DECIMAL' || logical_type?.type === 'UUID' || logical_type?.type === 'FLOAT16') return undefined + if (logical_type?.type === 'GEOMETRY' || logical_type?.type === 'GEOGRAPHY') return undefined + if (value instanceof Uint8Array) return xxhash64(value) + return undefined + } + + // INT96 deprecated, or type missing on group columns + return undefined +} + +/** + * Top-level column names that appear in $eq or $in predicates within a filter. + * These are the only columns where a bloom filter can prove a value's absence + * and let us skip a row group; any other operator can't be helped by a bloom. + * + * @param {ParquetQueryFilter | undefined} filter + * @returns {Set} + */ +export function bloomEligibleColumns(filter) { + /** @type {Set} */ + const out = new Set() + walkBloomEligible(filter, out) + return out +} + +/** + * @param {ParquetQueryFilter | undefined} filter + * @param {Set} out + */ +function walkBloomEligible(filter, out) { + if (!filter) return + if ('$and' in filter && Array.isArray(filter.$and)) { + for (const sub of filter.$and) walkBloomEligible(sub, out) + return + } + if ('$or' in filter && Array.isArray(filter.$or)) { + for (const sub of filter.$or) walkBloomEligible(sub, out) + return + } + // $nor would need to prove presence, not absence — bloom can't help. + if ('$nor' in filter) return + for (const [field, condition] of Object.entries(filter)) { + if (field.startsWith('$')) continue + if (typeof condition === 'object' && condition !== null && !Array.isArray(condition)) { + if ('$eq' in condition || '$in' in condition) out.add(field) + } else { + // primitive / null / array condition is an implicit $eq + out.add(field) + } + } +} diff --git a/web/node_modules/hyparquet/src/column.js b/web/node_modules/hyparquet/src/column.js new file mode 100644 index 0000000000000000000000000000000000000000..4af7fa575ab4e8327fba6cad96810ee7c62ec266 --- /dev/null +++ b/web/node_modules/hyparquet/src/column.js @@ -0,0 +1,202 @@ +/** + * @import {ColumnDecoder, DataReader, DecodedArray, PageHeader, PageResult, RowGroupSelect, SubColumnData} from '../src/types.js' + */ + +import { assembleLists } from './assemble.js' +import { Encodings, PageTypes } from './constants.js' +import { convert, convertWithDictionary } from './convert.js' +import { decompressPage, readDataPage, readDataPageV2 } from './datapage.js' +import { readPlain } from './plain.js' +import { isFlatColumn } from './schema.js' +import { deserializeTCompactProtocol } from './thrift.js' + +/** + * Parse column data from a buffer. + * + * @param {DataReader} reader + * @param {RowGroupSelect} rowGroupSelect row group selection + * @param {ColumnDecoder} columnDecoder column decoder params + * @param {(chunk: SubColumnData) => void} [onPage] callback for each page + * @returns {{ data: DecodedArray[], skipped: number }} + */ +export function readColumn(reader, { groupStart, selectStart, selectEnd }, columnDecoder, onPage) { + const { pathInSchema, schemaPath } = columnDecoder + const isFlat = isFlatColumn(schemaPath) + /** @type {DecodedArray[]} */ + const chunks = [] + /** @type {DecodedArray | undefined} */ + let dictionary = undefined + /** @type {DecodedArray | undefined} */ + let lastChunk = undefined + let rowCount = 0 + let skipped = 0 + + const emitLastChunk = onPage && (() => { + lastChunk && onPage({ + pathInSchema, + columnData: lastChunk, + rowStart: groupStart + rowCount - lastChunk.length, + rowEnd: groupStart + rowCount, + }) + }) + + while (isFlat ? rowCount < selectEnd : reader.offset < reader.view.byteLength - 1) { + if (reader.offset >= reader.view.byteLength - 1) break // end of reader + + // read page header + const header = parquetHeader(reader) + if (header.type === 'DICTIONARY_PAGE') { + const { data } = readPage(reader, header, columnDecoder, dictionary, undefined, 0) + if (data) dictionary = convert(data, columnDecoder) + } else { + const lastChunkLength = lastChunk?.length || 0 + const result = readPage(reader, header, columnDecoder, dictionary, lastChunk, selectStart - rowCount) + if (result.skipped) { + // skipped page - just advance row count, don't add to chunks + if (!chunks.length) { + skipped += result.skipped + } + rowCount += result.skipped + } else if (result.data && lastChunk === result.data) { + // continued from previous page + rowCount += result.data.length - lastChunkLength + } else if (result.data && result.data.length) { + emitLastChunk?.() + chunks.push(result.data) + rowCount += result.data.length + lastChunk = result.data + } + } + } + emitLastChunk?.() + + return { data: chunks, skipped } +} + +/** + * Read a page (data or dictionary) from a buffer. + * + * @param {DataReader} reader + * @param {PageHeader} header + * @param {ColumnDecoder} columnDecoder + * @param {DecodedArray | undefined} dictionary + * @param {DecodedArray | undefined} previousChunk + * @param {number} pageStart skip this many rows in the page + * @returns {PageResult} + */ +export function readPage(reader, header, columnDecoder, dictionary, previousChunk, pageStart) { + const { type, element, schemaPath, codec, compressors } = columnDecoder + // read compressed_page_size bytes + const compressedBytes = new Uint8Array( + reader.view.buffer, reader.view.byteOffset + reader.offset, header.compressed_page_size + ) + reader.offset += header.compressed_page_size + + // parse page data by type + if (header.type === 'DATA_PAGE') { + const daph = header.data_page_header + if (!daph) throw new Error('parquet data page header is undefined') + + // skip unnecessary non-nested pages + if (pageStart > daph.num_values && isFlatColumn(schemaPath)) { + return { skipped: daph.num_values } + } + + const page = decompressPage(compressedBytes, Number(header.uncompressed_page_size), codec, compressors) + const { definitionLevels, repetitionLevels, dataPage } = readDataPage(page, daph, columnDecoder) + // assert(!daph.statistics?.null_count || daph.statistics.null_count === BigInt(daph.num_values - dataPage.length)) + + // convert types, dereference dictionary, and assemble lists + const values = convertWithDictionary(dataPage, dictionary, daph.encoding, columnDecoder) + const output = Array.isArray(previousChunk) ? previousChunk : [] + const assembled = assembleLists(output, definitionLevels, repetitionLevels, values, schemaPath) + return { skipped: 0, data: assembled } + } else if (header.type === 'DATA_PAGE_V2') { + const daph2 = header.data_page_header_v2 + if (!daph2) throw new Error('parquet data page header v2 is undefined') + + // skip unnecessary pages + if (pageStart > daph2.num_rows) { + return { skipped: daph2.num_values } + } + + const { definitionLevels, repetitionLevels, dataPage } = + readDataPageV2(compressedBytes, header, columnDecoder) + + // convert types, dereference dictionary, and assemble lists + const values = convertWithDictionary(dataPage, dictionary, daph2.encoding, columnDecoder) + const output = Array.isArray(previousChunk) ? previousChunk : [] + const assembled = assembleLists(output, definitionLevels, repetitionLevels, values, schemaPath) + return { skipped: 0, data: assembled } + } else if (header.type === 'DICTIONARY_PAGE') { + const diph = header.dictionary_page_header + if (!diph) throw new Error('parquet dictionary page header is undefined') + + const page = decompressPage( + compressedBytes, Number(header.uncompressed_page_size), codec, compressors + ) + + const reader = { view: new DataView(page.buffer, page.byteOffset, page.byteLength), offset: 0 } + const dictArray = readPlain(reader, type, diph.num_values, element.type_length) + return { skipped: 0, data: dictArray } + } else { + throw new Error(`parquet unsupported page type: ${header.type}`) + } +} + +/** + * Read parquet header from a buffer. + * + * @param {DataReader} reader + * @returns {PageHeader} + */ +function parquetHeader(reader) { + const header = deserializeTCompactProtocol(reader) + + // Parse parquet header from thrift data + const type = PageTypes[header.field_1] + const uncompressed_page_size = header.field_2 + const compressed_page_size = header.field_3 + const crc = header.field_4 + const data_page_header = header.field_5 && { + num_values: header.field_5.field_1, + encoding: Encodings[header.field_5.field_2], + definition_level_encoding: Encodings[header.field_5.field_3], + repetition_level_encoding: Encodings[header.field_5.field_4], + statistics: header.field_5.field_5 && { + max: header.field_5.field_5.field_1, + min: header.field_5.field_5.field_2, + null_count: header.field_5.field_5.field_3, + distinct_count: header.field_5.field_5.field_4, + max_value: header.field_5.field_5.field_5, + min_value: header.field_5.field_5.field_6, + }, + } + const index_page_header = header.field_6 + const dictionary_page_header = header.field_7 && { + num_values: header.field_7.field_1, + encoding: Encodings[header.field_7.field_2], + is_sorted: header.field_7.field_3, + } + const data_page_header_v2 = header.field_8 && { + num_values: header.field_8.field_1, + num_nulls: header.field_8.field_2, + num_rows: header.field_8.field_3, + encoding: Encodings[header.field_8.field_4], + definition_levels_byte_length: header.field_8.field_5, + repetition_levels_byte_length: header.field_8.field_6, + is_compressed: header.field_8.field_7 === undefined ? true : header.field_8.field_7, // default true + statistics: header.field_8.field_8, + } + + return { + type, + uncompressed_page_size, + compressed_page_size, + crc, + data_page_header, + index_page_header, + dictionary_page_header, + data_page_header_v2, + } +} diff --git a/web/node_modules/hyparquet/src/constants.js b/web/node_modules/hyparquet/src/constants.js new file mode 100644 index 0000000000000000000000000000000000000000..4c1f9fc86827acbf9307a7fe18a7f5055d84b4bc --- /dev/null +++ b/web/node_modules/hyparquet/src/constants.js @@ -0,0 +1,95 @@ + +/** @type {import('../src/types.d.ts').ParquetType[]} */ +export const ParquetTypes = [ + 'BOOLEAN', + 'INT32', + 'INT64', + 'INT96', // deprecated + 'FLOAT', + 'DOUBLE', + 'BYTE_ARRAY', + 'FIXED_LEN_BYTE_ARRAY', +] + +/** @type {import('../src/types.d.ts').Encoding[]} */ +export const Encodings = [ + 'PLAIN', + 'GROUP_VAR_INT', // deprecated + 'PLAIN_DICTIONARY', + 'RLE', + 'BIT_PACKED', // deprecated + 'DELTA_BINARY_PACKED', + 'DELTA_LENGTH_BYTE_ARRAY', + 'DELTA_BYTE_ARRAY', + 'RLE_DICTIONARY', + 'BYTE_STREAM_SPLIT', +] + +/** @type {import('../src/types.d.ts').FieldRepetitionType[]} */ +export const FieldRepetitionTypes = [ + 'REQUIRED', + 'OPTIONAL', + 'REPEATED', +] + +/** @type {import('../src/types.d.ts').ConvertedType[]} */ +export const ConvertedTypes = [ + 'UTF8', + 'MAP', + 'MAP_KEY_VALUE', + 'LIST', + 'ENUM', + 'DECIMAL', + 'DATE', + 'TIME_MILLIS', + 'TIME_MICROS', + 'TIMESTAMP_MILLIS', + 'TIMESTAMP_MICROS', + 'UINT_8', + 'UINT_16', + 'UINT_32', + 'UINT_64', + 'INT_8', + 'INT_16', + 'INT_32', + 'INT_64', + 'JSON', + 'BSON', + 'INTERVAL', +] + +/** @type {import('../src/types.d.ts').CompressionCodec[]} */ +export const CompressionCodecs = [ + 'UNCOMPRESSED', + 'SNAPPY', + 'GZIP', + 'LZO', + 'BROTLI', + 'LZ4', + 'ZSTD', + 'LZ4_RAW', +] + +/** @type {import('../src/types.d.ts').PageType[]} */ +export const PageTypes = [ + 'DATA_PAGE', + 'INDEX_PAGE', + 'DICTIONARY_PAGE', + 'DATA_PAGE_V2', +] + +/** @type {import('../src/types.d.ts').BoundaryOrder[]} */ +export const BoundaryOrders = [ + 'UNORDERED', + 'ASCENDING', + 'DESCENDING', +] + +/** @type {import('../src/types.d.ts').EdgeInterpolationAlgorithm[]} */ +export const EdgeInterpolationAlgorithms = [ + 'SPHERICAL', + 'VINCENTY', + 'THOMAS', + 'ANDOYER', + 'KARNEY', +] diff --git a/web/node_modules/hyparquet/src/convert.js b/web/node_modules/hyparquet/src/convert.js new file mode 100644 index 0000000000000000000000000000000000000000..f3a524d2d4a5504e38b270f635f87feeafbffef4 --- /dev/null +++ b/web/node_modules/hyparquet/src/convert.js @@ -0,0 +1,210 @@ +import { wkbToGeojson } from './wkb.js' + +/** + * @import {ColumnDecoder, DecodedArray, Encoding, ParquetParsers} from '../src/types.js' + */ + +const decoder = new TextDecoder() + +/** + * Default type parsers when no custom ones are given + * @type ParquetParsers + */ +export const DEFAULT_PARSERS = { + timestampFromMilliseconds(millis) { + return new Date(Number(millis)) + }, + timestampFromMicroseconds(micros) { + return new Date(Number(micros / 1000n)) + }, + timestampFromNanoseconds(nanos) { + return new Date(Number(nanos / 1000000n)) + }, + dateFromDays(days) { + return new Date(days * 86400000) + }, + stringFromBytes(bytes) { + return bytes && decoder.decode(bytes) + }, + jsonFromBytes(bytes) { + return bytes && JSON.parse(decoder.decode(bytes)) + }, + geometryFromBytes(bytes) { + return bytes && wkbToGeojson({ view: new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), offset: 0 }) + }, + geographyFromBytes(bytes) { + return bytes && wkbToGeojson({ view: new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), offset: 0 }) + }, + uuidFromBytes(bytes) { + if (!bytes) return undefined + const hex = Array.from(bytes, b => b.toString(16).padStart(2, '0')).join('') + return hex.slice(0, 8) + '-' + hex.slice(8, 12) + '-' + hex.slice(12, 16) + '-' + hex.slice(16, 20) + '-' + hex.slice(20, 32) + }, +} + +/** + * Convert known types from primitive to rich, and dereference dictionary. + * + * @param {DecodedArray} data series of primitive types + * @param {DecodedArray | undefined} dictionary + * @param {Encoding} encoding + * @param {ColumnDecoder} columnDecoder + * @returns {DecodedArray} series of rich types + */ +export function convertWithDictionary(data, dictionary, encoding, columnDecoder) { + if (dictionary && encoding.endsWith('_DICTIONARY')) { + let output = data + if (data instanceof Uint8Array && !(dictionary instanceof Uint8Array)) { + // @ts-expect-error upgrade data to match dictionary type with fancy constructor + output = new dictionary.constructor(data.length) + } + for (let i = 0; i < data.length; i++) { + output[i] = dictionary[data[i]] + } + return output + } else { + return convert(data, columnDecoder) + } +} + +/** + * Convert known types from primitive to rich. + * + * @param {DecodedArray} data series of primitive types + * @param {ColumnDecoder} columnDecoder + * @returns {DecodedArray} series of rich types + */ +export function convert(data, columnDecoder) { + const { element, parsers, utf8 = true, schemaPath } = columnDecoder + const { type, converted_type: ctype, logical_type: ltype } = element + const nullable = element.repetition_type !== 'REQUIRED' + + // Skip utf8 conversion for plain BYTE_ARRAY inside VARIANT + const isVariant = schemaPath?.some(s => s.element.logical_type?.type === 'VARIANT') + if (isVariant && type === 'BYTE_ARRAY' && ctype !== 'UTF8' && ltype?.type !== 'STRING') { + return data + } + if (ctype === 'DECIMAL') { + const scale = element.scale || 0 + const factor = 10 ** -scale + const arr = new Array(data.length) + for (let i = 0; i < arr.length; i++) { + if (data[i] instanceof Uint8Array) { + arr[i] = parseDecimal(data[i]) * factor + } else { + arr[i] = Number(data[i]) * factor + } + } + return arr + } + if (!ctype && type === 'INT96') { + return Array.from(data).map(v => parsers.timestampFromNanoseconds(parseInt96Nanos(v))) + } + if (ctype === 'DATE') { + return Array.from(data).map(v => parsers.dateFromDays(v)) + } + if (ctype === 'TIMESTAMP_MILLIS') { + return Array.from(data).map(v => parsers.timestampFromMilliseconds(v)) + } + if (ctype === 'TIMESTAMP_MICROS') { + return Array.from(data).map(v => parsers.timestampFromMicroseconds(v)) + } + if (ctype === 'JSON') { + return data.map(v => parsers.jsonFromBytes(v)) + } + if (ctype === 'BSON') { + throw new Error('parquet bson not supported') + } + if (ctype === 'INTERVAL') { + throw new Error('parquet interval not supported') + } + if (ltype?.type === 'GEOMETRY') { + return data.map(v => parsers.geometryFromBytes(v)) + } + if (ltype?.type === 'GEOGRAPHY') { + return data.map(v => parsers.geographyFromBytes(v)) + } + if (ltype?.type === 'UUID') { + return data.map(v => parsers.uuidFromBytes(v)) + } + if (ctype === 'UTF8' || ltype?.type === 'STRING' || utf8 && type === 'BYTE_ARRAY') { + return data.map(v => parsers.stringFromBytes(v)) + } + if (ctype === 'UINT_64' || ltype?.type === 'INTEGER' && ltype.bitWidth === 64 && !ltype.isSigned) { + if (data instanceof BigInt64Array) return new BigUint64Array(data.buffer, data.byteOffset, data.length) + const arr = nullable ? new Array(data.length) : new BigUint64Array(data.length) + for (let i = 0; i < arr.length; i++) arr[i] = data[i] + return arr + } + if (ctype === 'UINT_32' || ltype?.type === 'INTEGER' && ltype.bitWidth === 32 && !ltype.isSigned) { + if (data instanceof Int32Array) return new Uint32Array(data.buffer, data.byteOffset, data.length) + const arr = nullable ? new Array(data.length) : new Uint32Array(data.length) + for (let i = 0; i < arr.length; i++) { + arr[i] = data[i] < 0 ? 4294967296 + data[i] : data[i] + } + return arr + } + if (ltype?.type === 'FLOAT16') { + return Array.from(data).map(parseFloat16) + } + if (ltype?.type === 'TIMESTAMP') { + const { unit } = ltype + /** @type {ParquetParsers[keyof ParquetParsers]} */ + let parser = parsers.timestampFromMilliseconds + if (unit === 'MICROS') parser = parsers.timestampFromMicroseconds + if (unit === 'NANOS') parser = parsers.timestampFromNanoseconds + const arr = new Array(data.length) + for (let i = 0; i < arr.length; i++) { + arr[i] = parser(data[i]) + } + return arr + } + return data +} + +/** + * @param {Uint8Array} bytes + * @returns {number} + */ +export function parseDecimal(bytes) { + if (!bytes.length) return 0 + + let value = 0n + for (const byte of bytes) { + value = value * 256n + BigInt(byte) + } + + // handle signed + const bits = bytes.length * 8 + if (value >= 2n ** BigInt(bits - 1)) { + value -= 2n ** BigInt(bits) + } + + return Number(value) +} + +/** + * Converts INT96 date format (hi 32bit days, lo 64bit nanos) to nanos since epoch + * @param {bigint} value + * @returns {bigint} + */ +function parseInt96Nanos(value) { + const days = (value >> 64n) - 2440588n + const nano = value & 0xffffffffffffffffn + return days * 86400000000000n + nano +} + +/** + * @param {Uint8Array | undefined} bytes + * @returns {number | undefined} + */ +export function parseFloat16(bytes) { + if (!bytes) return undefined + const int16 = bytes[1] << 8 | bytes[0] + const sign = int16 >> 15 ? -1 : 1 + const exp = int16 >> 10 & 0x1f + const frac = int16 & 0x3ff + if (exp === 0) return sign * 2 ** -14 * (frac / 1024) // subnormals + if (exp === 0x1f) return frac ? NaN : sign * Infinity + return sign * 2 ** (exp - 15) * (1 + frac / 1024) +} diff --git a/web/node_modules/hyparquet/src/datapage.js b/web/node_modules/hyparquet/src/datapage.js new file mode 100644 index 0000000000000000000000000000000000000000..998cc92643610b0220e57dcba0d3b715805692d9 --- /dev/null +++ b/web/node_modules/hyparquet/src/datapage.js @@ -0,0 +1,246 @@ +/** + * @import {ColumnDecoder, CompressionCodec, Compressors, DataPage, DataPageHeader, DataPageHeaderV2, DataReader, DecodedArray, PageHeader, SchemaTree} from '../src/types.js' + */ + +import { deltaBinaryUnpack, deltaByteArray, deltaLengthByteArray } from './delta.js' +import { byteStreamSplit, readRleBitPackedHybrid } from './encoding.js' +import { readPlain } from './plain.js' +import { getMaxDefinitionLevel, getMaxRepetitionLevel } from './schema.js' +import { snappyUncompress } from './snappy.js' + +/** + * Read a data page from uncompressed reader. + * + * @param {Uint8Array} bytes raw page data (should already be decompressed) + * @param {DataPageHeader} daph data page header + * @param {ColumnDecoder} columnDecoder + * @returns {DataPage} definition levels, repetition levels, and array of values + */ +export function readDataPage(bytes, daph, { type, element, schemaPath }) { + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength) + const reader = { view, offset: 0 } + /** @type {DecodedArray} */ + let dataPage + + // repetition and definition levels + const repetitionLevels = readRepetitionLevels(reader, daph, schemaPath) + // assert(!repetitionLevels.length || repetitionLevels.length === daph.num_values) + const { definitionLevels, numNulls } = readDefinitionLevels(reader, daph, schemaPath) + // assert(!definitionLevels.length || definitionLevels.length === daph.num_values) + + // read values based on encoding + const nValues = daph.num_values - numNulls + if (daph.encoding === 'PLAIN') { + dataPage = readPlain(reader, type, nValues, element.type_length) + } else if ( + daph.encoding === 'PLAIN_DICTIONARY' || + daph.encoding === 'RLE_DICTIONARY' || + daph.encoding === 'RLE' + ) { + const bitWidth = type === 'BOOLEAN' ? 1 : view.getUint8(reader.offset++) + if (bitWidth) { + dataPage = new Array(nValues) + if (type === 'BOOLEAN') { + readRleBitPackedHybrid(reader, bitWidth, dataPage) + dataPage = dataPage.map(x => !!x) // convert to boolean + } else { + // assert(daph.encoding.endsWith('_DICTIONARY')) + readRleBitPackedHybrid(reader, bitWidth, dataPage, view.byteLength - reader.offset) + } + } else { + dataPage = new Uint8Array(nValues) // nValue zeroes + } + } else if (daph.encoding === 'BYTE_STREAM_SPLIT') { + dataPage = byteStreamSplit(reader, nValues, type, element.type_length) + } else if (daph.encoding === 'DELTA_BINARY_PACKED') { + const int32 = type === 'INT32' + dataPage = int32 ? new Int32Array(nValues) : new BigInt64Array(nValues) + deltaBinaryUnpack(reader, nValues, dataPage) + } else if (daph.encoding === 'DELTA_LENGTH_BYTE_ARRAY') { + dataPage = new Array(nValues) + deltaLengthByteArray(reader, nValues, dataPage) + } else { + throw new Error(`parquet unsupported encoding: ${daph.encoding}`) + } + + return { definitionLevels, repetitionLevels, dataPage } +} + +/** + * @param {DataReader} reader data view for the page + * @param {DataPageHeader} daph data page header + * @param {SchemaTree[]} schemaPath + * @returns {any[]} repetition levels and number of bytes read + */ +function readRepetitionLevels(reader, daph, schemaPath) { + if (schemaPath.length > 1) { + const maxRepetitionLevel = getMaxRepetitionLevel(schemaPath) + if (maxRepetitionLevel) { + const values = new Array(daph.num_values) + readRleBitPackedHybrid(reader, bitWidth(maxRepetitionLevel), values) + return values + } + } + return [] +} + +/** + * @param {DataReader} reader data view for the page + * @param {DataPageHeader} daph data page header + * @param {SchemaTree[]} schemaPath + * @returns {{ definitionLevels: number[], numNulls: number }} definition levels + */ +function readDefinitionLevels(reader, daph, schemaPath) { + const maxDefinitionLevel = getMaxDefinitionLevel(schemaPath) + if (!maxDefinitionLevel) return { definitionLevels: [], numNulls: 0 } + + const definitionLevels = new Array(daph.num_values) + readRleBitPackedHybrid(reader, bitWidth(maxDefinitionLevel), definitionLevels) + + // count nulls + let numNulls = daph.num_values + for (const def of definitionLevels) { + if (def === maxDefinitionLevel) numNulls-- + } + if (numNulls === 0) definitionLevels.length = 0 + + return { definitionLevels, numNulls } +} + +/** + * @param {Uint8Array} compressedBytes + * @param {number} uncompressed_page_size + * @param {CompressionCodec} codec + * @param {Compressors | undefined} compressors + * @returns {Uint8Array} + */ +export function decompressPage(compressedBytes, uncompressed_page_size, codec, compressors) { + /** @type {Uint8Array} */ + let page + const customDecompressor = compressors?.[codec] + if (codec === 'UNCOMPRESSED') { + page = compressedBytes + } else if (customDecompressor) { + page = customDecompressor(compressedBytes, uncompressed_page_size) + } else if (codec === 'SNAPPY') { + page = new Uint8Array(uncompressed_page_size) + snappyUncompress(compressedBytes, page) + } else { + throw new Error(`parquet unsupported compression codec: ${codec}`) + } + if (page?.length !== uncompressed_page_size) { + throw new Error(`parquet decompressed page length ${page?.length} does not match header ${uncompressed_page_size}`) + } + return page +} + + +/** + * Read a data page from the given Uint8Array. + * + * @param {Uint8Array} compressedBytes raw page data + * @param {PageHeader} ph page header + * @param {ColumnDecoder} columnDecoder + * @returns {DataPage} definition levels, repetition levels, and array of values + */ +export function readDataPageV2(compressedBytes, ph, columnDecoder) { + const view = new DataView(compressedBytes.buffer, compressedBytes.byteOffset, compressedBytes.byteLength) + const reader = { view, offset: 0 } + const { type, element, schemaPath, codec, compressors } = columnDecoder + const daph2 = ph.data_page_header_v2 + if (!daph2) throw new Error('parquet data page header v2 is undefined') + + // repetition levels + const repetitionLevels = readRepetitionLevelsV2(reader, daph2, schemaPath) + reader.offset = daph2.repetition_levels_byte_length // readVarInt() => len for boolean v2? + + // definition levels + const definitionLevels = readDefinitionLevelsV2(reader, daph2, schemaPath) + // assert(reader.offset === daph2.repetition_levels_byte_length + daph2.definition_levels_byte_length) + + const uncompressedPageSize = ph.uncompressed_page_size - daph2.definition_levels_byte_length - daph2.repetition_levels_byte_length + + let page = compressedBytes.subarray(reader.offset) + if (daph2.is_compressed !== false) { + page = decompressPage(page, uncompressedPageSize, codec, compressors) + } + const pageView = new DataView(page.buffer, page.byteOffset, page.byteLength) + const pageReader = { view: pageView, offset: 0 } + + // read values based on encoding + /** @type {DecodedArray} */ + let dataPage + const nValues = daph2.num_values - daph2.num_nulls + if (daph2.encoding === 'PLAIN') { + dataPage = readPlain(pageReader, type, nValues, element.type_length) + } else if (daph2.encoding === 'RLE') { + // assert(type === 'BOOLEAN') + dataPage = new Array(nValues) + readRleBitPackedHybrid(pageReader, 1, dataPage) + dataPage = dataPage.map(x => !!x) + } else if ( + daph2.encoding === 'PLAIN_DICTIONARY' || + daph2.encoding === 'RLE_DICTIONARY' + ) { + const bitWidth = pageView.getUint8(pageReader.offset++) + dataPage = new Array(nValues) + readRleBitPackedHybrid(pageReader, bitWidth, dataPage, uncompressedPageSize - 1) + } else if (daph2.encoding === 'DELTA_BINARY_PACKED') { + const int32 = type === 'INT32' + dataPage = int32 ? new Int32Array(nValues) : new BigInt64Array(nValues) + deltaBinaryUnpack(pageReader, nValues, dataPage) + } else if (daph2.encoding === 'DELTA_LENGTH_BYTE_ARRAY') { + dataPage = new Array(nValues) + deltaLengthByteArray(pageReader, nValues, dataPage) + } else if (daph2.encoding === 'DELTA_BYTE_ARRAY') { + dataPage = new Array(nValues) + deltaByteArray(pageReader, nValues, dataPage) + } else if (daph2.encoding === 'BYTE_STREAM_SPLIT') { + dataPage = byteStreamSplit(pageReader, nValues, type, element.type_length) + } else { + throw new Error(`parquet unsupported encoding: ${daph2.encoding}`) + } + + return { definitionLevels, repetitionLevels, dataPage } +} + +/** + * @param {DataReader} reader + * @param {DataPageHeaderV2} daph2 data page header v2 + * @param {SchemaTree[]} schemaPath + * @returns {any[]} repetition levels + */ +function readRepetitionLevelsV2(reader, daph2, schemaPath) { + const maxRepetitionLevel = getMaxRepetitionLevel(schemaPath) + if (!maxRepetitionLevel) return [] + + const values = new Array(daph2.num_values) + readRleBitPackedHybrid(reader, bitWidth(maxRepetitionLevel), values, daph2.repetition_levels_byte_length) + return values +} + +/** + * @param {DataReader} reader + * @param {DataPageHeaderV2} daph2 data page header v2 + * @param {SchemaTree[]} schemaPath + * @returns {number[] | undefined} definition levels + */ +function readDefinitionLevelsV2(reader, daph2, schemaPath) { + const maxDefinitionLevel = getMaxDefinitionLevel(schemaPath) + if (maxDefinitionLevel) { + // V2 we know the length + const values = new Array(daph2.num_values) + readRleBitPackedHybrid(reader, bitWidth(maxDefinitionLevel), values, daph2.definition_levels_byte_length) + return values + } +} + +/** + * Minimum bits needed to store value. + * + * @param {number} value + * @returns {number} + */ +function bitWidth(value) { + return 32 - Math.clz32(value) +} diff --git a/web/node_modules/hyparquet/src/delta.js b/web/node_modules/hyparquet/src/delta.js new file mode 100644 index 0000000000000000000000000000000000000000..b275313174f8a43f050e9d7d8e7f8b912a560f61 --- /dev/null +++ b/web/node_modules/hyparquet/src/delta.js @@ -0,0 +1,104 @@ +/** + * @import {DataReader} from '../src/types.js' + */ + +import { readVarInt, readZigZagBigInt } from './thrift.js' + +/** + * @param {DataReader} reader + * @param {number} count number of values to read + * @param {Int32Array | BigInt64Array} output + */ +export function deltaBinaryUnpack(reader, count, output) { + const int32 = output instanceof Int32Array + const blockSize = readVarInt(reader) + const miniblockPerBlock = readVarInt(reader) + readVarInt(reader) // assert(=== count) + let value = readZigZagBigInt(reader) // first value + let outputIndex = 0 + output[outputIndex++] = int32 ? Number(value) : value + + const valuesPerMiniblock = blockSize / miniblockPerBlock + + while (outputIndex < count) { + // new block + const minDelta = readZigZagBigInt(reader) + const bitWidths = new Uint8Array(miniblockPerBlock) + for (let i = 0; i < miniblockPerBlock; i++) { + bitWidths[i] = reader.view.getUint8(reader.offset++) + } + + for (let i = 0; i < miniblockPerBlock && outputIndex < count; i++) { + // new miniblock + const bitWidth = BigInt(bitWidths[i]) + if (bitWidth) { + let bitpackPos = 0n + let miniblockCount = valuesPerMiniblock + const mask = (1n << bitWidth) - 1n + while (miniblockCount && outputIndex < count) { + let bits = BigInt(reader.view.getUint8(reader.offset)) >> bitpackPos & mask // TODO: don't re-read value every time + bitpackPos += bitWidth + while (bitpackPos >= 8) { + bitpackPos -= 8n + reader.offset++ + if (bitpackPos) { + bits |= BigInt(reader.view.getUint8(reader.offset)) << bitWidth - bitpackPos & mask + } + } + const delta = minDelta + bits + value += delta + output[outputIndex++] = int32 ? Number(value) : value + miniblockCount-- + } + if (miniblockCount) { + // consume leftover miniblock + reader.offset += Math.ceil((miniblockCount * Number(bitWidth) + Number(bitpackPos)) / 8) + } + } else { + for (let j = 0; j < valuesPerMiniblock && outputIndex < count; j++) { + value += minDelta + output[outputIndex++] = int32 ? Number(value) : value + } + } + } + } +} + +/** + * @param {DataReader} reader + * @param {number} count + * @param {Uint8Array[]} output + */ +export function deltaLengthByteArray(reader, count, output) { + const lengths = new Int32Array(count) + deltaBinaryUnpack(reader, count, lengths) + for (let i = 0; i < count; i++) { + output[i] = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, lengths[i]) + reader.offset += lengths[i] + } +} + +/** + * @param {DataReader} reader + * @param {number} count + * @param {Uint8Array[]} output + */ +export function deltaByteArray(reader, count, output) { + const prefixData = new Int32Array(count) + deltaBinaryUnpack(reader, count, prefixData) + const suffixData = new Int32Array(count) + deltaBinaryUnpack(reader, count, suffixData) + + for (let i = 0; i < count; i++) { + const suffix = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, suffixData[i]) + if (prefixData[i]) { + // copy from previous value + output[i] = new Uint8Array(prefixData[i] + suffixData[i]) + output[i].set(output[i - 1].subarray(0, prefixData[i])) + output[i].set(suffix, prefixData[i]) + } else { + output[i] = suffix + } + reader.offset += suffixData[i] + } +} diff --git a/web/node_modules/hyparquet/src/encoding.js b/web/node_modules/hyparquet/src/encoding.js new file mode 100644 index 0000000000000000000000000000000000000000..3ffd6b3a292800cb228c66e2fa5cb16946964032 --- /dev/null +++ b/web/node_modules/hyparquet/src/encoding.js @@ -0,0 +1,162 @@ +/** + * @import {DataReader, DecodedArray, ParquetType} from '../src/types.js' + */ + +import { readVarInt } from './thrift.js' + +/** + * Read values from a run-length encoded/bit-packed hybrid encoding. + * + * If length is zero, then read int32 length at the start. + * + * @param {DataReader} reader + * @param {number} width - bitwidth + * @param {DecodedArray} output + * @param {number} [length] - length of the encoded data + */ +export function readRleBitPackedHybrid(reader, width, output, length) { + if (length === undefined) { + length = reader.view.getUint32(reader.offset, true) + reader.offset += 4 + } + const startOffset = reader.offset + let seen = 0 + while (seen < output.length) { + const header = readVarInt(reader) + if (header & 1) { + // bit-packed + seen = readBitPacked(reader, header, width, output, seen) + } else { + // rle + const count = header >>> 1 + readRle(reader, count, width, output, seen) + seen += count + } + } + reader.offset = startOffset + length // duckdb writes an empty block +} + +/** + * Run-length encoding: read value with bitWidth and repeat it count times. + * + * @param {DataReader} reader + * @param {number} count + * @param {number} bitWidth + * @param {DecodedArray} output + * @param {number} seen + */ +function readRle(reader, count, bitWidth, output, seen) { + const width = bitWidth + 7 >> 3 + let value = 0 + for (let i = 0; i < width; i++) { + value |= reader.view.getUint8(reader.offset++) << (i << 3) + } + // assert(value < 1 << bitWidth) + + // repeat value count times + for (let i = 0; i < count; i++) { + output[seen + i] = value + } +} + +/** + * Read a bit-packed run of the rle/bitpack hybrid. + * Supports width > 8 (crossing bytes). + * + * @param {DataReader} reader + * @param {number} header - bit-pack header + * @param {number} bitWidth + * @param {DecodedArray} output + * @param {number} seen + * @returns {number} total output values so far + */ +function readBitPacked(reader, header, bitWidth, output, seen) { + let count = header >> 1 << 3 // values to read + const mask = (1 << bitWidth) - 1 + + let data = 0 + if (reader.offset < reader.view.byteLength) { + data = reader.view.getUint8(reader.offset++) + } else if (mask) { + // sometimes out-of-bounds reads are masked out + throw new Error(`parquet bitpack offset ${reader.offset} out of range`) + } + let left = 8 + let right = 0 + + // read values + while (count) { + // if we have crossed a byte boundary, shift the data + if (right > 8) { + right -= 8 + left -= 8 + data >>>= 8 + } else if (left - right < bitWidth) { + // if we don't have bitWidth number of bits to read, read next byte + data |= reader.view.getUint8(reader.offset) << left + reader.offset++ + left += 8 + } else { + if (seen < output.length) { + // emit value + output[seen++] = data >> right & mask + } + count-- + right += bitWidth + } + } + + return seen +} + +/** + * @param {DataReader} reader + * @param {number} count + * @param {ParquetType} type + * @param {number | undefined} typeLength + * @returns {DecodedArray} + */ +export function byteStreamSplit(reader, count, type, typeLength) { + const width = byteWidth(type, typeLength) + const bytes = new Uint8Array(count * width) + for (let b = 0; b < width; b++) { + for (let i = 0; i < count; i++) { + bytes[i * width + b] = reader.view.getUint8(reader.offset++) + } + } + // interpret bytes as typed array + if (type === 'FLOAT') return new Float32Array(bytes.buffer) + else if (type === 'DOUBLE') return new Float64Array(bytes.buffer) + else if (type === 'INT32') return new Int32Array(bytes.buffer) + else if (type === 'INT64') return new BigInt64Array(bytes.buffer) + else if (type === 'FIXED_LEN_BYTE_ARRAY') { + // split into arrays of typeLength + const split = new Array(count) + for (let i = 0; i < count; i++) { + split[i] = bytes.subarray(i * width, (i + 1) * width) + } + return split + } + throw new Error(`parquet byte_stream_split unsupported type: ${type}`) +} + +/** + * @param {ParquetType} type + * @param {number | undefined} typeLength + * @returns {number} + */ +function byteWidth(type, typeLength) { + switch (type) { + case 'INT32': + case 'FLOAT': + return 4 + case 'INT64': + case 'DOUBLE': + return 8 + case 'FIXED_LEN_BYTE_ARRAY': + if (!typeLength) throw new Error('parquet byteWidth missing type_length') + return typeLength + default: + throw new Error(`parquet unsupported type: ${type}`) + } +} diff --git a/web/node_modules/hyparquet/src/filter.js b/web/node_modules/hyparquet/src/filter.js new file mode 100644 index 0000000000000000000000000000000000000000..9bafb6ef6c748153f6b3f6f77548c5305b32a37e --- /dev/null +++ b/web/node_modules/hyparquet/src/filter.js @@ -0,0 +1,170 @@ +/** + * @import {BloomFilter, ParquetQueryFilter, RowGroup, SchemaElement} from '../src/types.js' + */ + +import { hashParquetValue, sbbfContains } from './bloom.js' +import { equals } from './utils.js' + +/** + * Returns an array of top-level column names needed to evaluate the filter. + * + * @param {ParquetQueryFilter} [filter] + * @returns {string[]} + */ +export function columnsNeededForFilter(filter) { + if (!filter) return [] + /** @type {string[]} */ + const columns = [] + if ('$and' in filter && Array.isArray(filter.$and)) { + columns.push(...filter.$and.flatMap(columnsNeededForFilter)) + } else if ('$or' in filter && Array.isArray(filter.$or)) { + columns.push(...filter.$or.flatMap(columnsNeededForFilter)) + } else if ('$nor' in filter && Array.isArray(filter.$nor)) { + columns.push(...filter.$nor.flatMap(columnsNeededForFilter)) + } else { + // Map dot-notation paths to top-level column names + columns.push(...Object.keys(filter).map(key => key.split('.')[0])) + } + return [...new Set(columns)] +} + +/** + * Match a record against a query filter + * + * @param {Record} record + * @param {ParquetQueryFilter} filter + * @param {boolean} [strict] + * @returns {boolean} + */ +export function matchFilter(record, filter, strict = true) { + if ('$and' in filter && Array.isArray(filter.$and)) { + return filter.$and.every(subQuery => matchFilter(record, subQuery, strict)) + } + if ('$or' in filter && Array.isArray(filter.$or)) { + return filter.$or.some(subQuery => matchFilter(record, subQuery, strict)) + } + if ('$nor' in filter && Array.isArray(filter.$nor)) { + return !filter.$nor.some(subQuery => matchFilter(record, subQuery, strict)) + } + + return Object.entries(filter).every(([field, condition]) => { + const value = resolve(record, field) + + // implicit $eq for non-object conditions + if (typeof condition !== 'object' || condition === null || Array.isArray(condition)) { + return equals(value, condition, strict) + } + + return Object.entries(condition || {}).every(([operator, target]) => { + if (operator === '$gt') return value > target + if (operator === '$gte') return value >= target + if (operator === '$lt') return value < target + if (operator === '$lte') return value <= target + if (operator === '$eq') return equals(value, target, strict) + if (operator === '$ne') return !equals(value, target, strict) + if (operator === '$in') return Array.isArray(target) && target.includes(value) + if (operator === '$nin') return Array.isArray(target) && !target.includes(value) + if (operator === '$not') return !matchFilter({ [field]: value }, { [field]: target }, strict) + return true + }) + }) +} + +/** + * Check if a row group can be skipped based on filter and column statistics, + * optionally consulting per-column bloom filters for equality predicates that + * statistics can't decide. + * + * @param {object} options + * @param {RowGroup} options.rowGroup + * @param {string[]} options.physicalColumns + * @param {ParquetQueryFilter | undefined} options.filter + * @param {boolean} [options.strict] + * @param {Record} [options.bloomFilters] keyed by top-level column name + * @param {Record} [options.schemaElements] keyed by top-level column name, required to use bloomFilters + * @returns {boolean} true if the row group can be skipped + */ +export function canSkipRowGroup({ rowGroup, physicalColumns, filter, strict = true, bloomFilters, schemaElements }) { + if (!filter) return false + + // Handle logical operators + if ('$and' in filter && Array.isArray(filter.$and)) { + // For AND, we can skip if ANY condition allows skipping + return filter.$and.some(subFilter => canSkipRowGroup({ rowGroup, physicalColumns, filter: subFilter, strict, bloomFilters, schemaElements })) + } + if ('$or' in filter && Array.isArray(filter.$or)) { + // For OR, we can skip only if ALL conditions allow skipping + return filter.$or.every(subFilter => canSkipRowGroup({ rowGroup, physicalColumns, filter: subFilter, strict, bloomFilters, schemaElements })) + } + if ('$nor' in filter && Array.isArray(filter.$nor)) { + // For NOR, we can skip if none of the conditions allow skipping + // This is complex, so we'll be conservative and not skip + return false + } + + // Check column filters + for (const [field, condition] of Object.entries(filter)) { + // Find the column chunk for this field + const columnIndex = physicalColumns.indexOf(field) + if (columnIndex === -1) continue + + const stats = rowGroup.columns[columnIndex].meta_data?.statistics + const { min, max, min_value, max_value } = stats || {} + const minVal = min_value !== undefined ? min_value : min + const maxVal = max_value !== undefined ? max_value : max + const haveStats = minVal !== undefined && maxVal !== undefined + + const bloom = bloomFilters?.[field] + const element = schemaElements?.[field] + + // Handle operators + for (const [operator, target] of Object.entries(condition || {})) { + // Statistics-based skipping + if (haveStats) { + if (operator === '$gt' && maxVal <= target) return true + if (operator === '$gte' && maxVal < target) return true + if (operator === '$lt' && minVal >= target) return true + if (operator === '$lte' && minVal > target) return true + if (operator === '$eq' && (target < minVal || target > maxVal)) return true + if (operator === '$ne' && equals(minVal, maxVal, strict) && equals(minVal, target, strict)) return true + if (operator === '$in' && Array.isArray(target) && target.every(v => v < minVal || v > maxVal)) return true + if (operator === '$nin' && Array.isArray(target) && equals(minVal, maxVal, strict) && target.includes(minVal)) return true + } + // Bloom-filter skipping for equality predicates (proves absence, never presence) + if (bloom && element) { + if (operator === '$eq') { + const hash = hashParquetValue(target, element) + if (hash !== undefined && !sbbfContains(bloom.blocks, hash)) return true + } + if (operator === '$in' && Array.isArray(target) && target.length > 0) { + let allAbsent = true + for (const v of target) { + const h = hashParquetValue(v, element) + if (h === undefined || sbbfContains(bloom.blocks, h)) { + allAbsent = false + break + } + } + if (allAbsent) return true + } + } + } + } + + return false +} + +/** + * Resolve a dot-notation path to a value in a nested object. + * + * @param {Record} record + * @param {string} path + * @returns {any} + */ +function resolve(record, path) { + let value = record + for (const part of path.split('.')) { + value = value?.[part] + } + return value +} diff --git a/web/node_modules/hyparquet/src/geoparquet.js b/web/node_modules/hyparquet/src/geoparquet.js new file mode 100644 index 0000000000000000000000000000000000000000..45183a1c7acf4d247274cccf0e477fc3132ab590 --- /dev/null +++ b/web/node_modules/hyparquet/src/geoparquet.js @@ -0,0 +1,38 @@ +/** + * @param {SchemaElement[]} schema + * @param {KeyValue[] | undefined} key_value_metadata + * @returns {void} + */ +export function markGeoColumns(schema, key_value_metadata) { + // Prepare the list of GeoParquet columns + /** @type {Map} */ + const columns = new Map() + const geo = key_value_metadata?.find(({ key }) => key === 'geo')?.value + const decodedColumns = (geo && JSON.parse(geo)?.columns) ?? {} + for (const [name, column] of Object.entries(decodedColumns)) { + if (column.encoding !== 'WKB') continue + + const type = column.edges === 'spherical' ? 'GEOGRAPHY' : 'GEOMETRY' + const id = column.crs?.id ?? column.crs?.ids?.[0] + const crs = id ? `${id.authority}:${id.code.toString()}` : undefined + // Note: we can't infer GEOGRAPHY's algorithm from GeoParquet + columns.set(name, { type, crs }) + } + + // Mark schema elements with logical type + // Only look at root-level columns of type BYTE_ARRAY without existing logical_type + for (let i = 1; i < schema.length; i++) { // skip root + const { logical_type, name, num_children, type } = schema[i] + if (num_children) { + i += num_children + continue // skip the element and its children + } + if (type === 'BYTE_ARRAY' && !logical_type) { + schema[i].logical_type = columns.get(name) + } + } +} + +/** + * @import {KeyValue, LogicalType, SchemaElement} from '../src/types.js' + */ diff --git a/web/node_modules/hyparquet/src/index.js b/web/node_modules/hyparquet/src/index.js new file mode 100644 index 0000000000000000000000000000000000000000..c024e2fa575dcae5ae9e20ff158d17a6608e5cf0 --- /dev/null +++ b/web/node_modules/hyparquet/src/index.js @@ -0,0 +1,51 @@ +export { readColumnIndex, readOffsetIndex } from './indexes.js' +export { parquetMetadata, parquetMetadataAsync, parquetSchema } from './metadata.js' +export { parquetRead, parquetReadObjects } from './read.js' +export { parquetQuery } from './query.js' +export { snappyUncompress } from './snappy.js' +export { asyncBufferFromUrl, byteLengthFromUrl, cachedAsyncBuffer, flatten, toJson } from './utils.js' + +/** + * Explicitly export types for use in downstream typescript projects through + * `import { ParquetReadOptions } from 'hyparquet'` for example. + * + * @template {any} T + * @typedef {import('../src/types.d.ts').Awaitable} Awaitable + */ +/** + * @typedef {import('../src/types.d.ts').AsyncBuffer} AsyncBuffer + * @typedef {import('../src/types.d.ts').AsyncRowGroup} AsyncRowGroup + * @typedef {import('../src/types.d.ts').DataReader} DataReader + * @typedef {import('../src/types.d.ts').FileMetaData} FileMetaData + * @typedef {import('../src/types.d.ts').SchemaTree} SchemaTree + * @typedef {import('../src/types.d.ts').SchemaElement} SchemaElement + * @typedef {import('../src/types.d.ts').ParquetType} ParquetType + * @typedef {import('../src/types.d.ts').FieldRepetitionType} FieldRepetitionType + * @typedef {import('../src/types.d.ts').ConvertedType} ConvertedType + * @typedef {import('../src/types.d.ts').TimeUnit} TimeUnit + * @typedef {import('../src/types.d.ts').LogicalType} LogicalType + * @typedef {import('../src/types.d.ts').RowGroup} RowGroup + * @typedef {import('../src/types.d.ts').ColumnChunk} ColumnChunk + * @typedef {import('../src/types.d.ts').ColumnMetaData} ColumnMetaData + * @typedef {import('../src/types.d.ts').Encoding} Encoding + * @typedef {import('../src/types.d.ts').CompressionCodec} CompressionCodec + * @typedef {import('../src/types.d.ts').Compressors} Compressors + * @typedef {import('../src/types.d.ts').KeyValue} KeyValue + * @typedef {import('../src/types.d.ts').Statistics} Statistics + * @typedef {import('../src/types.d.ts').GeospatialStatistics} GeospatialStatistics + * @typedef {import('../src/types.d.ts').BoundingBox} BoundingBox + * @typedef {import('../src/types.d.ts').PageType} PageType + * @typedef {import('../src/types.d.ts').PageHeader} PageHeader + * @typedef {import('../src/types.d.ts').DataPageHeader} DataPageHeader + * @typedef {import('../src/types.d.ts').DictionaryPageHeader} DictionaryPageHeader + * @typedef {import('../src/types.d.ts').DecodedArray} DecodedArray + * @typedef {import('../src/types.d.ts').OffsetIndex} OffsetIndex + * @typedef {import('../src/types.d.ts').ColumnIndex} ColumnIndex + * @typedef {import('../src/types.d.ts').BoundaryOrder} BoundaryOrder + * @typedef {import('../src/types.d.ts').ColumnData} ColumnData + * @typedef {import('../src/types.d.ts').SubColumnData} SubColumnData + * @typedef {import('../src/types.d.ts').ParquetReadOptions} ParquetReadOptions + * @typedef {import('../src/types.d.ts').MetadataOptions} MetadataOptions + * @typedef {import('../src/types.d.ts').ParquetParsers} ParquetParsers + * @typedef {import('../src/types.d.ts').ParquetQueryFilter} ParquetQueryFilter + */ diff --git a/web/node_modules/hyparquet/src/indexes.js b/web/node_modules/hyparquet/src/indexes.js new file mode 100644 index 0000000000000000000000000000000000000000..e12509eaa9d7955dd0d83ef2dd8215d395bd7559 --- /dev/null +++ b/web/node_modules/hyparquet/src/indexes.js @@ -0,0 +1,46 @@ +/** + * @import {ColumnIndex, DataReader, OffsetIndex, PageLocation, ParquetParsers, SchemaElement} from '../src/types.js' + */ + +import { BoundaryOrders } from './constants.js' +import { DEFAULT_PARSERS } from './convert.js' +import { convertMetadata } from './metadata.js' +import { deserializeTCompactProtocol } from './thrift.js' + +/** + * @param {DataReader} reader + * @param {SchemaElement} schema + * @param {ParquetParsers | undefined} parsers + * @returns {ColumnIndex} + */ +export function readColumnIndex(reader, schema, parsers = undefined) { + parsers = { ...DEFAULT_PARSERS, ...parsers } + + const thrift = deserializeTCompactProtocol(reader) + return { + null_pages: thrift.field_1, + min_values: thrift.field_2.map((/** @type {any} */ m) => convertMetadata(m, schema, parsers)), + max_values: thrift.field_3.map((/** @type {any} */ m) => convertMetadata(m, schema, parsers)), + boundary_order: BoundaryOrders[thrift.field_4], + null_counts: thrift.field_5, + repetition_level_histograms: thrift.field_6, + definition_level_histograms: thrift.field_7, + } +} + +/** + * @param {DataReader} reader + * @returns {OffsetIndex} + */ +export function readOffsetIndex(reader) { + const thrift = deserializeTCompactProtocol(reader) + return { + // @ts-ignore + page_locations: thrift.field_1.map(loc => ({ + offset: loc.field_1, + compressed_page_size: loc.field_2, + first_row_index: loc.field_3, + })), + unencoded_byte_array_data_bytes: thrift.field_2, + } +} diff --git a/web/node_modules/hyparquet/src/metadata.js b/web/node_modules/hyparquet/src/metadata.js new file mode 100644 index 0000000000000000000000000000000000000000..78a5605d912ebec90a594266c623e0ed198892dc --- /dev/null +++ b/web/node_modules/hyparquet/src/metadata.js @@ -0,0 +1,339 @@ +import { CompressionCodecs, ConvertedTypes, EdgeInterpolationAlgorithms, Encodings, FieldRepetitionTypes, PageTypes, ParquetTypes } from './constants.js' +import { DEFAULT_PARSERS, parseDecimal, parseFloat16 } from './convert.js' +import { getSchemaPath } from './schema.js' +import { deserializeTCompactProtocol } from './thrift.js' +import { markGeoColumns } from './geoparquet.js' + +/** + * @import {AsyncBuffer, FileMetaData, KeyValue, LogicalType, MetadataOptions, MinMaxType, ParquetParsers, SchemaElement, SchemaTree, Statistics, TimeUnit} from '../src/types.js' + */ + +export const defaultInitialFetchSize = 1 << 19 // 512kb + +const decoder = new TextDecoder() +function decode(/** @type {Uint8Array} */ value) { + return value && decoder.decode(value) +} + +/** + * Read parquet metadata from an async buffer. + * + * An AsyncBuffer is like an ArrayBuffer, but the slices are loaded + * asynchronously, possibly over the network. + * + * You must provide the byteLength of the buffer, typically from a HEAD request. + * + * In theory, you could use suffix-range requests to fetch the end of the file, + * and save a round trip. But in practice, this doesn't work because chrome + * deems suffix-range requests as a not-safe-listed header, and will require + * a pre-flight. So the byteLength is required. + * + * To make this efficient, we initially request the last 512kb of the file, + * which is likely to contain the metadata. If the metadata length exceeds the + * initial fetch, 512kb, we request the rest of the metadata from the AsyncBuffer. + * + * This ensures that we either make one 512kb initial request for the metadata, + * or a second request for up to the metadata size. + * + * @param {AsyncBuffer} asyncBuffer parquet file contents + * @param {MetadataOptions & { initialFetchSize?: number }} options initial fetch size in bytes (default 512kb) + * @returns {Promise} parquet metadata object + */ +export async function parquetMetadataAsync(asyncBuffer, { parsers, initialFetchSize = defaultInitialFetchSize, geoparquet = true } = {}) { + if (!asyncBuffer || !(asyncBuffer.byteLength >= 0)) throw new Error('parquet expected AsyncBuffer') + + // fetch last bytes (footer) of the file + const footerOffset = Math.max(0, asyncBuffer.byteLength - initialFetchSize) + const footerBuffer = await asyncBuffer.slice(footerOffset, asyncBuffer.byteLength) + + // Check for parquet magic number "PAR1" + const footerView = new DataView(footerBuffer) + if (footerView.getUint32(footerBuffer.byteLength - 4, true) !== 0x31524150) { + throw new Error('parquet file invalid (footer != PAR1)') + } + + // Parquet files store metadata at the end of the file + // Metadata length is 4 bytes before the last PAR1 + const metadataLength = footerView.getUint32(footerBuffer.byteLength - 8, true) + if (metadataLength > asyncBuffer.byteLength - 8) { + throw new Error(`parquet metadata length ${metadataLength} exceeds available buffer ${asyncBuffer.byteLength - 8}`) + } + + // check if metadata size fits inside the initial fetch + if (metadataLength + 8 > initialFetchSize) { + // fetch the rest of the metadata + const metadataOffset = asyncBuffer.byteLength - metadataLength - 8 + const metadataBuffer = await asyncBuffer.slice(metadataOffset, footerOffset) + // combine initial fetch with the new slice + const combinedBuffer = new ArrayBuffer(metadataLength + 8) + const combinedView = new Uint8Array(combinedBuffer) + combinedView.set(new Uint8Array(metadataBuffer)) + combinedView.set(new Uint8Array(footerBuffer), footerOffset - metadataOffset) + return parquetMetadata(combinedBuffer, { parsers, geoparquet }) + } else { + // parse metadata from the footer + return parquetMetadata(footerBuffer, { parsers, geoparquet }) + } +} + +/** + * Read parquet metadata from a buffer synchronously. + * + * @param {ArrayBuffer} arrayBuffer parquet file footer + * @param {MetadataOptions} options metadata parsing options + * @returns {FileMetaData} parquet metadata object + */ +export function parquetMetadata(arrayBuffer, { parsers, geoparquet = true } = {}) { + if (!(arrayBuffer instanceof ArrayBuffer)) throw new Error('parquet expected ArrayBuffer') + const view = new DataView(arrayBuffer) + + // Use default parsers if not given + parsers = { ...DEFAULT_PARSERS, ...parsers } + + // Validate footer magic number "PAR1" + if (view.byteLength < 8) { + throw new Error('parquet file is too short') + } + if (view.getUint32(view.byteLength - 4, true) !== 0x31524150) { + throw new Error('parquet file invalid (footer != PAR1)') + } + + // Parquet files store metadata at the end of the file + // Metadata length is 4 bytes before the last PAR1 + const metadataLengthOffset = view.byteLength - 8 + const metadataLength = view.getUint32(metadataLengthOffset, true) + if (metadataLength > view.byteLength - 8) { + // {metadata}, metadata_length, PAR1 + throw new Error(`parquet metadata length ${metadataLength} exceeds available buffer ${view.byteLength - 8}`) + } + + const metadataOffset = metadataLengthOffset - metadataLength + const reader = { view, offset: metadataOffset } + const metadata = deserializeTCompactProtocol(reader) + + // Parse metadata from thrift data + const version = metadata.field_1 + /** @type {SchemaElement[]} */ + const schema = metadata.field_2.map((/** @type {any} */ field) => ({ + type: ParquetTypes[field.field_1], + type_length: field.field_2, + repetition_type: FieldRepetitionTypes[field.field_3], + name: decode(field.field_4), + num_children: field.field_5, + converted_type: ConvertedTypes[field.field_6], + scale: field.field_7, + precision: field.field_8, + field_id: field.field_9, + logical_type: logicalType(field.field_10), + })) + // schema element per column index + const columnSchema = schema.filter(e => e.type) + const num_rows = metadata.field_3 + const row_groups = metadata.field_4.map((/** @type {any} */ rowGroup) => ({ + columns: rowGroup.field_1.map((/** @type {any} */ column, /** @type {number} */ columnIndex) => ({ + file_path: decode(column.field_1), + file_offset: column.field_2, + meta_data: column.field_3 && { + type: ParquetTypes[column.field_3.field_1], + encodings: column.field_3.field_2?.map((/** @type {number} */ e) => Encodings[e]), + path_in_schema: column.field_3.field_3.map(decode), + codec: CompressionCodecs[column.field_3.field_4], + num_values: column.field_3.field_5, + total_uncompressed_size: column.field_3.field_6, + total_compressed_size: column.field_3.field_7, + key_value_metadata: column.field_3.field_8?.map((/** @type {any} */ kv) => ({ + key: decode(kv.field_1), + value: decode(kv.field_2), + })), + data_page_offset: column.field_3.field_9, + index_page_offset: column.field_3.field_10, + dictionary_page_offset: column.field_3.field_11, + statistics: convertStats(column.field_3.field_12, columnSchema[columnIndex], parsers), + encoding_stats: column.field_3.field_13?.map((/** @type {any} */ encodingStat) => ({ + page_type: PageTypes[encodingStat.field_1], + encoding: Encodings[encodingStat.field_2], + count: encodingStat.field_3, + })), + bloom_filter_offset: column.field_3.field_14, + bloom_filter_length: column.field_3.field_15, + size_statistics: column.field_3.field_16 && { + unencoded_byte_array_data_bytes: column.field_3.field_16.field_1, + repetition_level_histogram: column.field_3.field_16.field_2, + definition_level_histogram: column.field_3.field_16.field_3, + }, + geospatial_statistics: column.field_3.field_17 && { + bbox: column.field_3.field_17.field_1 && { + xmin: column.field_3.field_17.field_1.field_1, + xmax: column.field_3.field_17.field_1.field_2, + ymin: column.field_3.field_17.field_1.field_3, + ymax: column.field_3.field_17.field_1.field_4, + zmin: column.field_3.field_17.field_1.field_5, + zmax: column.field_3.field_17.field_1.field_6, + mmin: column.field_3.field_17.field_1.field_7, + mmax: column.field_3.field_17.field_1.field_8, + }, + geospatial_types: column.field_3.field_17.field_2, + }, + }, + offset_index_offset: column.field_4, + offset_index_length: column.field_5, + column_index_offset: column.field_6, + column_index_length: column.field_7, + crypto_metadata: column.field_8, + encrypted_column_metadata: column.field_9, + })), + total_byte_size: rowGroup.field_2, + num_rows: rowGroup.field_3, + sorting_columns: rowGroup.field_4?.map((/** @type {any} */ sortingColumn) => ({ + column_idx: sortingColumn.field_1, + descending: sortingColumn.field_2, + nulls_first: sortingColumn.field_3, + })), + file_offset: rowGroup.field_5, + total_compressed_size: rowGroup.field_6, + ordinal: rowGroup.field_7, + })) + /** @type {KeyValue[] | undefined} */ + const key_value_metadata = metadata.field_5?.map((/** @type {any} */ kv) => ({ + key: decode(kv.field_1), + value: decode(kv.field_2), + })) + const created_by = decode(metadata.field_6) + + if (geoparquet) { + markGeoColumns(schema, key_value_metadata) + } + + return { + version, + schema, + num_rows, + row_groups, + key_value_metadata, + created_by, + metadata_length: metadataLength, + } +} + +/** + * Return a tree of schema elements from parquet metadata. + * + * @param {{schema: SchemaElement[]}} metadata parquet metadata object + * @returns {SchemaTree} tree of schema elements + */ +export function parquetSchema({ schema }) { + return getSchemaPath(schema, [])[0] +} + +/** + * @param {any} logicalType + * @returns {LogicalType | undefined} + */ +function logicalType(logicalType) { + if (logicalType?.field_1) return { type: 'STRING' } + if (logicalType?.field_2) return { type: 'MAP' } + if (logicalType?.field_3) return { type: 'LIST' } + if (logicalType?.field_4) return { type: 'ENUM' } + if (logicalType?.field_5) return { + type: 'DECIMAL', + scale: logicalType.field_5.field_1, + precision: logicalType.field_5.field_2, + } + if (logicalType?.field_6) return { type: 'DATE' } + if (logicalType?.field_7) return { + type: 'TIME', + isAdjustedToUTC: logicalType.field_7.field_1, + unit: timeUnit(logicalType.field_7.field_2), + } + if (logicalType?.field_8) return { + type: 'TIMESTAMP', + isAdjustedToUTC: logicalType.field_8.field_1, + unit: timeUnit(logicalType.field_8.field_2), + } + if (logicalType?.field_10) return { + type: 'INTEGER', + bitWidth: logicalType.field_10.field_1, + isSigned: logicalType.field_10.field_2, + } + if (logicalType?.field_11) return { type: 'NULL' } + if (logicalType?.field_12) return { type: 'JSON' } + if (logicalType?.field_13) return { type: 'BSON' } + if (logicalType?.field_14) return { type: 'UUID' } + if (logicalType?.field_15) return { type: 'FLOAT16' } + if (logicalType?.field_16) return { + type: 'VARIANT', + specification_version: logicalType.field_16.field_1, + } + if (logicalType?.field_17) return { + type: 'GEOMETRY', + crs: decode(logicalType.field_17.field_1), + } + if (logicalType?.field_18) return { + type: 'GEOGRAPHY', + crs: decode(logicalType.field_18.field_1), + algorithm: EdgeInterpolationAlgorithms[logicalType.field_18.field_2], + } + return logicalType +} + +/** + * @param {any} unit + * @returns {TimeUnit} + */ +function timeUnit(unit) { + if (unit.field_1) return 'MILLIS' + if (unit.field_2) return 'MICROS' + if (unit.field_3) return 'NANOS' + throw new Error('parquet time unit required') +} + +/** + * Convert column statistics based on column type. + * + * @param {any} stats + * @param {SchemaElement} schema + * @param {ParquetParsers} parsers + * @returns {Statistics} + */ +function convertStats(stats, schema, parsers) { + return stats && { + max: convertMetadata(stats.field_1, schema, parsers), + min: convertMetadata(stats.field_2, schema, parsers), + null_count: stats.field_3, + distinct_count: stats.field_4, + max_value: convertMetadata(stats.field_5, schema, parsers), + min_value: convertMetadata(stats.field_6, schema, parsers), + is_max_value_exact: stats.field_7, + is_min_value_exact: stats.field_8, + } +} + +/** + * @param {Uint8Array | undefined} value + * @param {SchemaElement} schema + * @param {ParquetParsers} parsers + * @returns {MinMaxType | undefined} + */ +export function convertMetadata(value, schema, parsers) { + const { type, converted_type, logical_type } = schema + if (value === undefined) return value + if (type === 'BOOLEAN') return value[0] === 1 + if (type === 'BYTE_ARRAY') return parsers.stringFromBytes(value) + const view = new DataView(value.buffer, value.byteOffset, value.byteLength) + if (type === 'FLOAT' && view.byteLength === 4) return view.getFloat32(0, true) + if (type === 'DOUBLE' && view.byteLength === 8) return view.getFloat64(0, true) + if (type === 'INT32' && converted_type === 'DATE') return parsers.dateFromDays(view.getInt32(0, true)) + if (type === 'INT64' && converted_type === 'TIMESTAMP_MILLIS') return parsers.timestampFromMilliseconds(view.getBigInt64(0, true)) + if (type === 'INT64' && converted_type === 'TIMESTAMP_MICROS') return parsers.timestampFromMicroseconds(view.getBigInt64(0, true)) + if (type === 'INT64' && logical_type?.type === 'TIMESTAMP' && logical_type?.unit === 'NANOS') return parsers.timestampFromNanoseconds(view.getBigInt64(0, true)) + if (type === 'INT64' && logical_type?.type === 'TIMESTAMP' && logical_type?.unit === 'MICROS') return parsers.timestampFromMicroseconds(view.getBigInt64(0, true)) + if (type === 'INT64' && logical_type?.type === 'TIMESTAMP') return parsers.timestampFromMilliseconds(view.getBigInt64(0, true)) + if (type === 'INT32' && view.byteLength === 4) return view.getInt32(0, true) + if (type === 'INT64' && view.byteLength === 8) return view.getBigInt64(0, true) + if (converted_type === 'DECIMAL') return parseDecimal(value) * 10 ** -(schema.scale || 0) + if (logical_type?.type === 'FLOAT16') return parseFloat16(value) + if (logical_type?.type === 'UUID') return parsers.uuidFromBytes(value) + if (type === 'FIXED_LEN_BYTE_ARRAY') return value + // assert(false) + return value +} diff --git a/web/node_modules/hyparquet/src/node.js b/web/node_modules/hyparquet/src/node.js new file mode 100644 index 0000000000000000000000000000000000000000..416140fe4ca8b932f3bef52f373291ae260983f1 --- /dev/null +++ b/web/node_modules/hyparquet/src/node.js @@ -0,0 +1,34 @@ +/** + * @import {AsyncBuffer} from '../src/types.js' + */ + +import { createReadStream, promises as fs } from 'fs' + +export * from './index.js' + +/** + * Construct an AsyncBuffer for a local file using node fs package. + * + * @param {string} filename + * @returns {Promise} + */ +export async function asyncBufferFromFile(filename) { + const { size } = await fs.stat(filename) + return { + byteLength: size, + slice(start, end) { + // read file slice + const reader = createReadStream(filename, { start, end }) + return new Promise((resolve, reject) => { + /** @type {any[]} */ + const chunks = [] + reader.on('data', chunk => chunks.push(chunk)) + reader.on('error', reject) + reader.on('end', () => { + const buffer = Buffer.concat(chunks) + resolve(buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength)) + }) + }) + }, + } +} diff --git a/web/node_modules/hyparquet/src/plain.js b/web/node_modules/hyparquet/src/plain.js new file mode 100644 index 0000000000000000000000000000000000000000..7a513d461704772bbde225525ccf4b66aa8b486e --- /dev/null +++ b/web/node_modules/hyparquet/src/plain.js @@ -0,0 +1,180 @@ +/** + * Read `count` values of the given type from the reader.view. + * + * @param {DataReader} reader - buffer to read data from + * @param {ParquetType} type - parquet type of the data + * @param {number} count - number of values to read + * @param {number | undefined} fixedLength - length of each fixed length byte array + * @returns {DecodedArray} array of values + */ +export function readPlain(reader, type, count, fixedLength) { + if (count === 0) return [] + if (type === 'BOOLEAN') { + return readPlainBoolean(reader, count) + } else if (type === 'INT32') { + return readPlainInt32(reader, count) + } else if (type === 'INT64') { + return readPlainInt64(reader, count) + } else if (type === 'INT96') { + return readPlainInt96(reader, count) + } else if (type === 'FLOAT') { + return readPlainFloat(reader, count) + } else if (type === 'DOUBLE') { + return readPlainDouble(reader, count) + } else if (type === 'BYTE_ARRAY') { + return readPlainByteArray(reader, count) + } else if (type === 'FIXED_LEN_BYTE_ARRAY') { + if (!fixedLength) throw new Error('parquet missing fixed length') + return readPlainByteArrayFixed(reader, count, fixedLength) + } else { + throw new Error(`parquet unhandled type: ${type}`) + } +} + +/** + * Read `count` boolean values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {boolean[]} + */ +function readPlainBoolean(reader, count) { + const values = new Array(count) + for (let i = 0; i < count; i++) { + const byteOffset = reader.offset + (i / 8 | 0) + const bitOffset = i % 8 + const byte = reader.view.getUint8(byteOffset) + values[i] = (byte & 1 << bitOffset) !== 0 + } + reader.offset += Math.ceil(count / 8) + return values +} + +/** + * Read `count` int32 values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {Int32Array} + */ +function readPlainInt32(reader, count) { + const values = (reader.view.byteOffset + reader.offset) % 4 + ? new Int32Array(align(reader.view.buffer, reader.view.byteOffset + reader.offset, count * 4)) + : new Int32Array(reader.view.buffer, reader.view.byteOffset + reader.offset, count) + reader.offset += count * 4 + return values +} + +/** + * Read `count` int64 values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {BigInt64Array} + */ +function readPlainInt64(reader, count) { + const values = (reader.view.byteOffset + reader.offset) % 8 + ? new BigInt64Array(align(reader.view.buffer, reader.view.byteOffset + reader.offset, count * 8)) + : new BigInt64Array(reader.view.buffer, reader.view.byteOffset + reader.offset, count) + reader.offset += count * 8 + return values +} + +/** + * Read `count` int96 values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {bigint[]} + */ +function readPlainInt96(reader, count) { + const values = new Array(count) + for (let i = 0; i < count; i++) { + const low = reader.view.getBigInt64(reader.offset + i * 12, true) + const high = reader.view.getInt32(reader.offset + i * 12 + 8, true) + values[i] = BigInt(high) << 64n | low + } + reader.offset += count * 12 + return values +} + +/** + * Read `count` float values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {Float32Array} + */ +function readPlainFloat(reader, count) { + const values = (reader.view.byteOffset + reader.offset) % 4 + ? new Float32Array(align(reader.view.buffer, reader.view.byteOffset + reader.offset, count * 4)) + : new Float32Array(reader.view.buffer, reader.view.byteOffset + reader.offset, count) + reader.offset += count * 4 + return values +} + +/** + * Read `count` double values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {Float64Array} + */ +function readPlainDouble(reader, count) { + const values = (reader.view.byteOffset + reader.offset) % 8 + ? new Float64Array(align(reader.view.buffer, reader.view.byteOffset + reader.offset, count * 8)) + : new Float64Array(reader.view.buffer, reader.view.byteOffset + reader.offset, count) + reader.offset += count * 8 + return values +} + +/** + * Read `count` byte array values. + * + * @param {DataReader} reader + * @param {number} count + * @returns {Uint8Array[]} + */ +function readPlainByteArray(reader, count) { + const values = new Array(count) + for (let i = 0; i < count; i++) { + const length = reader.view.getUint32(reader.offset, true) + reader.offset += 4 + values[i] = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, length) + reader.offset += length + } + return values +} + +/** + * Read a fixed length byte array. + * + * @param {DataReader} reader + * @param {number} count + * @param {number} fixedLength + * @returns {Uint8Array[]} + */ +function readPlainByteArrayFixed(reader, count, fixedLength) { + // assert(reader.view.byteLength - reader.offset >= count * fixedLength) + const values = new Array(count) + for (let i = 0; i < count; i++) { + values[i] = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, fixedLength) + reader.offset += fixedLength + } + return values +} + +/** + * Create a new buffer with the offset and size. + * + * @import {DataReader, DecodedArray, ParquetType} from '../src/types.js' + * @param {ArrayBufferLike} buffer + * @param {number} offset + * @param {number} size + * @returns {ArrayBuffer} + */ +function align(buffer, offset, size) { + const aligned = new ArrayBuffer(size) + new Uint8Array(aligned).set(new Uint8Array(buffer, offset, size)) + return aligned +} diff --git a/web/node_modules/hyparquet/src/plan.js b/web/node_modules/hyparquet/src/plan.js new file mode 100644 index 0000000000000000000000000000000000000000..e243dafcd148f03d90e6a82deaa4e8f3a61739a0 --- /dev/null +++ b/web/node_modules/hyparquet/src/plan.js @@ -0,0 +1,189 @@ +import { bloomEligibleColumns, readBloomFilter } from './bloom.js' +import { canSkipRowGroup } from './filter.js' +import { parquetSchema } from './metadata.js' +import { getPhysicalColumns } from './schema.js' + +/** + * @import {AsyncBuffer, BloomFilter, ByteRange, ChunkPlan, FileMetaData, GroupPlan, ParquetQueryFilter, ParquetReadOptions, QueryPlan, SchemaElement} from '../src/types.js' + */ + +// Combine column chunks if less than 2mb +const runLimit = 1 << 21 // 2mb + +/** + * Plan which byte ranges to read to satisfy a read request. + * Metadata must be non-null. + * + * @param {ParquetReadOptions & { bloomFiltersByGroup?: Record[], schemaElements?: Record }} options + * @returns {QueryPlan} + */ +export function parquetPlan({ metadata, rowStart = 0, rowEnd = Infinity, columns, filter, filterStrict = true, useOffsetIndex = false, bloomFiltersByGroup, schemaElements }) { + if (!metadata) throw new Error('parquetPlan requires metadata') + /** @type {GroupPlan[]} */ + const groups = [] + /** @type {ByteRange[]} */ + const fetches = [] + /** @type {ByteRange[]} */ + const indexes = [] + const physicalColumns = getPhysicalColumns(parquetSchema(metadata)) + + // find which row groups to read + let groupStart = 0 // first row index of the current group + let rgIdx = 0 + for (const rowGroup of metadata.row_groups) { + const groupRows = Number(rowGroup.num_rows) + const groupEnd = groupStart + groupRows + const bloomFilters = bloomFiltersByGroup?.[rgIdx] + // if row group overlaps with row range, add it to the plan + if (groupRows > 0 && groupEnd > rowStart && groupStart < rowEnd && !canSkipRowGroup({ rowGroup, physicalColumns, filter, strict: filterStrict, bloomFilters, schemaElements })) { + /** @type {ChunkPlan[]} */ + const chunks = [] + let groupStartByte = Infinity + let groupEndByte = -Infinity + // loop through each column chunk + for (const chunk of rowGroup.columns) { + const meta = chunk.meta_data + if (chunk.file_path) throw new Error('parquet file_path not supported') + if (!meta) throw new Error('parquet column metadata is undefined') + // add included column chunks to the plan + if (!columns || columns.includes(meta.path_in_schema[0])) { + // full column chunk + const columnOffset = meta.dictionary_page_offset || meta.data_page_offset + const startByte = Number(columnOffset) + const endByte = Number(columnOffset + meta.total_compressed_size) + // update group byte range + if (startByte < groupStartByte) groupStartByte = startByte + if (endByte > groupEndByte) groupEndByte = endByte + + if (useOffsetIndex && chunk.offset_index_offset && chunk.offset_index_length && (rowStart > groupStart || rowEnd < groupEnd)) { + const offsetIndexStart = Number(chunk.offset_index_offset) + chunks.push({ + columnMetadata: meta, + offsetIndex: { + startByte: offsetIndexStart, + endByte: offsetIndexStart + chunk.offset_index_length, + }, + range: { startByte, endByte }, + }) + } else { + chunks.push({ + columnMetadata: meta, + range: { startByte, endByte }, + }) + } + + } + } + const selectStart = Math.max(rowStart - groupStart, 0) + const selectEnd = Math.min(rowEnd - groupStart, groupRows) + groups.push({ chunks, rowGroup, groupStart, groupRows, selectStart, selectEnd }) + + // combine runs of column chunks + /** @type {ByteRange | undefined} */ + let run + for (const chunk of chunks) { + if ('offsetIndex' in chunk) { + indexes.push(chunk.offsetIndex) + } else { + const { range } = chunk + if (columns) { + fetches.push(range) + } else if (run && range.endByte - run.startByte <= runLimit) { + // extend range + run.endByte = range.endByte + } else { + // new range + if (run) fetches.push(run) + run = { ...range } + } + } + } + if (run) fetches.push(run) + } + + groupStart = groupEnd + rgIdx++ + } + if (!isFinite(rowEnd)) rowEnd = groupStart + fetches.push(...indexes) + + return { metadata, rowStart, rowEnd, columns, fetches, groups } +} + +/** + * Fetch bloom filters for $eq / $in columns of row groups not already provably + * skippable by statistics alone. Returns an array indexed by row-group ordinal; + * each entry maps top-level column name → BloomFilter for any chunk whose + * bloom filter we were able to parse. Adds one round-trip when at least one + * bloom filter is fetched; otherwise returns synchronously. + * + * @param {object} options + * @param {AsyncBuffer} options.file + * @param {FileMetaData} options.metadata + * @param {ParquetQueryFilter} options.filter + * @param {boolean} [options.filterStrict] + * @returns {Promise[]>} + */ +export async function prefetchBloomFilters({ file, metadata, filter, filterStrict = true }) { + const result = metadata.row_groups.map(() => /** @type {Record} */ ({})) + const eligibleCols = bloomEligibleColumns(filter) + if (eligibleCols.size === 0) return result + const physicalColumns = getPhysicalColumns(parquetSchema(metadata)) + + /** @type {Promise[]} */ + const tasks = [] + metadata.row_groups.forEach((rowGroup, rgIdx) => { + if (canSkipRowGroup({ rowGroup, physicalColumns, filter, strict: filterStrict })) return + for (const colName of eligibleCols) { + const columnIdx = physicalColumns.indexOf(colName) + if (columnIdx === -1) continue + const meta = rowGroup.columns[columnIdx]?.meta_data + if (!meta?.bloom_filter_offset || !meta.bloom_filter_length) continue + const start = Number(meta.bloom_filter_offset) + const end = start + meta.bloom_filter_length + tasks.push((async () => { + const buffer = await file.slice(start, end) + const bloom = readBloomFilter({ view: new DataView(buffer), offset: 0 }) + if (bloom) result[rgIdx][colName] = bloom + })()) + } + }) + + if (tasks.length) await Promise.all(tasks) + return result +} + +/** + * Prefetch byte ranges from an AsyncBuffer. + * + * @param {AsyncBuffer} file + * @param {QueryPlan} plan + * @returns {AsyncBuffer} + */ +export function prefetchAsyncBuffer(file, { fetches }) { + // fetch byte ranges from the file + const promises = fetches.map(({ startByte, endByte }) => file.slice(startByte, endByte)) + return { + byteLength: file.byteLength, + slice(start, end = file.byteLength) { + // find matching slice + const index = fetches.findIndex(({ startByte, endByte }) => startByte <= start && end <= endByte) + if (index < 0) { + // fallback to direct read + return file.slice(start, end) + } + if (fetches[index].startByte !== start || fetches[index].endByte !== end) { + // slice a subrange of the prefetch + const startOffset = start - fetches[index].startByte + const endOffset = end - fetches[index].startByte + if (promises[index] instanceof Promise) { + return promises[index].then(buffer => buffer.slice(startOffset, endOffset)) + } else { + return promises[index].slice(startOffset, endOffset) + } + } else { + return promises[index] + } + }, + } +} diff --git a/web/node_modules/hyparquet/src/query.js b/web/node_modules/hyparquet/src/query.js new file mode 100644 index 0000000000000000000000000000000000000000..85ade5771edfe9066ae82333e43f97fe20aa6b91 --- /dev/null +++ b/web/node_modules/hyparquet/src/query.js @@ -0,0 +1,163 @@ +/** + * @import {BaseParquetReadOptions} from '../src/types.js' + */ + +import { parquetMetadataAsync, parquetSchema } from './metadata.js' +import { parquetReadColumn, parquetReadObjects } from './read.js' + +/** + * Wraps parquetRead with orderBy support. + * This is a parquet-aware query engine that can read a subset of rows and columns. + * Accepts optional orderBy column name to sort the results. + * Note that using orderBy may SIGNIFICANTLY increase the query time. + * + * @param {BaseParquetReadOptions & { orderBy?: string }} options + * @returns {Promise[]>} resolves when all requested rows and columns are parsed + */ +export async function parquetQuery(options) { + if (!options.file || !(options.file.byteLength >= 0)) { + throw new Error('parquet expected AsyncBuffer') + } + options.metadata ??= await parquetMetadataAsync(options.file, options) + + const { metadata, rowStart = 0, columns, orderBy, filter } = options + if (rowStart < 0) throw new Error('parquet rowStart must be positive') + const rowEnd = options.rowEnd ?? Number(metadata.num_rows) + + // Validate orderBy column exists + if (orderBy) { + const allColumns = parquetSchema(options.metadata).children.map(c => c.element.name) + if (!allColumns.includes(orderBy)) { + throw new Error(`parquet orderBy column not found: ${orderBy}`) + } + } + + if (filter && !orderBy && rowEnd < metadata.num_rows) { + // iterate through row groups and filter until we have enough rows + /** @type {Record[]} */ + const filteredRows = [] + let groupStart = 0 + for (const group of metadata.row_groups) { + const groupEnd = groupStart + Number(group.num_rows) + // TODO: if expected > group size, start fetching next groups + const groupData = await parquetReadObjects({ + ...options, rowStart: groupStart, rowEnd: groupEnd, + }) + filteredRows.push(...groupData) + if (filteredRows.length >= rowEnd) break + groupStart = groupEnd + } + return filteredRows.slice(rowStart, rowEnd) + } else if (filter && orderBy) { + // read all rows with orderBy column included for sorting + const readColumns = columns && !columns.includes(orderBy) + ? [...columns, orderBy] + : columns + + const results = await parquetReadObjects({ + ...options, rowStart: undefined, rowEnd: undefined, columns: readColumns, + }) + + // sort by orderBy column + results.sort((a, b) => compare(a[orderBy], b[orderBy])) + + // project out orderBy column if not originally requested + if (readColumns !== columns) { + for (const row of results) { + delete row[orderBy] + } + } + + return results.slice(rowStart, rowEnd) + } else if (filter) { + // filter without orderBy, read all matching rows + const results = await parquetReadObjects({ + ...options, rowStart: undefined, rowEnd: undefined, + }) + return results.slice(rowStart, rowEnd) + } else if (typeof orderBy === 'string') { + // sorted but unfiltered: fetch orderBy column first + const orderColumn = await parquetReadColumn({ + ...options, rowStart: undefined, rowEnd: undefined, columns: [orderBy], + }) + + // compute row groups to fetch + const sortedIndices = Array.from(orderColumn, (_, index) => index) + .sort((a, b) => compare(orderColumn[a], orderColumn[b])) + .slice(rowStart, rowEnd) + + const sparseData = await parquetReadRows({ ...options, rows: sortedIndices }) + // warning: the type Record & {__index__: number})[] is simplified into Record[] + // when returning. The data contains the __index__ property, but it's not exposed as such. + const data = sortedIndices.map(index => sparseData[index]) + return data + } else { + return await parquetReadObjects(options) + } +} + +/** + * Reads a list rows from a parquet file, reading only the row groups that contain the rows. + * Returns a sparse array of rows. + * @param {BaseParquetReadOptions & { rows: number[] }} options + * @returns {Promise<(Record & {__index__: number})[]>} + */ +async function parquetReadRows(options) { + const { file, rows } = options + options.metadata ??= await parquetMetadataAsync(file, options) + const { row_groups: rowGroups } = options.metadata + // Compute row groups to fetch + const groupIncluded = Array(rowGroups.length).fill(false) + let groupStart = 0 + const groupEnds = rowGroups.map(group => groupStart += Number(group.num_rows)) + for (const index of rows) { + const groupIndex = groupEnds.findIndex(end => index < end) + groupIncluded[groupIndex] = true + } + + // Compute row ranges to fetch + const rowRanges = [] + let rangeStart + groupStart = 0 + for (let i = 0; i < groupIncluded.length; i++) { + const groupEnd = groupStart + Number(rowGroups[i].num_rows) + if (groupIncluded[i]) { + if (rangeStart === undefined) { + rangeStart = groupStart + } + } else { + if (rangeStart !== undefined) { + rowRanges.push([rangeStart, groupEnd]) + rangeStart = undefined + } + } + groupStart = groupEnd + } + if (rangeStart !== undefined) { + rowRanges.push([rangeStart, groupStart]) + } + + // Fetch by row group and map to rows + /** @type {(Record & {__index__: number})[]} */ + const sparseData = Array(Number(options.metadata.num_rows)) + for (const [rangeStart, rangeEnd] of rowRanges) { + // TODO: fetch in parallel + const groupData = await parquetReadObjects({ ...options, rowStart: rangeStart, rowEnd: rangeEnd }) + for (let i = rangeStart; i < rangeEnd; i++) { + // warning: if the row contains a column named __index__, it will overwrite the index. + sparseData[i] = { __index__: i, ...groupData[i - rangeStart] } + } + } + return sparseData +} + +/** + * @param {any} a + * @param {any} b + * @returns {number} + */ +function compare(a, b) { + if (a < b) return -1 + if (a > b) return 1 + return 0 // TODO: null handling +} diff --git a/web/node_modules/hyparquet/src/read.js b/web/node_modules/hyparquet/src/read.js new file mode 100644 index 0000000000000000000000000000000000000000..8987cad01ad8efa8bb3963cfde79fdc5db412417 --- /dev/null +++ b/web/node_modules/hyparquet/src/read.js @@ -0,0 +1,230 @@ +/** + * @import {AsyncRowGroup, BaseParquetReadOptions, DecodedArray, ParquetReadOptions, SchemaElement} from '../src/types.js' + */ + +import { columnsNeededForFilter, matchFilter } from './filter.js' +import { parquetMetadataAsync, parquetSchema } from './metadata.js' +import { parquetPlan, prefetchAsyncBuffer, prefetchBloomFilters } from './plan.js' +import { assembleAsync, asyncGroupToRows, readRowGroup } from './rowgroup.js' +import { concat } from './utils.js' + +/** + * Read parquet data rows from a file-like object. + * Reads the minimal number of row groups and columns to satisfy the request. + * + * Returns a void promise when complete. + * Errors are thrown on the returned promise. + * Data is returned in callbacks onComplete, onChunk, onPage, NOT the return promise. + * See parquetReadObjects for a more convenient API. + * + * @param {ParquetReadOptions} options read options + * @returns {Promise} resolves when all requested rows and columns are parsed, all errors are thrown here + */ +export async function parquetRead(options) { + // load metadata if not provided + options.metadata ??= await parquetMetadataAsync(options.file, options) + + const { rowStart = 0, rowEnd, columns, onChunk, onComplete, rowFormat, filter, filterStrict = true } = options + + // Filter requires object format to match column names + if (filter && rowFormat !== 'object') { + throw new Error('parquet filter requires rowFormat: "object"') + } + + // Include filter columns in the read plan + const filterColumns = columnsNeededForFilter(filter) + if (filterColumns.length) { + const schemaColumns = parquetSchema(options.metadata).children.map(c => c.element.name) + const missingColumns = filterColumns.filter(c => !schemaColumns.includes(c)) + if (missingColumns.length) { + throw new Error(`parquet filter columns not found: ${missingColumns.join(', ')}`) + } + } + let readColumns = columns + let requiresProjection = false + if (columns && filter) { + const missingFilterColumns = filterColumns.filter(c => !columns.includes(c)) + if (missingFilterColumns.length) { + readColumns = [...columns, ...missingFilterColumns] + requiresProjection = true + } + } + + // read row groups with expanded columns + let readOptions = readColumns !== columns ? { ...options, columns: readColumns } : options + readOptions = await withBloomFilters(readOptions) + const asyncGroups = parquetReadAsync(readOptions) + + // skip assembly if no onComplete or onChunk, but wait for reading to finish + if (!onComplete && !onChunk) { + await awaitAllColumns(asyncGroups) + return + } + + // assemble struct columns + const schemaTree = parquetSchema(options.metadata) + const assembled = asyncGroups.map(arg => assembleAsync(arg, schemaTree, options.parsers)) + + // onChunk emit all chunks (don't await). Rejection is surfaced by awaitAllColumns below. + if (onChunk) { + for (const asyncGroup of assembled) { + for (const asyncColumn of asyncGroup.asyncColumns) { + asyncColumn.data.then(({ data, skipped }) => { + let rowStart = asyncGroup.groupStart + skipped + for (const columnData of data) { + onChunk({ + columnName: asyncColumn.pathInSchema[0], + columnData, + rowStart, + rowEnd: rowStart + columnData.length, + }) + rowStart += columnData.length + } + }, () => {}) + } + } + } + + // onComplete transpose column chunks to rows + if (onComplete) { + // wait for all reads to settle so a sibling rejection cannot leak + await awaitAllColumns(assembled) + // loosen the types to avoid duplicate code + /** @type {any[]} */ + const rows = [] + for (const asyncGroup of assembled) { + // filter to rows in range + const selectStart = Math.max(rowStart - asyncGroup.groupStart, 0) + const selectEnd = Math.min((rowEnd ?? Infinity) - asyncGroup.groupStart, asyncGroup.groupRows) + // transpose column chunks to rows in output + const groupData = rowFormat === 'object' ? + await asyncGroupToRows(asyncGroup, selectStart, selectEnd, readColumns, 'object') : + await asyncGroupToRows(asyncGroup, selectStart, selectEnd, columns, 'array') + + // Apply filter and projection + if (filter) { + // eslint-disable-next-line no-extra-parens + for (const row of /** @type {Record[]} */ (groupData)) { + if (matchFilter(row, filter, filterStrict)) { + if (requiresProjection && columns) { + for (const col of filterColumns) { + if (!columns.includes(col)) delete row[col] + } + } + rows.push(row) + } + } + } else { + concat(rows, groupData) + } + } + onComplete(rows) + } else { + // wait for all async groups to finish (complete takes care of this) + await awaitAllColumns(assembled) + } +} + +/** + * Await every column promise across the given row groups via Promise.allSettled + * so no rejection escapes as an unhandledRejection. Throws the first rejection. + * + * @param {AsyncRowGroup[]} asyncGroups + * @returns {Promise} + */ +async function awaitAllColumns(asyncGroups) { + const all = asyncGroups.flatMap(g => g.asyncColumns.map(c => c.data)) + const results = await Promise.allSettled(all) + const failed = results.find(r => r.status === 'rejected') + if (failed) throw failed.reason +} + +/** + * @param {ParquetReadOptions} options read options + * @returns {AsyncRowGroup[]} + */ +export function parquetReadAsync(options) { + if (!options.metadata) throw new Error('parquet requires metadata') + // TODO: validate options (start, end, columns, etc) + + // prefetch byte ranges + const plan = parquetPlan(options) + options.file = prefetchAsyncBuffer(options.file, plan) + + // read row groups + return plan.groups.map(groupPlan => readRowGroup(options, plan, groupPlan)) +} + +/** + * Reads a single column from a parquet file. + * + * @param {BaseParquetReadOptions} options + * @returns {Promise} + */ +export async function parquetReadColumn(options) { + if (options.columns?.length !== 1) { + throw new Error('parquetReadColumn expected columns: [columnName]') + } + options.metadata ??= await parquetMetadataAsync(options.file, options) + const asyncGroups = parquetReadAsync(await withBloomFilters(options)) + + // assemble struct columns + const schemaTree = parquetSchema(options.metadata) + const assembled = asyncGroups.map(arg => assembleAsync(arg, schemaTree, options.parsers)) + + // wait for all reads to settle so a sibling rejection cannot leak + await awaitAllColumns(assembled) + + /** @type {DecodedArray} */ + const columnData = [] + for (const rg of assembled) { + const { data } = await rg.asyncColumns[0].data + for (const chunk of data) { + concat(columnData, chunk) + } + } + return columnData +} + +/** + * Conditionally fetch bloom filters and attach them (and the per-column schema + * elements they require) to options so parquetPlan can use them for row-group + * pruning. Returns options unchanged when there's no filter or the user has + * disabled bloom pushdown. + * + * @param {BaseParquetReadOptions} options + * @returns {Promise} + */ +async function withBloomFilters(options) { + if (!options.useBloomFilters) return options + if (!options.filter || !options.metadata) return options + const schemaTree = parquetSchema(options.metadata) + /** @type {Record} */ + const schemaElements = {} + for (const child of schemaTree.children) schemaElements[child.element.name] = child.element + const bloomFiltersByGroup = await prefetchBloomFilters({ + file: options.file, + metadata: options.metadata, + filter: options.filter, + filterStrict: options.filterStrict, + }) + // eslint-disable-next-line no-extra-parens + return /** @type {BaseParquetReadOptions} */ ({ ...options, bloomFiltersByGroup, schemaElements }) +} + +/** + * This is a helper function to read parquet row data as a promise. + * It is a wrapper around the more configurable parquetRead function. + * + * @param {Omit} options + * @returns {Promise[]>} resolves when all requested rows and columns are parsed + */ +export function parquetReadObjects(options) { + return new Promise((onComplete, reject) => { + parquetRead({ + ...options, + rowFormat: 'object', // force object output + onComplete, + }).catch(reject) + }) +} diff --git a/web/node_modules/hyparquet/src/rowgroup.js b/web/node_modules/hyparquet/src/rowgroup.js new file mode 100644 index 0000000000000000000000000000000000000000..b2239529384febda9e628aa8d9d732c4de02cd87 --- /dev/null +++ b/web/node_modules/hyparquet/src/rowgroup.js @@ -0,0 +1,225 @@ +/** + * @import {AsyncColumn, AsyncRowGroup, DecodedArray, GroupPlan, ParquetParsers, ParquetReadOptions, QueryPlan, SchemaTree} from '../src/types.js' + */ + +import { assembleNested } from './assemble.js' +import { readColumn } from './column.js' +import { DEFAULT_PARSERS } from './convert.js' +import { readOffsetIndex } from './indexes.js' +import { getSchemaPath } from './schema.js' +import { flatten } from './utils.js' + +/** + * Read a row group from a file-like object. + * + * @param {ParquetReadOptions} options + * @param {QueryPlan} plan + * @param {GroupPlan} groupPlan + * @returns {AsyncRowGroup} resolves to column data + */ +export function readRowGroup(options, { metadata }, groupPlan) { + /** @type {AsyncColumn[]} */ + const asyncColumns = [] + + // read column data + for (const chunk of groupPlan.chunks) { + const { data_page_offset, dictionary_page_offset, path_in_schema: pathInSchema } = chunk.columnMetadata + const schemaPath = getSchemaPath(metadata.schema, pathInSchema) + const columnDecoder = { + pathInSchema, + element: schemaPath[schemaPath.length - 1].element, + schemaPath, + parsers: { ...DEFAULT_PARSERS, ...options.parsers }, + ...options, + ...chunk.columnMetadata, + } + let { startByte, endByte } = chunk.range + + // non-offset-index case + if (!('offsetIndex' in chunk)) { + asyncColumns.push({ + pathInSchema, + data: Promise.resolve(options.file.slice(startByte, endByte)) + .then(buffer => { + const reader = { view: new DataView(buffer), offset: 0 } + return readColumn(reader, groupPlan, columnDecoder, options.onPage) + }), + }) + continue + } + + // offset-index case + asyncColumns.push({ + pathInSchema, + // fetch offset index + data: Promise.resolve(options.file.slice(chunk.offsetIndex.startByte, chunk.offsetIndex.endByte)) + .then(async arrayBuffer => { + // use offset index to read only necessary pages + const { selectStart, selectEnd } = groupPlan + const pages = readOffsetIndex({ view: new DataView(arrayBuffer), offset: 0 }).page_locations + let skipped = -1 + // include dictionary if present, handle polars missing dictionary_page_offset + const hasDict = dictionary_page_offset || data_page_offset < pages[0].offset + for (let i = 0; i < pages.length; i++) { + const page = pages[i] + const pageStart = Number(page.first_row_index) + const pageEnd = i + 1 < pages.length + ? Number(pages[i + 1].first_row_index) + : groupPlan.groupRows // last page extends to end of row group + // check if page overlaps with [selectStart, selectEnd) + if (skipped < 0 && !hasDict && pageEnd > selectStart) { + startByte = Number(page.offset) + skipped = pageStart + } + if (pageStart < selectEnd) { + endByte = Number(page.offset) + page.compressed_page_size + } + } + if (skipped < 0) skipped = 0 + const buffer = await options.file.slice(startByte, endByte) + const reader = { view: new DataView(buffer), offset: 0 } + // adjust row selection for skipped pages + const adjustedGroupPlan = skipped ? { + ...groupPlan, + groupStart: groupPlan.groupStart + skipped, + selectStart: groupPlan.selectStart - skipped, + selectEnd: groupPlan.selectEnd - skipped, + } : groupPlan + const { data, skipped: columnSkipped } = readColumn(reader, adjustedGroupPlan, columnDecoder, options.onPage) + return { + data, + skipped: skipped + columnSkipped, + } + }), + }) + } + + return { groupStart: groupPlan.groupStart, groupRows: groupPlan.groupRows, asyncColumns } +} + +/** + * @overload + * @param {AsyncRowGroup} asyncGroup + * @param {number} selectStart + * @param {number} selectEnd + * @param {string[] | undefined} columns + * @param {'object'} rowFormat + * @returns {Promise[]>} resolves to row data + */ +/** + * @overload + * @param {AsyncRowGroup} asyncGroup + * @param {number} selectStart + * @param {number} selectEnd + * @param {string[] | undefined} columns + * @param {'array'} [rowFormat] + * @returns {Promise} resolves to row data + */ +/** + * @param {AsyncRowGroup} asyncGroup + * @param {number} selectStart + * @param {number} selectEnd + * @param {string[] | undefined} columns + * @param {'object' | 'array'} [rowFormat] + * @returns {Promise[] | any[][]>} resolves to row data + */ +export async function asyncGroupToRows({ asyncColumns }, selectStart, selectEnd, columns, rowFormat) { + // TODO: do it without flatten + const asyncPages = await Promise.all(asyncColumns.map(column => + column.data.then(({ skipped, data }) => ({ skipped, data: flatten(data) })) + )) + + // transpose columns into rows + const selectCount = selectEnd - selectStart + if (rowFormat === 'object') { + /** @type {Record[]} */ + const groupData = Array(selectCount) + for (let selectRow = 0; selectRow < selectCount; selectRow++) { + // return each row as an object + /** @type {Record} */ + const rowData = {} + for (let i = 0; i < asyncColumns.length; i++) { + const { data, skipped } = asyncPages[i] + rowData[asyncColumns[i].pathInSchema[0]] = data[selectStart + selectRow - skipped] + } + groupData[selectRow] = rowData + } + return groupData + } + + // careful mapping of column order for rowFormat: array + const includedColumnNames = asyncColumns + .map(child => child.pathInSchema[0]) + .filter(name => !columns || columns.includes(name)) + const columnOrder = columns ?? includedColumnNames + const columnIndexes = columnOrder.map(name => asyncColumns.findIndex(column => column.pathInSchema[0] === name)) + + /** @type {any[][]} */ + const groupData = Array(selectCount) + for (let selectRow = 0; selectRow < selectCount; selectRow++) { + // return each row as an array + const rowData = Array(asyncColumns.length) + for (let i = 0; i < columnOrder.length; i++) { + const colIdx = columnIndexes[i] + if (colIdx < 0) throw new Error(`parquet column not found: ${columnOrder[i]}`) + const { data, skipped } = asyncPages[colIdx] + rowData[i] = data[selectStart + selectRow - skipped] + } + groupData[selectRow] = rowData + } + return groupData +} + +/** + * Assemble physical columns into top-level columns asynchronously. + * + * @param {AsyncRowGroup} asyncRowGroup + * @param {SchemaTree} schemaTree + * @param {ParquetParsers} [parsers] + * @returns {AsyncRowGroup} + */ +export function assembleAsync(asyncRowGroup, schemaTree, parsers) { + const { asyncColumns } = asyncRowGroup + parsers = { ...DEFAULT_PARSERS, ...parsers } + /** @type {AsyncColumn[]} */ + const assembled = [] + for (const child of schemaTree.children) { + if (child.children.length) { + const childColumns = asyncColumns.filter(column => column.pathInSchema[0] === child.element.name) + if (!childColumns.length) continue + + assembled.push({ + pathInSchema: child.path, + data: (async () => { + // collect subcolumn data — Promise.all observes every rejection so + // a sibling failure cannot leak as an unhandledRejection + const resolved = await Promise.all(childColumns.map(c => c.data)) + /** @type {Map} */ + const subcolumnData = new Map() + let minLength = Infinity + for (let i = 0; i < childColumns.length; i++) { + const flat = flatten(resolved[i].data) + subcolumnData.set(childColumns[i].pathInSchema.join('.'), flat) + minLength = Math.min(minLength, flat.length) + } + // trim sub-columns to same length (offset index may read different pages per column) + for (const [key, value] of subcolumnData) { + if (value.length > minLength) { + subcolumnData.set(key, value.slice(0, minLength)) + } + } + // assemble the column + assembleNested(subcolumnData, child, parsers) + const assembled = subcolumnData.get(child.element.name) + if (!assembled) throw new Error('parquet column data not assembled') + return { data: [assembled], skipped: 0 } + })(), + }) + } else { + // leaf node, return the column + const asyncColumn = asyncColumns.find(column => column.pathInSchema[0] === child.element.name) + if (asyncColumn) assembled.push(asyncColumn) + } + } + return { ...asyncRowGroup, asyncColumns: assembled } +} diff --git a/web/node_modules/hyparquet/src/schema.js b/web/node_modules/hyparquet/src/schema.js new file mode 100644 index 0000000000000000000000000000000000000000..1dd9ae87307f5b04d89248d8ab84dc078d24bf29 --- /dev/null +++ b/web/node_modules/hyparquet/src/schema.js @@ -0,0 +1,159 @@ +/** + * Build a tree from the schema elements. + * + * @param {SchemaElement[]} schema + * @param {number} rootIndex index of the root element + * @param {string[]} path path to the element + * @returns {SchemaTree} tree of schema elements + */ +function schemaTree(schema, rootIndex, path) { + const element = schema[rootIndex] + const children = [] + let count = 1 + + // Read the specified number of children + if (element.num_children) { + while (children.length < element.num_children) { + const childElement = schema[rootIndex + count] + const child = schemaTree(schema, rootIndex + count, [...path, childElement.name]) + count += child.count + children.push(child) + } + } + + return { count, element, children, path } +} + +/** + * Get schema elements from the root to the given element name. + * + * @param {SchemaElement[]} schema + * @param {string[]} name path to the element + * @returns {SchemaTree[]} list of schema elements + */ +export function getSchemaPath(schema, name) { + let tree = schemaTree(schema, 0, []) + const path = [tree] + for (const part of name) { + const child = tree.children.find(child => child.element.name === part) + if (!child) throw new Error(`parquet schema element not found: ${name}`) + path.push(child) + tree = child + } + return path +} + +/** + * Get all physical (leaf) column names. + * + * @param {SchemaTree} schemaTree + * @returns {string[]} list of physical column names + */ +export function getPhysicalColumns(schemaTree) { + /** @type {string[]} */ + const columns = [] + /** @param {SchemaTree} node */ + function traverse(node) { + if (node.children.length) { + for (const child of node.children) { + traverse(child) + } + } else { + columns.push(node.path.join('.')) + } + } + traverse(schemaTree) + return columns +} + +/** + * Get the max repetition level for a given schema path. + * + * @param {SchemaTree[]} schemaPath + * @returns {number} max repetition level + */ +export function getMaxRepetitionLevel(schemaPath) { + let maxLevel = 0 + for (const { element } of schemaPath) { + if (element.repetition_type === 'REPEATED') { + maxLevel++ + } + } + return maxLevel +} + +/** + * Get the max definition level for a given schema path. + * + * @param {SchemaTree[]} schemaPath + * @returns {number} max definition level + */ +export function getMaxDefinitionLevel(schemaPath) { + let maxLevel = 0 + for (const { element } of schemaPath.slice(1)) { + if (element.repetition_type !== 'REQUIRED') { + maxLevel++ + } + } + return maxLevel +} + +/** + * Check if a column is list-like. + * + * @param {SchemaTree} schema + * @returns {boolean} true if list-like + */ +export function isListLike(schema) { + if (!schema) return false + if (schema.element.converted_type !== 'LIST') return false + if (schema.children.length > 1) return false + + const firstChild = schema.children[0] + if (firstChild.children.length > 1) return false + if (firstChild.element.repetition_type !== 'REPEATED') return false + + return true +} + +/** + * Check if a column is map-like. + * + * @param {SchemaTree} schema + * @returns {boolean} true if map-like + */ +export function isMapLike(schema) { + if (!schema) return false + if (schema.element.converted_type !== 'MAP') return false + if (schema.children.length > 1) return false + + const firstChild = schema.children[0] + if (firstChild.children.length !== 2) return false + if (firstChild.element.repetition_type !== 'REPEATED') return false + + const keyChild = firstChild.children.find(child => child.element.name === 'key') + if (keyChild?.element.repetition_type === 'REPEATED') return false + + const valueChild = firstChild.children.find(child => child.element.name === 'value') + if (valueChild?.element.repetition_type === 'REPEATED') return false + + return true +} + +/** + * Returns true if a column is non-nested. + * + * @param {SchemaTree[]} schemaPath + * @returns {boolean} + */ +export function isFlatColumn(schemaPath) { + if (schemaPath.length !== 2) return false + const [, column] = schemaPath + if (column.element.repetition_type === 'REPEATED') return false + if (column.children.length) return false + return true +} + +/** + * @import {SchemaElement, SchemaTree} from '../src/types.js' + */ diff --git a/web/node_modules/hyparquet/src/snappy.js b/web/node_modules/hyparquet/src/snappy.js new file mode 100644 index 0000000000000000000000000000000000000000..f7bf7698d83e4e3688bbcf4e7ebfd64bec4c1828 --- /dev/null +++ b/web/node_modules/hyparquet/src/snappy.js @@ -0,0 +1,127 @@ +/** + * The MIT License (MIT) + * Copyright (c) 2016 Zhipeng Jia + * https://github.com/zhipeng-jia/snappyjs + */ + +const WORD_MASK = [0, 0xff, 0xffff, 0xffffff, 0xffffffff] + +/** + * Copy bytes from one array to another + * + * @param {Uint8Array} fromArray source array + * @param {number} fromPos source position + * @param {Uint8Array} toArray destination array + * @param {number} toPos destination position + * @param {number} length number of bytes to copy + */ +function copyBytes(fromArray, fromPos, toArray, toPos, length) { + for (let i = 0; i < length; i++) { + toArray[toPos + i] = fromArray[fromPos + i] + } +} + +/** + * Decompress snappy data. + * Accepts an output buffer to avoid allocating a new buffer for each call. + * + * @param {Uint8Array} input compressed data + * @param {Uint8Array} output output buffer + */ +export function snappyUncompress(input, output) { + const inputLength = input.byteLength + const outputLength = output.byteLength + let pos = 0 + let outPos = 0 + + // skip preamble (contains uncompressed length as varint) + while (pos < inputLength) { + const c = input[pos] + pos++ + if (c < 128) { + break + } + } + if (outputLength && pos >= inputLength) { + throw new Error('invalid snappy length header') + } + + while (pos < inputLength) { + const c = input[pos] + let len = 0 + pos++ + + if (pos >= inputLength) { + throw new Error('missing eof marker') + } + + // There are two types of elements, literals and copies (back references) + if ((c & 0x3) === 0) { + // Literals are uncompressed data stored directly in the byte stream + let len = (c >>> 2) + 1 + // Longer literal length is encoded in multiple bytes + if (len > 60) { + if (pos + 3 >= inputLength) { + throw new Error('snappy error literal pos + 3 >= inputLength') + } + const lengthSize = len - 60 // length bytes - 1 + len = input[pos] + + (input[pos + 1] << 8) + + (input[pos + 2] << 16) + + (input[pos + 3] << 24) + len = (len & WORD_MASK[lengthSize]) + 1 + pos += lengthSize + } + if (pos + len > inputLength) { + throw new Error('snappy error literal exceeds input length') + } + copyBytes(input, pos, output, outPos, len) + pos += len + outPos += len + } else { + // Copy elements + let offset = 0 // offset back from current position to read + switch (c & 0x3) { + case 1: + // Copy with 1-byte offset + len = (c >>> 2 & 0x7) + 4 + offset = input[pos] + (c >>> 5 << 8) + pos++ + break + case 2: + // Copy with 2-byte offset + if (inputLength <= pos + 1) { + throw new Error('snappy error end of input') + } + len = (c >>> 2) + 1 + offset = input[pos] + (input[pos + 1] << 8) + pos += 2 + break + case 3: + // Copy with 4-byte offset + if (inputLength <= pos + 3) { + throw new Error('snappy error end of input') + } + len = (c >>> 2) + 1 + offset = input[pos] + + (input[pos + 1] << 8) + + (input[pos + 2] << 16) + + (input[pos + 3] << 24) + pos += 4 + break + default: + break + } + if (offset === 0 || isNaN(offset)) { + throw new Error(`invalid offset ${offset} pos ${pos} inputLength ${inputLength}`) + } + if (offset > outPos) { + throw new Error('cannot copy from before start of buffer') + } + copyBytes(output, outPos - offset, output, outPos, len) + outPos += len + } + } + + if (outPos !== outputLength) throw new Error('premature end of input') +} diff --git a/web/node_modules/hyparquet/src/thrift.js b/web/node_modules/hyparquet/src/thrift.js new file mode 100644 index 0000000000000000000000000000000000000000..5fcf8860eb318cfe0a9967102100110d2413e3fc --- /dev/null +++ b/web/node_modules/hyparquet/src/thrift.js @@ -0,0 +1,156 @@ +/** + * @import {DataReader, ThriftObject, ThriftType} from '../src/types.js' + */ + +// TCompactProtocol types +const STOP = 0 +const TRUE = 1 +const FALSE = 2 +const BYTE = 3 +const I16 = 4 +const I32 = 5 +const I64 = 6 +const DOUBLE = 7 +const BINARY = 8 +const LIST = 9 +const STRUCT = 12 + +/** + * Parse TCompactProtocol + * + * @param {DataReader} reader + * @returns {{ [key: `field_${number}`]: any }} + */ +export function deserializeTCompactProtocol(reader) { + /** @type {ThriftObject} */ + const value = {} + let fid = 0 + + while (reader.offset < reader.view.byteLength) { + // Parse each field based on its type and add to the result object + const byte = reader.view.getUint8(reader.offset++) + const type = byte & 0x0f + if (type === STOP) break + const delta = byte >> 4 + fid = delta ? fid + delta : readZigZag(reader) + value[`field_${fid}`] = readElement(reader, type) + } + + return value +} + +/** + * Read a single element based on its type + * + * @param {DataReader} reader + * @param {number} type + * @returns {ThriftType} + */ +function readElement(reader, type) { + switch (type) { + case TRUE: + return true + case FALSE: + return false + case BYTE: + return reader.view.getInt8(reader.offset++) + case I16: + case I32: + return readZigZag(reader) + case I64: + return readZigZagBigInt(reader) + case DOUBLE: { + const value = reader.view.getFloat64(reader.offset, true) + reader.offset += 8 + return value + } + case BINARY: { + const stringLength = readVarInt(reader) + const strBytes = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, stringLength) + reader.offset += stringLength + return strBytes + } + case LIST: { + const byte = reader.view.getUint8(reader.offset++) + const elemType = byte & 0x0f + let listSize = byte >> 4 + if (listSize === 15) { + listSize = readVarInt(reader) + } + const boolType = elemType === TRUE || elemType === FALSE + const values = new Array(listSize) + for (let i = 0; i < listSize; i++) { + values[i] = boolType ? readElement(reader, BYTE) === 1 : readElement(reader, elemType) + } + return values + } + case STRUCT: + // main function handles struct parsing + return deserializeTCompactProtocol(reader) + default: + // MAP, SET, UUID not used by parquet + throw new Error(`thrift unhandled type: ${type}`) + } +} + +/** + * Read varint aka Unsigned LEB128. + * + * @param {DataReader} reader + * @returns {number} + */ +export function readVarInt(reader) { + let result = 0 + let shift = 0 + while (true) { + // Read groups of 7 low bits until high bit is 0 + const byte = reader.view.getUint8(reader.offset++) + result |= (byte & 0x7f) << shift + if (!(byte & 0x80)) { + return result + } + shift += 7 + } +} + +/** + * Read a varint as a bigint. + * + * @param {DataReader} reader + * @returns {bigint} + */ +function readVarBigInt(reader) { + let result = 0n + let shift = 0n + while (true) { + const byte = reader.view.getUint8(reader.offset++) + result |= BigInt(byte & 0x7f) << shift + if (!(byte & 0x80)) { + return result + } + shift += 7n + } +} + +/** + * Read a zigzag number. + * Zigzag folds positive and negative numbers into the positive number space. + * + * @param {DataReader} reader + * @returns {number} + */ +export function readZigZag(reader) { + const zigzag = readVarInt(reader) + return zigzag >>> 1 ^ -(zigzag & 1) +} + +/** + * Read a zigzag bigint. + * + * @param {DataReader} reader + * @returns {bigint} + */ +export function readZigZagBigInt(reader) { + const zigzag = readVarBigInt(reader) + return zigzag >> 1n ^ -(zigzag & 1n) +} diff --git a/web/node_modules/hyparquet/src/types.d.ts b/web/node_modules/hyparquet/src/types.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..c4ea9d85d8688cdcfdb4a305134d9da06bf8ab0e --- /dev/null +++ b/web/node_modules/hyparquet/src/types.d.ts @@ -0,0 +1,563 @@ + +/** + * Custom parsers for columns + */ +export interface ParquetParsers { + timestampFromMilliseconds(millis: bigint): any + timestampFromMicroseconds(micros: bigint): any + timestampFromNanoseconds(nanos: bigint): any + dateFromDays(days: number): any + stringFromBytes(bytes: Uint8Array): any + jsonFromBytes(bytes: Uint8Array): any + geometryFromBytes(bytes: Uint8Array): any + geographyFromBytes(bytes: Uint8Array): any + uuidFromBytes(bytes: Uint8Array): any +} + +/** + * Parquet Metadata options for metadata parsing + */ +export interface MetadataOptions { + parsers?: ParquetParsers // custom parsers to decode advanced types + geoparquet?: boolean // parse geoparquet metadata and set logical type to geometry/geography for geospatial columns (default true) +} + +/** + * Parquet query options for reading data + */ +export interface BaseParquetReadOptions { + file: AsyncBuffer // file-like object containing parquet data + metadata?: FileMetaData // parquet metadata, will be parsed if not provided + columns?: string[] // columns to read, all columns if undefined + filter?: ParquetQueryFilter // filter applied to rows (requires rowFormat: 'object', onChunk is not filtered) + filterStrict?: boolean // if true filtering uses strict equality (default true) + rowStart?: number // first requested row index (inclusive) + rowEnd?: number // last requested row index (exclusive) + onChunk?: (chunk: ColumnData) => void // called when a column chunk is parsed. chunks may contain data outside the requested range. + onPage?: (chunk: SubColumnData) => void // called when a data page is parsed. pages may contain data outside the requested range. + compressors?: Compressors // custom decompressors + utf8?: boolean // decode byte arrays as utf8 strings (default true) + parsers?: ParquetParsers // custom parsers to decode advanced types + geoparquet?: boolean // parse geoparquet metadata and set logical type to geometry/geography for geospatial columns (default true) + useOffsetIndex?: boolean // use offset index to limit column chunk reads when available (default false) + useBloomFilters?: boolean // fetch bloom filters to enable row-group skipping on $eq/$in predicates (default false) +} + +interface ArrayRowFormat { + rowFormat?: 'array' // format of each row passed to the onComplete function. Can be omitted, as it's the default. + onComplete?: (rows: any[][]) => void // called when all requested rows and columns are parsed +} +interface ObjectRowFormat { + rowFormat: 'object' // format of each row passed to the onComplete function + onComplete?: (rows: Record[]) => void // called when all requested rows and columns are parsed +} +export type ParquetReadOptions = BaseParquetReadOptions & (ArrayRowFormat | ObjectRowFormat) + +/** + * Parquet query options for filtering data + */ +export type ParquetQueryFilter = + | ParquetQueryColumnsFilter + | { $and: ParquetQueryFilter[] } + | { $or: ParquetQueryFilter[] } + | { $nor: ParquetQueryFilter[] } +type ParquetQueryColumnsFilter = { [key: string]: ParquetQueryOperator } +export type ParquetQueryValue = string | number | bigint | boolean | object | null | undefined +export type ParquetQueryOperator = { + $gt?: ParquetQueryValue + $gte?: ParquetQueryValue + $lt?: ParquetQueryValue + $lte?: ParquetQueryValue + $eq?: ParquetQueryValue + $ne?: ParquetQueryValue + $in?: ParquetQueryValue[] + $nin?: ParquetQueryValue[] + $not?: ParquetQueryOperator +} + +/** + * A run of column data + */ +export interface ColumnData { + columnName: string + columnData: DecodedArray + rowStart: number + rowEnd: number // exclusive +} +/** + * A run of sub-column data (pre-assembly) + */ +export interface SubColumnData { + pathInSchema: string[] + columnData: DecodedArray + rowStart: number + rowEnd: number // exclusive +} + +/** + * File-like object that can read slices of a file asynchronously. + */ +export interface AsyncBuffer { + byteLength: number + slice(start: number, end?: number): Awaitable +} +export type Awaitable = T | Promise +export interface ByteRange { + startByte: number + endByte: number // exclusive +} + +export interface DataReader { + view: DataView + offset: number +} + +// Parquet file metadata types +export interface FileMetaData { + version: number + schema: SchemaElement[] + num_rows: bigint + row_groups: RowGroup[] + key_value_metadata?: KeyValue[] + created_by?: string + // column_orders?: ColumnOrder[] + // encryption_algorithm?: EncryptionAlgorithm + // footer_signing_key_metadata?: Uint8Array + metadata_length: number +} + +export interface SchemaTree { + children: SchemaTree[] + count: number + element: SchemaElement + path: string[] +} + +export interface SchemaElement { + type?: ParquetType + type_length?: number + repetition_type?: FieldRepetitionType + name: string + num_children?: number + converted_type?: ConvertedType + scale?: number + precision?: number + field_id?: number + logical_type?: LogicalType +} + +export type ParquetType = + | 'BOOLEAN' + | 'INT32' + | 'INT64' + | 'INT96' // deprecated + | 'FLOAT' + | 'DOUBLE' + | 'BYTE_ARRAY' + | 'FIXED_LEN_BYTE_ARRAY' + +export type FieldRepetitionType = + | 'REQUIRED' + | 'OPTIONAL' + | 'REPEATED' + +export type ConvertedType = + | 'UTF8' + | 'MAP' + | 'MAP_KEY_VALUE' + | 'LIST' + | 'ENUM' + | 'DECIMAL' + | 'DATE' + | 'TIME_MILLIS' + | 'TIME_MICROS' + | 'TIMESTAMP_MILLIS' + | 'TIMESTAMP_MICROS' + | 'UINT_8' + | 'UINT_16' + | 'UINT_32' + | 'UINT_64' + | 'INT_8' + | 'INT_16' + | 'INT_32' + | 'INT_64' + | 'JSON' + | 'BSON' + | 'INTERVAL' + +export type TimeUnit = 'MILLIS' | 'MICROS' | 'NANOS' + +type EdgeInterpolationAlgorithm = 'SPHERICAL' | 'VINCENTY' | 'THOMAS' | 'ANDOYER' | 'KARNEY' + +export type LogicalType = + | { type: 'STRING' } + | { type: 'MAP' } + | { type: 'LIST' } + | { type: 'ENUM' } + | { type: 'DATE' } + | { type: 'INTERVAL' } + | { type: 'NULL' } + | { type: 'JSON' } + | { type: 'BSON' } + | { type: 'UUID' } + | { type: 'FLOAT16' } + | { type: 'DECIMAL', precision: number, scale: number } + | { type: 'TIME', isAdjustedToUTC: boolean, unit: TimeUnit } + | { type: 'TIMESTAMP', isAdjustedToUTC: boolean, unit: TimeUnit } + | { type: 'INTEGER', bitWidth: number, isSigned: boolean } + | { type: 'VARIANT', specification_version?: number } + | { type: 'GEOMETRY', crs?: string } + | { type: 'GEOGRAPHY', crs?: string, algorithm?: EdgeInterpolationAlgorithm } + +export interface RowGroup { + columns: ColumnChunk[] + total_byte_size: bigint + num_rows: bigint + sorting_columns?: SortingColumn[] + file_offset?: bigint + total_compressed_size?: bigint + ordinal?: number +} + +export interface ColumnChunk { + file_path?: string + file_offset: bigint + meta_data?: ColumnMetaData + offset_index_offset?: bigint + offset_index_length?: number + column_index_offset?: bigint + column_index_length?: number + crypto_metadata?: ColumnCryptoMetaData + encrypted_column_metadata?: Uint8Array +} + +export interface ColumnMetaData { + type: ParquetType + encodings: Encoding[] + path_in_schema: string[] + codec: CompressionCodec + num_values: bigint + total_uncompressed_size: bigint + total_compressed_size: bigint + key_value_metadata?: KeyValue[] + data_page_offset: bigint + index_page_offset?: bigint + dictionary_page_offset?: bigint + statistics?: Statistics + encoding_stats?: PageEncodingStats[] + bloom_filter_offset?: bigint + bloom_filter_length?: number + size_statistics?: SizeStatistics + geospatial_statistics?: GeospatialStatistics +} + +type ColumnCryptoMetaData = Record + +export type Encoding = + | 'PLAIN' + | 'GROUP_VAR_INT' // deprecated + | 'PLAIN_DICTIONARY' + | 'RLE' + | 'BIT_PACKED' // deprecated + | 'DELTA_BINARY_PACKED' + | 'DELTA_LENGTH_BYTE_ARRAY' + | 'DELTA_BYTE_ARRAY' + | 'RLE_DICTIONARY' + | 'BYTE_STREAM_SPLIT' + +export type CompressionCodec = + | 'UNCOMPRESSED' + | 'SNAPPY' + | 'GZIP' + | 'LZO' + | 'BROTLI' + | 'LZ4' + | 'ZSTD' + | 'LZ4_RAW' + +export type Compressors = { + [K in CompressionCodec]?: (input: Uint8Array, outputLength: number) => Uint8Array +} + +export interface KeyValue { + key: string + value?: string +} + +export type MinMaxType = bigint | boolean | number | string | Date | Uint8Array + +export interface Statistics { + max?: MinMaxType + min?: MinMaxType + null_count?: bigint + distinct_count?: bigint + max_value?: MinMaxType + min_value?: MinMaxType + is_max_value_exact?: boolean + is_min_value_exact?: boolean +} + +interface SizeStatistics { + unencoded_byte_array_data_bytes?: bigint + repetition_level_histogram?: bigint[] + definition_level_histogram?: bigint[] +} + +export interface GeospatialStatistics { + bbox?: BoundingBox + geospatial_types?: number[] +} + +export interface BoundingBox { + xmin: number + xmax: number + ymin: number + ymax: number + zmin?: number + zmax?: number + mmin?: number + mmax?: number +} + +export interface PageEncodingStats { + page_type: PageType + encoding: Encoding + count: number +} + +export interface BloomFilter { + numBytes: number + blocks: Uint32Array +} + +export type PageType = + 'DATA_PAGE' | + 'INDEX_PAGE' | + 'DICTIONARY_PAGE' | + 'DATA_PAGE_V2' + +interface SortingColumn { + column_idx: number + descending: boolean + nulls_first: boolean +} + +// Parquet file header types +export interface PageHeader { + type: PageType + uncompressed_page_size: number + compressed_page_size: number + crc?: number + data_page_header?: DataPageHeader + index_page_header?: IndexPageHeader + dictionary_page_header?: DictionaryPageHeader + data_page_header_v2?: DataPageHeaderV2 +} + +export interface DataPageHeader { + num_values: number + encoding: Encoding + definition_level_encoding: Encoding + repetition_level_encoding: Encoding + statistics?: Statistics +} + +type IndexPageHeader = Record + +export interface DictionaryPageHeader { + num_values: number + encoding: Encoding + is_sorted?: boolean +} + +export interface DataPageHeaderV2 { + num_values: number + num_nulls: number + num_rows: number + encoding: Encoding + definition_levels_byte_length: number + repetition_levels_byte_length: number + is_compressed?: boolean + statistics?: Statistics +} + +interface DataPage { + definitionLevels: number[] | undefined + repetitionLevels: number[] + dataPage: DecodedArray +} + +export type DecodedArray = + | Uint8Array + | Uint32Array + | Int32Array + | BigInt64Array + | BigUint64Array + | Float32Array + | Float64Array + | any[] + +/** Wrapper around decoded page data */ +export interface PageResult { + skipped: number + data?: DecodedArray +} + +export interface OffsetIndex { + page_locations: PageLocation[] + unencoded_byte_array_data_bytes?: bigint[] +} + +interface PageLocation { + offset: bigint + compressed_page_size: number + first_row_index: bigint +} + +export interface ColumnIndex { + null_pages: boolean[] + min_values: Uint8Array[] + max_values: Uint8Array[] + boundary_order: BoundaryOrder + null_counts?: bigint[] + repetition_level_histograms?: bigint[] + definition_level_histograms?: bigint[] +} + +export type BoundaryOrder = 'UNORDERED' | 'ASCENDING' | 'DESCENDING' + +export interface VariantMetadata { + dictionary: string[] + sorted: boolean +} + +export type ThriftObject = { [ key: `field_${number}` ]: ThriftType } +export type ThriftType = boolean | number | bigint | Uint8Array | ThriftType[] | ThriftObject + +/** + * Query plan for which byte ranges to read. + */ +export interface QueryPlan { + metadata: FileMetaData + rowStart: number + rowEnd?: number + columns?: string[] // columns to read + fetches: ByteRange[] // byte ranges to fetch + groups: GroupPlan[] // byte ranges by row group +} +// Plan for one group +interface GroupPlan { + chunks: ChunkPlan[] + rowGroup: RowGroup // row group metadata + groupStart: number // row index of the first row in the group + selectStart: number // row index in the group to start reading + selectEnd: number // row index in the group to stop reading + groupRows: number // number of rows in the group +} +// Plan for one column within a row group +type ChunkPlan = ChunkFull | ChunkOffsetIndexed +// full column chunk +interface ChunkFull { + columnMetadata: ColumnMetaData + range: ByteRange +} +// column chunk with offset index pending +interface ChunkOffsetIndexed { + columnMetadata: ColumnMetaData + offsetIndex: ByteRange + range: ByteRange +} + +export interface ColumnDecoder { + pathInSchema: string[] + type: ParquetType + element: SchemaElement + schemaPath: SchemaTree[] + codec: CompressionCodec + parsers: ParquetParsers + compressors?: Compressors + utf8?: boolean +} + +export interface RowGroupSelect { + groupStart: number // row index of the first row in the group + selectStart: number // row index in the group to start reading + selectEnd: number // row index in the group to stop reading + groupRows: number +} + +export interface AsyncRowGroup { + groupStart: number + groupRows: number + asyncColumns: AsyncColumn[] +} +export interface AsyncColumn { + pathInSchema: string[] + data: Promise +} +interface AsyncPages { + skipped: number // rows skipped from groupStart to first row of this column data + data: DecodedArray[] +} + +/** + * Geometry types based on the GeoJSON specification (RFC 7946) + */ +export type Geometry = + | Point + | MultiPoint + | LineString + | MultiLineString + | Polygon + | MultiPolygon + | GeometryCollection + +/** + * Position is an array of at least two numbers. + * The order should be [longitude, latitude] with optional properties (eg- altitude). + */ +export type Position = number[] + +export interface Point { + type: 'Point' + coordinates: Position +} + +export interface MultiPoint { + type: 'MultiPoint' + coordinates: Position[] +} + +export interface LineString { + type: 'LineString' + coordinates: Position[] +} + +/** + * Each element is one LineString. + */ +export interface MultiLineString { + type: 'MultiLineString' + coordinates: Position[][] +} + +/** + * Each element is a linear ring. + */ +export interface Polygon { + type: 'Polygon' + coordinates: Position[][] +} + +/** + * Each element is one Polygon. + */ +export interface MultiPolygon { + type: 'MultiPolygon' + coordinates: Position[][][] +} + +export interface GeometryCollection { + type: 'GeometryCollection' + geometries: Geometry[] +} diff --git a/web/node_modules/hyparquet/src/utils.js b/web/node_modules/hyparquet/src/utils.js new file mode 100644 index 0000000000000000000000000000000000000000..c80625ef11b6fac118ef5294cf19ff3d31b52eb2 --- /dev/null +++ b/web/node_modules/hyparquet/src/utils.js @@ -0,0 +1,290 @@ +/** + * @import {AsyncBuffer, Awaitable, DecodedArray} from '../src/types.js' + */ + +import { defaultInitialFetchSize } from './metadata.js' + +/** + * Replace bigint, date, etc with legal JSON types. + * + * @param {any} obj object to convert + * @returns {unknown} converted object + */ +export function toJson(obj) { + if (obj === undefined) return null + if (typeof obj === 'bigint') return Number(obj) + if (Object.is(obj, -0)) return 0 + if (Array.isArray(obj)) return obj.map(toJson) + if (obj instanceof Uint8Array) return Array.from(obj) + if (obj instanceof Date) return obj.toISOString() + if (obj instanceof Object) { + /** @type {Record} */ + const newObj = {} + for (const key of Object.keys(obj)) { + if (obj[key] === undefined) continue + newObj[key] = toJson(obj[key]) + } + return newObj + } + return obj +} + +/** + * Concatenate two arrays fast. + * + * @param {any[]} aaa + * @param {DecodedArray} bbb + */ +export function concat(aaa, bbb) { + const chunk = 10000 + for (let i = 0; i < bbb.length; i += chunk) { + aaa.push(...bbb.slice(i, i + chunk)) + } +} + +/** + * Deep equality. + * + * @param {any} a + * @param {any} b + * @param {boolean} [strict] + * @returns {boolean} + */ +export function equals(a, b, strict = true) { + // eslint-disable-next-line eqeqeq + if (strict ? a === b : a == b) return true + if (!a || !b || typeof a !== 'object' || typeof b !== 'object') return false + + if (a instanceof Uint8Array && b instanceof Uint8Array) { + if (a.length !== b.length) return false + for (let i = 0; i < a.length; i++) { + if (a[i] !== b[i]) return false + } + return true + } + if (Array.isArray(a) && Array.isArray(b)) { + if (a.length !== b.length) return false + for (let i = 0; i < a.length; i++) { + if (!equals(a[i], b[i], strict)) return false + } + return true + } + + const aKeys = Object.keys(a) + if (aKeys.length !== Object.keys(b).length) return false + for (const k of aKeys) { + if (!equals(a[k], b[k], strict)) return false + } + return true +} + +/** + * Get the byte length using fetch with a ranged GET request. + * Aborts the request if server returns 200 instead of 206. + * + * @param {string} url + * @param {RequestInit} [requestInit] fetch options + * @param {typeof globalThis.fetch} [fetchFn] fetch function to use + * @returns {Promise} + */ +async function byteLengthFromUrlUsingGet(url, requestInit = {}, fetchFn = globalThis.fetch) { + const controller = new AbortController() + const headers = new Headers(requestInit.headers) + headers.set('Range', 'bytes=0-0') + + const res = await fetchFn(url, { + ...requestInit, + headers, + signal: controller.signal, + }) + + if (!res.ok) throw new Error(`fetch with range failed ${res.status}`) + + // Server supports Range requests (206 Partial Content) + if (res.status === 206) { + const contentRange = res.headers.get('Content-Range') + if (!contentRange) throw new Error('missing content-range header') + + // Parse "bytes 0-0/9446073" to get total length + const match = contentRange.match(/bytes \d+-\d+\/(\d+)/) + if (!match) throw new Error(`invalid content-range header: ${contentRange}`) + + return parseInt(match[1]) + } + + // Server ignored Range and returned 200 - get Content-Length and abort request + if (res.status === 200) { + const contentLength = res.headers.get('Content-Length') + + // Abort the request to stop any ongoing download + controller.abort() + + if (contentLength) return parseInt(contentLength) + } + + throw new Error('server does not support range requests and missing content-length') +} + +/** + * Get the byte length of a URL using a HEAD request. + * If HEAD fails with 403 (e.g., with signed S3 URLs), falls back to a ranged GET request. + * If HEAD succeeds but Content-Length is missing, falls back to GET with range. + * If requestInit is provided, it will be passed to fetch. + * + * @param {string} url + * @param {RequestInit} [requestInit] fetch options + * @param {typeof globalThis.fetch} [customFetch] fetch function to use + * @returns {Promise} + */ +export async function byteLengthFromUrl(url, requestInit, customFetch) { + const fetch = customFetch ?? globalThis.fetch + const res = await fetch(url, { ...requestInit, method: 'HEAD' }) + + // If HEAD request is forbidden (common with signed S3 URLs), try GET with range + if (res.status === 403) { + return byteLengthFromUrlUsingGet(url, requestInit, fetch) + } + + if (!res.ok) throw new Error(`fetch head failed ${res.status}`) + const length = res.headers.get('Content-Length') + // If Content-Length is missing from HEAD, fallback to GET with range + if (!length) { + return byteLengthFromUrlUsingGet(url, requestInit, fetch) + } + return parseInt(length) +} + +/** + * Construct an AsyncBuffer for a URL. + * If byteLength is not provided, will make a HEAD request to get the file size. + * If fetch is provided, it will be used instead of the global fetch. + * If requestInit is provided, it will be passed to fetch. + * + * @param {object} options + * @param {string} options.url + * @param {number} [options.byteLength] + * @param {typeof globalThis.fetch} [options.fetch] fetch function to use + * @param {RequestInit} [options.requestInit] + * @returns {Promise} + */ +export async function asyncBufferFromUrl({ url, byteLength, requestInit, fetch: customFetch }) { + if (!url) throw new Error('missing url') + const fetch = customFetch ?? globalThis.fetch + // byte length from HEAD request + byteLength ??= await byteLengthFromUrl(url, requestInit, fetch) + + /** + * A promise for the whole buffer, if range requests are not supported. + * @type {Promise|undefined} + */ + let buffer = undefined + const init = requestInit || {} + + return { + byteLength, + async slice(start, end) { + if (buffer) { + return buffer.then(buffer => buffer.slice(start, end)) + } + + const headers = new Headers(init.headers) + const endStr = end === undefined ? '' : end - 1 + headers.set('Range', `bytes=${start}-${endStr}`) + + const res = await fetch(url, { ...init, headers }) + if (!res.ok || !res.body) throw new Error(`fetch failed ${res.status}`) + + if (res.status === 200) { + // Endpoint does not support range requests and returned the whole object + buffer = res.arrayBuffer() + return buffer.then(buffer => buffer.slice(start, end)) + } else if (res.status === 206) { + // The endpoint supports range requests and sent us the requested range + return res.arrayBuffer() + } else { + throw new Error(`fetch received unexpected status code ${res.status}`) + } + }, + } +} + +/** + * Returns a cached layer on top of an AsyncBuffer. For caching slices of a file + * that are read multiple times, possibly over a network. + * + * @param {AsyncBuffer} file file-like object to cache + * @param {{ minSize?: number }} [options] + * @returns {AsyncBuffer} cached file-like object + */ +export function cachedAsyncBuffer({ byteLength, slice }, { minSize = defaultInitialFetchSize } = {}) { + if (byteLength < minSize) { + // Cache whole file if it's small + const buffer = slice(0, byteLength) + return { + byteLength, + async slice(start, end) { + return (await buffer).slice(start, end) + }, + } + } + const cache = new Map() + return { + byteLength, + /** + * @param {number} start + * @param {number} [end] + * @returns {Awaitable} + */ + slice(start, end) { + const key = cacheKey(start, end, byteLength) + const cached = cache.get(key) + if (cached) return cached + // cache miss, read from file + const promise = slice(start, end) + cache.set(key, promise) + return promise + }, + } +} + + +/** + * Returns canonical cache key for a byte range 'start,end'. + * Normalize int-range and suffix-range requests to the same key. + * + * @param {number} start start byte of range + * @param {number} [end] end byte of range, or undefined for suffix range + * @param {number} [size] size of file, or undefined for suffix range + * @returns {string} + */ +function cacheKey(start, end, size) { + if (start < 0) { + if (end !== undefined) throw new Error(`invalid suffix range [${start}, ${end}]`) + if (size === undefined) return `${start},` + return `${size + start},${size}` + } else if (end !== undefined) { + if (start > end) throw new Error(`invalid empty range [${start}, ${end}]`) + return `${start},${end}` + } else if (size === undefined) { + return `${start},` + } else { + return `${start},${size}` + } +} + +/** + * Flatten a list of lists into a single list. + * + * @param {DecodedArray[]} [chunks] + * @returns {DecodedArray} + */ +export function flatten(chunks) { + if (!chunks) return [] + if (chunks.length === 1) return chunks[0] + /** @type {any[]} */ + const output = [] + for (const chunk of chunks) { + concat(output, chunk) + } + return output +} diff --git a/web/node_modules/hyparquet/src/variant.js b/web/node_modules/hyparquet/src/variant.js new file mode 100644 index 0000000000000000000000000000000000000000..3efa7335335275c1e3a44409bd59b4928c860a49 --- /dev/null +++ b/web/node_modules/hyparquet/src/variant.js @@ -0,0 +1,366 @@ +import { DEFAULT_PARSERS } from './convert.js' + +/** + * @import {DataReader, ParquetParsers, VariantMetadata} from '../src/types.js' + */ + +const decoder = new TextDecoder() +/** @type {WeakMap>} */ +const metadataCache = new WeakMap() + +/** + * Recursively decode variant structs into native values. + * + * @param {any} value + * @param {ParquetParsers} [parsers] + * @returns {any} + */ +export function decodeVariantColumn(value, parsers = DEFAULT_PARSERS) { + if (Array.isArray(value)) { + return value.map(entry => decodeVariantColumn(entry, parsers)) + } + if (typeof value !== 'object') return value + + if ('metadata' in value) { + const metadata = parseVariantMetadata(value.metadata) + + // Decode shredded fields from typed_value + const shreddedFields = value.typed_value && decodeTypedValue(value.typed_value, metadata, parsers) + + // Decode binary value (may contain additional fields for partially shredded objects) + const binaryValue = value.value && readVariant(makeReader(value.value), metadata, parsers) + + // Merge shredded and binary values for partially shredded objects + if (shreddedFields && binaryValue) { + return { ...binaryValue, ...shreddedFields } + } + return shreddedFields ?? binaryValue + } + + return value +} + +/** + * Decode a shredded variant typed_value field. + * + * @param {any} typedValue + * @param {VariantMetadata} metadata + * @param {ParquetParsers} parsers + * @returns {any} + */ +function decodeTypedValue(typedValue, metadata, parsers) { + if (typedValue instanceof Date) return typedValue + + // Handle {typed_value, value} wrapper - unwrap and recurse + if (typedValue && typeof typedValue === 'object' && !Array.isArray(typedValue) && !(typedValue instanceof Uint8Array)) { + if ('typed_value' in typedValue && typedValue.typed_value !== null && typedValue.typed_value !== undefined) { + return decodeTypedValue(typedValue.typed_value, metadata, parsers) + } + if ('value' in typedValue && typedValue.value instanceof Uint8Array) { + return readVariant(makeReader(typedValue.value), metadata, parsers) + } + if ('typed_value' in typedValue || 'value' in typedValue) { + // both null/missing → field is null + return null + } + // Shredded object: each field value gets decoded + /** @type {Record} */ + const result = {} + for (const [key, field] of Object.entries(typedValue)) { + if (!metadata.dictionary.includes(key)) continue + result[key] = decodeTypedValue(field, metadata, parsers) + } + return result + } + + // Uint8Array: decode as binary variant + if (typedValue instanceof Uint8Array) { + return readVariant(makeReader(typedValue), metadata, parsers) + } + + // Arrays + if (Array.isArray(typedValue)) { + return typedValue.map(element => decodeTypedValue(element, metadata, parsers)) + } + + return typedValue +} + +/** + * @param {Uint8Array} bytes + * @returns {DataReader} + */ +function makeReader(bytes) { + return { view: new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), offset: 0 } +} + +/** + * Parse and cache variant metadata dictionary. + * + * @param {Uint8Array} bytes + * @returns {VariantMetadata} + */ +function parseVariantMetadata(bytes) { + let bufferCache = metadataCache.get(bytes.buffer) + if (!bufferCache) { + bufferCache = new Map() + metadataCache.set(bytes.buffer, bufferCache) + } + const key = `${bytes.byteOffset}:${bytes.byteLength}` + const cached = bufferCache.get(key) + if (cached) return cached + + const reader = makeReader(bytes) + const header = reader.view.getUint8(reader.offset++) + const version = header & 0x0f + if (version !== 1) throw new Error(`parquet unsupported variant metadata version: ${version}`) + const sorted = (header >> 4 & 0x1) === 1 + const offsetSize = (header >> 6 & 0x3) + 1 + + const dictionarySize = readUnsigned(reader, offsetSize) + + const offsets = new Array(dictionarySize + 1) + for (let i = 0; i < offsets.length; i++) { + offsets[i] = readUnsigned(reader, offsetSize) + } + + const base = reader.offset + const dictionary = new Array(dictionarySize) + for (let i = 0; i < dictionarySize; i++) { + const start = offsets[i] + const end = offsets[i + 1] + const strBytes = new Uint8Array(bytes.buffer, bytes.byteOffset + base + start, end - start) + dictionary[i] = decoder.decode(strBytes) + } + + const metadata = { dictionary, sorted } + bufferCache.set(key, metadata) + return metadata +} + +/** + * @param {DataReader} reader + * @param {number} byteWidth + * @returns {number} + */ +function readUnsigned(reader, byteWidth) { + let value = 0 + for (let i = 0; i < byteWidth; i++) { + value |= reader.view.getUint8(reader.offset + i) << i * 8 + } + reader.offset += byteWidth + return value +} + +/** + * @param {DataReader} reader + * @param {VariantMetadata} metadata + * @param {ParquetParsers} parsers + * @returns {any} + */ +function readVariant(reader, metadata, parsers) { + const typeByte = reader.view.getUint8(reader.offset++) + const basicType = typeByte & 0x3 + const header = typeByte >> 2 + if (basicType === 0) return readVariantPrimitive(reader, header, parsers) + if (basicType === 2) return readVariantObject(reader, header, metadata, parsers) + if (basicType === 3) return readVariantArray(reader, header, metadata, parsers) + // else short string + const bytes = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, header) + reader.offset += header + return decoder.decode(bytes) +} + +/** + * @param {DataReader} reader + * @param {number} typeId + * @param {ParquetParsers} parsers + * @returns {any} + */ +function readVariantPrimitive(reader, typeId, parsers) { + switch (typeId) { + case 0: return null + case 1: return true + case 2: return false + case 3: { + const value = reader.view.getInt8(reader.offset) + reader.offset += 1 + return value + } + case 4: { + const value = reader.view.getInt16(reader.offset, true) + reader.offset += 2 + return value + } + case 5: { + const value = reader.view.getInt32(reader.offset, true) + reader.offset += 4 + return value + } + case 6: { + const value = reader.view.getBigInt64(reader.offset, true) + reader.offset += 8 + return value + } + case 7: { + const value = reader.view.getFloat64(reader.offset, true) + reader.offset += 8 + return value + } + case 8: + return readVariantDecimal(reader, 4) + case 9: + return readVariantDecimal(reader, 8) + case 10: + return readVariantDecimal(reader, 16) + case 11: { + const value = reader.view.getInt32(reader.offset, true) + reader.offset += 4 + return parsers.dateFromDays(value) + } + case 12: // timestamp_micros (utc) + case 13: { // timestamp_micros_ntz (no timezone) + const value = reader.view.getBigInt64(reader.offset, true) + reader.offset += 8 + return parsers.timestampFromMicroseconds(value) + } + case 14: { + const value = reader.view.getFloat32(reader.offset, true) + reader.offset += 4 + return value + } + case 15: + return readVariantBinary(reader) + case 16: { + const bytes = readVariantBinary(reader) + return decoder.decode(bytes) + } + case 17: { + // time: microseconds since midnight + const value = reader.view.getBigInt64(reader.offset, true) + reader.offset += 8 + return value + } + case 18: // timestamp_nanos (utc) + case 19: { // timestamp_nanos_ntz (no timezone) + const value = reader.view.getBigInt64(reader.offset, true) + reader.offset += 8 + return parsers.timestampFromNanoseconds(value) + } + case 20: { + const bytes = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, 16) + reader.offset += 16 + const hex = Array.from(bytes, b => b.toString(16).padStart(2, '0')).join('') + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}` + } + default: + throw new Error(`parquet unsupported variant primitive type: ${typeId}`) + } +} + +/** + * @param {DataReader} reader + * @param {number} header + * @param {VariantMetadata} metadata + * @param {ParquetParsers} parsers + * @returns {Record} + */ +function readVariantObject(reader, header, metadata, parsers) { + const offsetWidth = (header & 0x3) + 1 + const idWidth = (header >> 2 & 0x3) + 1 + const isLarge = header >> 4 & 0x1 + const numElements = isLarge ? readUnsigned(reader, 4) : reader.view.getUint8(reader.offset++) + + /** @type {number[]} */ + const fieldIds = new Array(numElements) + for (let i = 0; i < numElements; i++) { + fieldIds[i] = readUnsigned(reader, idWidth) + } + + const offsets = new Array(numElements + 1) + for (let i = 0; i < offsets.length; i++) { + offsets[i] = readUnsigned(reader, offsetWidth) + } + + /** @type {Record} */ + const out = {} + for (let i = 0; i < numElements; i++) { + const key = metadata.dictionary[fieldIds[i]] + // Read value at the given offset + const valueReader = { + view: reader.view, + offset: reader.offset + offsets[i], + } + out[key] = readVariant(valueReader, metadata, parsers) + } + reader.offset += offsets[offsets.length - 1] + return out +} + +/** + * @param {DataReader} reader + * @param {number} header + * @param {VariantMetadata} metadata + * @param {ParquetParsers} parsers + * @returns {any[]} + */ +function readVariantArray(reader, header, metadata, parsers) { + const fieldOffsetSize = header & 0x3 + const isLarge = header >> 2 & 0x1 + const offsetWidth = fieldOffsetSize + 1 + const numElements = readUnsigned(reader, isLarge ? 4 : 1) + + const offsets = new Array(numElements + 1) + for (let i = 0; i < offsets.length; i++) { + offsets[i] = readUnsigned(reader, offsetWidth) + } + + const valuesStart = reader.offset + const result = new Array(numElements) + for (let i = 0; i < numElements; i++) { + const valueReader = { + view: reader.view, + offset: valuesStart + offsets[i], + } + result[i] = readVariant(valueReader, metadata, parsers) + } + reader.offset = valuesStart + offsets[offsets.length - 1] + return result +} + +/** + * @param {DataReader} reader + * @param {number} width + * @returns {number} + */ +function readVariantDecimal(reader, width) { + const scale = reader.view.getUint8(reader.offset) + reader.offset += 1 + let unscaled + if (width === 4) { + unscaled = BigInt(reader.view.getInt32(reader.offset, true)) + reader.offset += 4 + } else if (width === 8) { + unscaled = reader.view.getBigInt64(reader.offset, true) + reader.offset += 8 + } else { + const low = reader.view.getBigUint64(reader.offset, true) + const high = reader.view.getBigInt64(reader.offset + 8, true) + unscaled = high << 64n | low + reader.offset += 16 + } + + return Number(unscaled) * 10 ** -scale +} + +/** + * @param {DataReader} reader + * @returns {Uint8Array} + */ +function readVariantBinary(reader) { + const length = reader.view.getUint32(reader.offset, true) + reader.offset += 4 + const bytes = new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, length) + reader.offset += length + return bytes +} diff --git a/web/node_modules/hyparquet/src/wkb.js b/web/node_modules/hyparquet/src/wkb.js new file mode 100644 index 0000000000000000000000000000000000000000..f66e651d5af9b381795955f2d81877ee862e158a --- /dev/null +++ b/web/node_modules/hyparquet/src/wkb.js @@ -0,0 +1,128 @@ +/** + * WKB (Well-Known Binary) decoder for geometry objects. + * + * @param {DataReader} reader + * @returns {Geometry} geometry object + */ +export function wkbToGeojson(reader) { + const flags = getFlags(reader) + + if (flags.type === 1) { // Point + return { type: 'Point', coordinates: readPosition(reader, flags) } + } else if (flags.type === 2) { // LineString + return { type: 'LineString', coordinates: readLine(reader, flags) } + } else if (flags.type === 3) { // Polygon + return { type: 'Polygon', coordinates: readPolygon(reader, flags) } + } else if (flags.type === 4) { // MultiPoint + const points = [] + for (let i = 0; i < flags.count; i++) { + points.push(readPosition(reader, getFlags(reader))) + } + return { type: 'MultiPoint', coordinates: points } + } else if (flags.type === 5) { // MultiLineString + const lines = [] + for (let i = 0; i < flags.count; i++) { + lines.push(readLine(reader, getFlags(reader))) + } + return { type: 'MultiLineString', coordinates: lines } + } else if (flags.type === 6) { // MultiPolygon + const polygons = [] + for (let i = 0; i < flags.count; i++) { + polygons.push(readPolygon(reader, getFlags(reader))) + } + return { type: 'MultiPolygon', coordinates: polygons } + } else if (flags.type === 7) { // GeometryCollection + const geometries = [] + for (let i = 0; i < flags.count; i++) { + geometries.push(wkbToGeojson(reader)) + } + return { type: 'GeometryCollection', geometries } + } else { + throw new Error(`Unsupported geometry type: ${flags.type}`) + } +} + +/** + * Extract ISO WKB flags and base geometry type. + * + * @param {DataReader} reader + * @returns {WkbFlags} + */ +function getFlags(reader) { + const { view } = reader + const littleEndian = view.getUint8(reader.offset++) === 1 + const rawType = view.getUint32(reader.offset, littleEndian) + reader.offset += 4 + + const type = rawType % 1000 + const flags = Math.floor(rawType / 1000) + + let count = 0 + if (type > 1 && type <= 7) { + count = view.getUint32(reader.offset, littleEndian) + reader.offset += 4 + } + + // XY, XYZ, XYM, XYZM + let dim = 2 + if (flags) dim++ + if (flags === 3) dim++ + + return { littleEndian, type, dim, count } +} + +/** + * @param {DataReader} reader + * @param {WkbFlags} flags + * @returns {number[]} + */ +function readPosition(reader, flags) { + const points = [] + for (let i = 0; i < flags.dim; i++) { + const coord = reader.view.getFloat64(reader.offset, flags.littleEndian) + reader.offset += 8 + points.push(coord) + } + return points +} + +/** + * @param {DataReader} reader + * @param {WkbFlags} flags + * @returns {number[][]} + */ +function readLine(reader, flags) { + const points = [] + for (let i = 0; i < flags.count; i++) { + points.push(readPosition(reader, flags)) + } + return points +} + +/** + * @param {DataReader} reader + * @param {WkbFlags} flags + * @returns {number[][][]} + */ +function readPolygon(reader, flags) { + const { view } = reader + const rings = [] + for (let r = 0; r < flags.count; r++) { + const count = view.getUint32(reader.offset, flags.littleEndian) + reader.offset += 4 + rings.push(readLine(reader, { ...flags, count })) + } + return rings +} + +/** + * @typedef {object} WkbFlags + * @property {boolean} littleEndian + * @property {number} type + * @property {number} dim + * @property {number} count + */ + +/** + * @import {DataReader, Geometry} from '../src/types.js' + */ diff --git a/web/node_modules/hyparquet/src/xxhash.js b/web/node_modules/hyparquet/src/xxhash.js new file mode 100644 index 0000000000000000000000000000000000000000..aa006e9b34f874fa19fcf24949dd78adef63feb5 --- /dev/null +++ b/web/node_modules/hyparquet/src/xxhash.js @@ -0,0 +1,102 @@ +// XxHash64 (https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md) +// Used by parquet bloom filters: hashes the PLAIN encoding of column values with seed 0. + +const MASK = 0xffffffffffffffffn +const PRIME1 = 0x9e3779b185ebca87n +const PRIME2 = 0xc2b2ae3d27d4eb4fn +const PRIME3 = 0x165667b19e3779f9n +const PRIME4 = 0x85ebca77c2b2ae63n +const PRIME5 = 0x27d4eb2f165667c5n + +/** + * @param {bigint} x + * @param {bigint} r rotation amount in bits (1..63) + * @returns {bigint} + */ +function rotl64(x, r) { + return (x << r | x >> 64n - r) & MASK +} + +/** + * @param {bigint} acc + * @param {bigint} val + * @returns {bigint} + */ +function round(acc, val) { + acc = acc + val * PRIME2 & MASK + acc = rotl64(acc, 31n) + return acc * PRIME1 & MASK +} + +/** + * @param {bigint} acc + * @param {bigint} val + * @returns {bigint} + */ +function mergeRound(acc, val) { + acc ^= round(0n, val) + return acc * PRIME1 + PRIME4 & MASK +} + +/** + * Compute the 64-bit xxHash of a byte buffer. + * + * @param {Uint8Array} input + * @param {bigint} [seed] + * @returns {bigint} 64-bit hash + */ +export function xxhash64(input, seed = 0n) { + const view = new DataView(input.buffer, input.byteOffset, input.byteLength) + const len = input.byteLength + let offset = 0 + let h64 + + if (len >= 32) { + let v1 = seed + PRIME1 + PRIME2 & MASK + let v2 = seed + PRIME2 & MASK + let v3 = seed + let v4 = seed - PRIME1 & MASK + + while (offset + 32 <= len) { + v1 = round(v1, view.getBigUint64(offset, true)); offset += 8 + v2 = round(v2, view.getBigUint64(offset, true)); offset += 8 + v3 = round(v3, view.getBigUint64(offset, true)); offset += 8 + v4 = round(v4, view.getBigUint64(offset, true)); offset += 8 + } + + h64 = rotl64(v1, 1n) + rotl64(v2, 7n) + rotl64(v3, 12n) + rotl64(v4, 18n) & MASK + h64 = mergeRound(h64, v1) + h64 = mergeRound(h64, v2) + h64 = mergeRound(h64, v3) + h64 = mergeRound(h64, v4) + } else { + h64 = seed + PRIME5 & MASK + } + + h64 = h64 + BigInt(len) & MASK + + while (offset + 8 <= len) { + h64 ^= round(0n, view.getBigUint64(offset, true)) + h64 = rotl64(h64, 27n) * PRIME1 + PRIME4 & MASK + offset += 8 + } + + if (offset + 4 <= len) { + h64 ^= BigInt(view.getUint32(offset, true)) * PRIME1 & MASK + h64 = rotl64(h64, 23n) * PRIME2 + PRIME3 & MASK + offset += 4 + } + + while (offset < len) { + h64 ^= BigInt(view.getUint8(offset)) * PRIME5 & MASK + h64 = rotl64(h64, 11n) * PRIME1 & MASK + offset += 1 + } + + h64 ^= h64 >> 33n + h64 = h64 * PRIME2 & MASK + h64 ^= h64 >> 29n + h64 = h64 * PRIME3 & MASK + h64 ^= h64 >> 32n + return h64 +} diff --git a/web/node_modules/hyparquet/types/assemble.d.ts b/web/node_modules/hyparquet/types/assemble.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..caaa7c242667ff3d7f0b385d56e7a4edd3c92a36 --- /dev/null +++ b/web/node_modules/hyparquet/types/assemble.d.ts @@ -0,0 +1,25 @@ +/** + * Reconstructs a complex nested structure from flat arrays of values and + * definition and repetition levels, according to Dremel encoding. + * + * @param {any[]} output + * @param {number[] | undefined} definitionLevels + * @param {number[]} repetitionLevels + * @param {DecodedArray} values + * @param {SchemaTree[]} schemaPath + * @returns {DecodedArray} + */ +export function assembleLists(output: any[], definitionLevels: number[] | undefined, repetitionLevels: number[], values: DecodedArray, schemaPath: SchemaTree[]): DecodedArray; +/** + * Assemble a nested structure from subcolumn data. + * + * @param {Map} subcolumnData + * @param {SchemaTree} schema top-level schema element + * @param {ParquetParsers} parsers + * @param {number} [depth] depth of nested structure + */ +export function assembleNested(subcolumnData: Map, schema: SchemaTree, parsers: ParquetParsers, depth?: number): void; +import type { DecodedArray } from '../src/types.js'; +import type { SchemaTree } from '../src/types.js'; +import type { ParquetParsers } from '../src/types.js'; +//# sourceMappingURL=assemble.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/assemble.d.ts.map b/web/node_modules/hyparquet/types/assemble.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..47eeb23ab58a7e71741d9f1df76a763b2c7bd6eb --- /dev/null +++ b/web/node_modules/hyparquet/types/assemble.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"assemble.d.ts","sourceRoot":"","sources":["../src/assemble.js"],"names":[],"mappings":"AAGA;;;;;;;;;;GAUG;AACH,sCAPW,GAAG,EAAE,oBACL,MAAM,EAAE,GAAG,SAAS,oBACpB,MAAM,EAAE,UACR,YAAY,cACZ,UAAU,EAAE,GACV,YAAY,CA2FxB;AAED;;;;;;;GAOG;AACH,8CALW,GAAG,CAAC,MAAM,EAAE,YAAY,CAAC,UACzB,UAAU,WACV,cAAc,UACd,MAAM,QA2EhB;kCAG0D,iBAAiB;gCAAjB,iBAAiB;oCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/bloom.d.ts b/web/node_modules/hyparquet/types/bloom.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..dcc9d7d2ed3544db0ac35f2eeda0041288c0f963 --- /dev/null +++ b/web/node_modules/hyparquet/types/bloom.d.ts @@ -0,0 +1,51 @@ +/** + * Insert a hash into a Split Block Bloom Filter. + * + * @param {Uint32Array} blocks bloom filter words (8 * numBlocks long) + * @param {bigint} hash 64-bit xxhash of the parquet-plain-encoded value + */ +export function sbbfInsert(blocks: Uint32Array, hash: bigint): void; +/** + * Test whether a hash might be present in a Split Block Bloom Filter. + * False positives are possible; false negatives are not. + * + * @param {Uint32Array} blocks bloom filter words (8 * numBlocks long) + * @param {bigint} hash 64-bit xxhash of the parquet-plain-encoded value + * @returns {boolean} + */ +export function sbbfContains(blocks: Uint32Array, hash: bigint): boolean; +/** + * Parse a Split Block Bloom Filter from a reader positioned at the BloomFilterHeader. + * Returns undefined when the header advertises an unsupported algorithm, hash, or + * compression — callers should treat that as "cannot use this bloom filter." + * + * @param {DataReader} reader + * @returns {BloomFilter | undefined} + */ +export function readBloomFilter(reader: DataReader): BloomFilter | undefined; +/** + * Hash a JS filter value as its parquet PLAIN-encoded bytes, suitable for a + * bloom filter lookup. Returns undefined when the column's parser is lossy or + * ambiguous (DATE, TIMESTAMP_*, DECIMAL, JSON, BSON, INT96, FLOAT16, UUID, + * GEOMETRY, GEOGRAPHY, INTERVAL) or when the JS value type doesn't match the + * column. Callers must treat undefined as "bloom filter cannot help." + * + * @param {any} value + * @param {SchemaElement} element + * @returns {bigint | undefined} + */ +export function hashParquetValue(value: any, element: SchemaElement): bigint | undefined; +/** + * Top-level column names that appear in $eq or $in predicates within a filter. + * These are the only columns where a bloom filter can prove a value's absence + * and let us skip a row group; any other operator can't be helped by a bloom. + * + * @param {ParquetQueryFilter | undefined} filter + * @returns {Set} + */ +export function bloomEligibleColumns(filter: ParquetQueryFilter | undefined): Set; +import type { DataReader } from '../src/types.js'; +import type { BloomFilter } from '../src/types.js'; +import type { SchemaElement } from '../src/types.js'; +import type { ParquetQueryFilter } from '../src/types.js'; +//# sourceMappingURL=bloom.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/bloom.d.ts.map b/web/node_modules/hyparquet/types/bloom.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..f88f37a38d6227415cd1957a97585c7f157df511 --- /dev/null +++ b/web/node_modules/hyparquet/types/bloom.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"bloom.d.ts","sourceRoot":"","sources":["../src/bloom.js"],"names":[],"mappings":"AA6CA;;;;;GAKG;AACH,mCAHW,WAAW,QACX,MAAM,QAQhB;AAED;;;;;;;GAOG;AACH,qCAJW,WAAW,QACX,MAAM,GACJ,OAAO,CASnB;AAED;;;;;;;GAOG;AACH,wCAHW,UAAU,GACR,WAAW,GAAG,SAAS,CAsBnC;AAED;;;;;;;;;;GAUG;AACH,wCAJW,GAAG,WACH,aAAa,GACX,MAAM,GAAG,SAAS,CAkE9B;AAED;;;;;;;GAOG;AACH,6CAHW,kBAAkB,GAAG,SAAS,GAC5B,GAAG,CAAC,MAAM,CAAC,CAOvB;gCA3L4E,iBAAiB;iCAAjB,iBAAiB;mCAAjB,iBAAiB;wCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/column.d.ts b/web/node_modules/hyparquet/types/column.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..52b31582f59d30e6d257c5a94eb9b2ad559a22f3 --- /dev/null +++ b/web/node_modules/hyparquet/types/column.d.ts @@ -0,0 +1,33 @@ +/** + * Parse column data from a buffer. + * + * @param {DataReader} reader + * @param {RowGroupSelect} rowGroupSelect row group selection + * @param {ColumnDecoder} columnDecoder column decoder params + * @param {(chunk: SubColumnData) => void} [onPage] callback for each page + * @returns {{ data: DecodedArray[], skipped: number }} + */ +export function readColumn(reader: DataReader, { groupStart, selectStart, selectEnd }: RowGroupSelect, columnDecoder: ColumnDecoder, onPage?: (chunk: SubColumnData) => void): { + data: DecodedArray[]; + skipped: number; +}; +/** + * Read a page (data or dictionary) from a buffer. + * + * @param {DataReader} reader + * @param {PageHeader} header + * @param {ColumnDecoder} columnDecoder + * @param {DecodedArray | undefined} dictionary + * @param {DecodedArray | undefined} previousChunk + * @param {number} pageStart skip this many rows in the page + * @returns {PageResult} + */ +export function readPage(reader: DataReader, header: PageHeader, columnDecoder: ColumnDecoder, dictionary: DecodedArray | undefined, previousChunk: DecodedArray | undefined, pageStart: number): PageResult; +import type { DataReader } from '../src/types.js'; +import type { RowGroupSelect } from '../src/types.js'; +import type { ColumnDecoder } from '../src/types.js'; +import type { SubColumnData } from '../src/types.js'; +import type { DecodedArray } from '../src/types.js'; +import type { PageHeader } from '../src/types.js'; +import type { PageResult } from '../src/types.js'; +//# sourceMappingURL=column.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/column.d.ts.map b/web/node_modules/hyparquet/types/column.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..f10caed4d8ec3a80c57baf86f7f41fb8463a318d --- /dev/null +++ b/web/node_modules/hyparquet/types/column.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"column.d.ts","sourceRoot":"","sources":["../src/column.js"],"names":[],"mappings":"AAYA;;;;;;;;GAQG;AACH,mCANW,UAAU,0CACV,cAAc,iBACd,aAAa,WACb,CAAC,KAAK,EAAE,aAAa,KAAK,IAAI,GAC5B;IAAE,IAAI,EAAE,YAAY,EAAE,CAAC;IAAC,OAAO,EAAE,MAAM,CAAA;CAAE,CAsDrD;AAED;;;;;;;;;;GAUG;AACH,iCARW,UAAU,UACV,UAAU,iBACV,aAAa,cACb,YAAY,GAAG,SAAS,iBACxB,YAAY,GAAG,SAAS,aACxB,MAAM,GACJ,UAAU,CA4DtB;gCA/IgH,iBAAiB;oCAAjB,iBAAiB;mCAAjB,iBAAiB;mCAAjB,iBAAiB;kCAAjB,iBAAiB;gCAAjB,iBAAiB;gCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/constants.d.ts b/web/node_modules/hyparquet/types/constants.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..e0dbc46a9bf7a92eb9ba5ae4e983cf569a6818d0 --- /dev/null +++ b/web/node_modules/hyparquet/types/constants.d.ts @@ -0,0 +1,17 @@ +/** @type {import('../src/types.d.ts').ParquetType[]} */ +export const ParquetTypes: import("../src/types.d.ts").ParquetType[]; +/** @type {import('../src/types.d.ts').Encoding[]} */ +export const Encodings: import("../src/types.d.ts").Encoding[]; +/** @type {import('../src/types.d.ts').FieldRepetitionType[]} */ +export const FieldRepetitionTypes: import("../src/types.d.ts").FieldRepetitionType[]; +/** @type {import('../src/types.d.ts').ConvertedType[]} */ +export const ConvertedTypes: import("../src/types.d.ts").ConvertedType[]; +/** @type {import('../src/types.d.ts').CompressionCodec[]} */ +export const CompressionCodecs: import("../src/types.d.ts").CompressionCodec[]; +/** @type {import('../src/types.d.ts').PageType[]} */ +export const PageTypes: import("../src/types.d.ts").PageType[]; +/** @type {import('../src/types.d.ts').BoundaryOrder[]} */ +export const BoundaryOrders: import("../src/types.d.ts").BoundaryOrder[]; +/** @type {import('../src/types.d.ts').EdgeInterpolationAlgorithm[]} */ +export const EdgeInterpolationAlgorithms: import("../src/types.d.ts").EdgeInterpolationAlgorithm[]; +//# sourceMappingURL=constants.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/constants.d.ts.map b/web/node_modules/hyparquet/types/constants.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..f7a4dcb074eb502ff85e10c8ebc9b164e54eebcd --- /dev/null +++ b/web/node_modules/hyparquet/types/constants.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"constants.d.ts","sourceRoot":"","sources":["../src/constants.js"],"names":[],"mappings":"AACA,wDAAwD;AACxD,2BADW,OAAO,mBAAmB,EAAE,WAAW,EAAE,CAUnD;AAED,qDAAqD;AACrD,wBADW,OAAO,mBAAmB,EAAE,QAAQ,EAAE,CAYhD;AAED,gEAAgE;AAChE,mCADW,OAAO,mBAAmB,EAAE,mBAAmB,EAAE,CAK3D;AAED,0DAA0D;AAC1D,6BADW,OAAO,mBAAmB,EAAE,aAAa,EAAE,CAwBrD;AAED,6DAA6D;AAC7D,gCADW,OAAO,mBAAmB,EAAE,gBAAgB,EAAE,CAUxD;AAED,qDAAqD;AACrD,wBADW,OAAO,mBAAmB,EAAE,QAAQ,EAAE,CAMhD;AAED,0DAA0D;AAC1D,6BADW,OAAO,mBAAmB,EAAE,aAAa,EAAE,CAKrD;AAED,uEAAuE;AACvE,0CADW,OAAO,mBAAmB,EAAE,0BAA0B,EAAE,CAOlE"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/convert.d.ts b/web/node_modules/hyparquet/types/convert.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..43ca5700e227bf34ec27de710edfbf87469c2a3a --- /dev/null +++ b/web/node_modules/hyparquet/types/convert.d.ts @@ -0,0 +1,38 @@ +/** + * Convert known types from primitive to rich, and dereference dictionary. + * + * @param {DecodedArray} data series of primitive types + * @param {DecodedArray | undefined} dictionary + * @param {Encoding} encoding + * @param {ColumnDecoder} columnDecoder + * @returns {DecodedArray} series of rich types + */ +export function convertWithDictionary(data: DecodedArray, dictionary: DecodedArray | undefined, encoding: Encoding, columnDecoder: ColumnDecoder): DecodedArray; +/** + * Convert known types from primitive to rich. + * + * @param {DecodedArray} data series of primitive types + * @param {ColumnDecoder} columnDecoder + * @returns {DecodedArray} series of rich types + */ +export function convert(data: DecodedArray, columnDecoder: ColumnDecoder): DecodedArray; +/** + * @param {Uint8Array} bytes + * @returns {number} + */ +export function parseDecimal(bytes: Uint8Array): number; +/** + * @param {Uint8Array | undefined} bytes + * @returns {number | undefined} + */ +export function parseFloat16(bytes: Uint8Array | undefined): number | undefined; +/** + * Default type parsers when no custom ones are given + * @type ParquetParsers + */ +export const DEFAULT_PARSERS: ParquetParsers; +import type { DecodedArray } from '../src/types.js'; +import type { Encoding } from '../src/types.js'; +import type { ColumnDecoder } from '../src/types.js'; +import type { ParquetParsers } from '../src/types.js'; +//# sourceMappingURL=convert.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/convert.d.ts.map b/web/node_modules/hyparquet/types/convert.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..d9f64f6374c1b36b3a11d6e99d02c36fdd5bc520 --- /dev/null +++ b/web/node_modules/hyparquet/types/convert.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"convert.d.ts","sourceRoot":"","sources":["../src/convert.js"],"names":[],"mappings":"AA4CA;;;;;;;;GAQG;AACH,4CANW,YAAY,cACZ,YAAY,GAAG,SAAS,YACxB,QAAQ,iBACR,aAAa,GACX,YAAY,CAgBxB;AAED;;;;;;GAMG;AACH,8BAJW,YAAY,iBACZ,aAAa,GACX,YAAY,CAwFxB;AAED;;;GAGG;AACH,oCAHW,UAAU,GACR,MAAM,CAiBlB;AAaD;;;GAGG;AACH,oCAHW,UAAU,GAAG,SAAS,GACpB,MAAM,GAAG,SAAS,CAW9B;AAzMD;;;GAGG;AACH,8BAFS,cAAc,CAgCtB;kCAvCuE,iBAAiB;8BAAjB,iBAAiB;mCAAjB,iBAAiB;oCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/datapage.d.ts b/web/node_modules/hyparquet/types/datapage.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..1e011b94d9752b34e3540646dfff0fd69a99b3ba --- /dev/null +++ b/web/node_modules/hyparquet/types/datapage.d.ts @@ -0,0 +1,33 @@ +/** + * Read a data page from uncompressed reader. + * + * @param {Uint8Array} bytes raw page data (should already be decompressed) + * @param {DataPageHeader} daph data page header + * @param {ColumnDecoder} columnDecoder + * @returns {DataPage} definition levels, repetition levels, and array of values + */ +export function readDataPage(bytes: Uint8Array, daph: DataPageHeader, { type, element, schemaPath }: ColumnDecoder): DataPage; +/** + * @param {Uint8Array} compressedBytes + * @param {number} uncompressed_page_size + * @param {CompressionCodec} codec + * @param {Compressors | undefined} compressors + * @returns {Uint8Array} + */ +export function decompressPage(compressedBytes: Uint8Array, uncompressed_page_size: number, codec: CompressionCodec, compressors: Compressors | undefined): Uint8Array; +/** + * Read a data page from the given Uint8Array. + * + * @param {Uint8Array} compressedBytes raw page data + * @param {PageHeader} ph page header + * @param {ColumnDecoder} columnDecoder + * @returns {DataPage} definition levels, repetition levels, and array of values + */ +export function readDataPageV2(compressedBytes: Uint8Array, ph: PageHeader, columnDecoder: ColumnDecoder): DataPage; +import type { DataPageHeader } from '../src/types.js'; +import type { ColumnDecoder } from '../src/types.js'; +import type { DataPage } from '../src/types.js'; +import type { CompressionCodec } from '../src/types.js'; +import type { Compressors } from '../src/types.js'; +import type { PageHeader } from '../src/types.js'; +//# sourceMappingURL=datapage.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/datapage.d.ts.map b/web/node_modules/hyparquet/types/datapage.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..84fa0dbe23aef1af051281fb743c00bd839b7447 --- /dev/null +++ b/web/node_modules/hyparquet/types/datapage.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"datapage.d.ts","sourceRoot":"","sources":["../src/datapage.js"],"names":[],"mappings":"AAUA;;;;;;;GAOG;AACH,oCALW,UAAU,QACV,cAAc,iCACd,aAAa,GACX,QAAQ,CAkDpB;AA2CD;;;;;;GAMG;AACH,gDANW,UAAU,0BACV,MAAM,SACN,gBAAgB,eAChB,WAAW,GAAG,SAAS,GACrB,UAAU,CAoBtB;AAGD;;;;;;;GAOG;AACH,gDALW,UAAU,MACV,UAAU,iBACV,aAAa,GACX,QAAQ,CA6DpB;oCA3M4J,iBAAiB;mCAAjB,iBAAiB;8BAAjB,iBAAiB;sCAAjB,iBAAiB;iCAAjB,iBAAiB;gCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/delta.d.ts b/web/node_modules/hyparquet/types/delta.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..51aa5541ef3b4d796790b82174faa103c92ae637 --- /dev/null +++ b/web/node_modules/hyparquet/types/delta.d.ts @@ -0,0 +1,20 @@ +/** + * @param {DataReader} reader + * @param {number} count number of values to read + * @param {Int32Array | BigInt64Array} output + */ +export function deltaBinaryUnpack(reader: DataReader, count: number, output: Int32Array | BigInt64Array): void; +/** + * @param {DataReader} reader + * @param {number} count + * @param {Uint8Array[]} output + */ +export function deltaLengthByteArray(reader: DataReader, count: number, output: Uint8Array[]): void; +/** + * @param {DataReader} reader + * @param {number} count + * @param {Uint8Array[]} output + */ +export function deltaByteArray(reader: DataReader, count: number, output: Uint8Array[]): void; +import type { DataReader } from '../src/types.js'; +//# sourceMappingURL=delta.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/delta.d.ts.map b/web/node_modules/hyparquet/types/delta.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..1d420980cfd26e06f9e4cce4d61c0bead405d459 --- /dev/null +++ b/web/node_modules/hyparquet/types/delta.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"delta.d.ts","sourceRoot":"","sources":["../src/delta.js"],"names":[],"mappings":"AAMA;;;;GAIG;AACH,0CAJW,UAAU,SACV,MAAM,UACN,UAAU,GAAG,aAAa,QAuDpC;AAED;;;;GAIG;AACH,6CAJW,UAAU,SACV,MAAM,UACN,UAAU,EAAE,QAStB;AAED;;;;GAIG;AACH,uCAJW,UAAU,SACV,MAAM,UACN,UAAU,EAAE,QAoBtB;gCAtG4B,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/encoding.d.ts b/web/node_modules/hyparquet/types/encoding.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..ecca69c971828138e12efb8d70ddb7184df3c16b --- /dev/null +++ b/web/node_modules/hyparquet/types/encoding.d.ts @@ -0,0 +1,23 @@ +/** + * Read values from a run-length encoded/bit-packed hybrid encoding. + * + * If length is zero, then read int32 length at the start. + * + * @param {DataReader} reader + * @param {number} width - bitwidth + * @param {DecodedArray} output + * @param {number} [length] - length of the encoded data + */ +export function readRleBitPackedHybrid(reader: DataReader, width: number, output: DecodedArray, length?: number): void; +/** + * @param {DataReader} reader + * @param {number} count + * @param {ParquetType} type + * @param {number | undefined} typeLength + * @returns {DecodedArray} + */ +export function byteStreamSplit(reader: DataReader, count: number, type: ParquetType, typeLength: number | undefined): DecodedArray; +import type { DataReader } from '../src/types.js'; +import type { DecodedArray } from '../src/types.js'; +import type { ParquetType } from '../src/types.js'; +//# sourceMappingURL=encoding.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/encoding.d.ts.map b/web/node_modules/hyparquet/types/encoding.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..708a6432dd5aed760ac82a024294a9b891045f83 --- /dev/null +++ b/web/node_modules/hyparquet/types/encoding.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"encoding.d.ts","sourceRoot":"","sources":["../src/encoding.js"],"names":[],"mappings":"AAMA;;;;;;;;;GASG;AACH,+CALW,UAAU,SACV,MAAM,UACN,YAAY,WACZ,MAAM,QAsBhB;AA2ED;;;;;;GAMG;AACH,wCANW,UAAU,SACV,MAAM,QACN,WAAW,cACX,MAAM,GAAG,SAAS,GAChB,YAAY,CAwBxB;gCA3IuD,iBAAiB;kCAAjB,iBAAiB;iCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/filter.d.ts b/web/node_modules/hyparquet/types/filter.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..739d425e31f49cd0427b0424b6c731202b84daab --- /dev/null +++ b/web/node_modules/hyparquet/types/filter.d.ts @@ -0,0 +1,43 @@ +/** + * Returns an array of top-level column names needed to evaluate the filter. + * + * @param {ParquetQueryFilter} [filter] + * @returns {string[]} + */ +export function columnsNeededForFilter(filter?: ParquetQueryFilter): string[]; +/** + * Match a record against a query filter + * + * @param {Record} record + * @param {ParquetQueryFilter} filter + * @param {boolean} [strict] + * @returns {boolean} + */ +export function matchFilter(record: Record, filter: ParquetQueryFilter, strict?: boolean): boolean; +/** + * Check if a row group can be skipped based on filter and column statistics, + * optionally consulting per-column bloom filters for equality predicates that + * statistics can't decide. + * + * @param {object} options + * @param {RowGroup} options.rowGroup + * @param {string[]} options.physicalColumns + * @param {ParquetQueryFilter | undefined} options.filter + * @param {boolean} [options.strict] + * @param {Record} [options.bloomFilters] keyed by top-level column name + * @param {Record} [options.schemaElements] keyed by top-level column name, required to use bloomFilters + * @returns {boolean} true if the row group can be skipped + */ +export function canSkipRowGroup({ rowGroup, physicalColumns, filter, strict, bloomFilters, schemaElements }: { + rowGroup: RowGroup; + physicalColumns: string[]; + filter: ParquetQueryFilter | undefined; + strict?: boolean | undefined; + bloomFilters?: Record | undefined; + schemaElements?: Record | undefined; +}): boolean; +import type { ParquetQueryFilter } from '../src/types.js'; +import type { RowGroup } from '../src/types.js'; +import type { BloomFilter } from '../src/types.js'; +import type { SchemaElement } from '../src/types.js'; +//# sourceMappingURL=filter.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/filter.d.ts.map b/web/node_modules/hyparquet/types/filter.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..d1c8259f89cb15513c7ecb060395efb54bc7f72d --- /dev/null +++ b/web/node_modules/hyparquet/types/filter.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"filter.d.ts","sourceRoot":"","sources":["../src/filter.js"],"names":[],"mappings":"AAOA;;;;;GAKG;AACH,gDAHW,kBAAkB,GAChB,MAAM,EAAE,CAiBpB;AAED;;;;;;;GAOG;AACH,oCALW,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,UACnB,kBAAkB,WAClB,OAAO,GACL,OAAO,CAkCnB;AAED;;;;;;;;;;;;;GAaG;AACH,6GARG;IAA0B,QAAQ,EAA1B,QAAQ;IACU,eAAe,EAAjC,MAAM,EAAE;IACgC,MAAM,EAA9C,kBAAkB,GAAG,SAAS;IACZ,MAAM;IACc,YAAY;IACV,cAAc;CAC9D,GAAU,OAAO,CAsEnB;wCAzJ0E,iBAAiB;8BAAjB,iBAAiB;iCAAjB,iBAAiB;mCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/geoparquet.d.ts b/web/node_modules/hyparquet/types/geoparquet.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..fdd1134c7d94634c8a62f337fd8260a30aab50d6 --- /dev/null +++ b/web/node_modules/hyparquet/types/geoparquet.d.ts @@ -0,0 +1,9 @@ +/** + * @param {SchemaElement[]} schema + * @param {KeyValue[] | undefined} key_value_metadata + * @returns {void} + */ +export function markGeoColumns(schema: SchemaElement[], key_value_metadata: KeyValue[] | undefined): void; +import type { SchemaElement } from '../src/types.js'; +import type { KeyValue } from '../src/types.js'; +//# sourceMappingURL=geoparquet.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/geoparquet.d.ts.map b/web/node_modules/hyparquet/types/geoparquet.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..b1c50d161c98d83b84aa262875a0a3152a24ade5 --- /dev/null +++ b/web/node_modules/hyparquet/types/geoparquet.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"geoparquet.d.ts","sourceRoot":"","sources":["../src/geoparquet.js"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,uCAJW,aAAa,EAAE,sBACf,QAAQ,EAAE,GAAG,SAAS,GACpB,IAAI,CA8BhB;mCAGsD,iBAAiB;8BAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/index.d.ts b/web/node_modules/hyparquet/types/index.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..a10daf5d22ffd609817a034a3e83854ddbc6154a --- /dev/null +++ b/web/node_modules/hyparquet/types/index.d.ts @@ -0,0 +1,46 @@ +export { parquetQuery } from "./query.js"; +export { snappyUncompress } from "./snappy.js"; +/** + * + */ +export type Awaitable = import("../src/types.d.ts").Awaitable; +export type AsyncBuffer = import("../src/types.d.ts").AsyncBuffer; +export type AsyncRowGroup = import("../src/types.d.ts").AsyncRowGroup; +export type DataReader = import("../src/types.d.ts").DataReader; +export type FileMetaData = import("../src/types.d.ts").FileMetaData; +export type SchemaTree = import("../src/types.d.ts").SchemaTree; +export type SchemaElement = import("../src/types.d.ts").SchemaElement; +export type ParquetType = import("../src/types.d.ts").ParquetType; +export type FieldRepetitionType = import("../src/types.d.ts").FieldRepetitionType; +export type ConvertedType = import("../src/types.d.ts").ConvertedType; +export type TimeUnit = import("../src/types.d.ts").TimeUnit; +export type LogicalType = import("../src/types.d.ts").LogicalType; +export type RowGroup = import("../src/types.d.ts").RowGroup; +export type ColumnChunk = import("../src/types.d.ts").ColumnChunk; +export type ColumnMetaData = import("../src/types.d.ts").ColumnMetaData; +export type Encoding = import("../src/types.d.ts").Encoding; +export type CompressionCodec = import("../src/types.d.ts").CompressionCodec; +export type Compressors = import("../src/types.d.ts").Compressors; +export type KeyValue = import("../src/types.d.ts").KeyValue; +export type Statistics = import("../src/types.d.ts").Statistics; +export type GeospatialStatistics = import("../src/types.d.ts").GeospatialStatistics; +export type BoundingBox = import("../src/types.d.ts").BoundingBox; +export type PageType = import("../src/types.d.ts").PageType; +export type PageHeader = import("../src/types.d.ts").PageHeader; +export type DataPageHeader = import("../src/types.d.ts").DataPageHeader; +export type DictionaryPageHeader = import("../src/types.d.ts").DictionaryPageHeader; +export type DecodedArray = import("../src/types.d.ts").DecodedArray; +export type OffsetIndex = import("../src/types.d.ts").OffsetIndex; +export type ColumnIndex = import("../src/types.d.ts").ColumnIndex; +export type BoundaryOrder = import("../src/types.d.ts").BoundaryOrder; +export type ColumnData = import("../src/types.d.ts").ColumnData; +export type SubColumnData = import("../src/types.d.ts").SubColumnData; +export type ParquetReadOptions = import("../src/types.d.ts").ParquetReadOptions; +export type MetadataOptions = import("../src/types.d.ts").MetadataOptions; +export type ParquetParsers = import("../src/types.d.ts").ParquetParsers; +export type ParquetQueryFilter = import("../src/types.d.ts").ParquetQueryFilter; +export { readColumnIndex, readOffsetIndex } from "./indexes.js"; +export { parquetMetadata, parquetMetadataAsync, parquetSchema } from "./metadata.js"; +export { parquetRead, parquetReadObjects } from "./read.js"; +export { asyncBufferFromUrl, byteLengthFromUrl, cachedAsyncBuffer, flatten, toJson } from "./utils.js"; +//# sourceMappingURL=index.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/index.d.ts.map b/web/node_modules/hyparquet/types/index.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..142c2b691f6fafc574cdf6e4935c2138e4a958d9 --- /dev/null +++ b/web/node_modules/hyparquet/types/index.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.js"],"names":[],"mappings":";;;;;sBAWmB,CAAC,oBACP,OAAO,mBAAmB,EAAE,SAAS,CAAC,CAAC,CAAC;0BAGxC,OAAO,mBAAmB,EAAE,WAAW;4BACvC,OAAO,mBAAmB,EAAE,aAAa;yBACzC,OAAO,mBAAmB,EAAE,UAAU;2BACtC,OAAO,mBAAmB,EAAE,YAAY;yBACxC,OAAO,mBAAmB,EAAE,UAAU;4BACtC,OAAO,mBAAmB,EAAE,aAAa;0BACzC,OAAO,mBAAmB,EAAE,WAAW;kCACvC,OAAO,mBAAmB,EAAE,mBAAmB;4BAC/C,OAAO,mBAAmB,EAAE,aAAa;uBACzC,OAAO,mBAAmB,EAAE,QAAQ;0BACpC,OAAO,mBAAmB,EAAE,WAAW;uBACvC,OAAO,mBAAmB,EAAE,QAAQ;0BACpC,OAAO,mBAAmB,EAAE,WAAW;6BACvC,OAAO,mBAAmB,EAAE,cAAc;uBAC1C,OAAO,mBAAmB,EAAE,QAAQ;+BACpC,OAAO,mBAAmB,EAAE,gBAAgB;0BAC5C,OAAO,mBAAmB,EAAE,WAAW;uBACvC,OAAO,mBAAmB,EAAE,QAAQ;yBACpC,OAAO,mBAAmB,EAAE,UAAU;mCACtC,OAAO,mBAAmB,EAAE,oBAAoB;0BAChD,OAAO,mBAAmB,EAAE,WAAW;uBACvC,OAAO,mBAAmB,EAAE,QAAQ;yBACpC,OAAO,mBAAmB,EAAE,UAAU;6BACtC,OAAO,mBAAmB,EAAE,cAAc;mCAC1C,OAAO,mBAAmB,EAAE,oBAAoB;2BAChD,OAAO,mBAAmB,EAAE,YAAY;0BACxC,OAAO,mBAAmB,EAAE,WAAW;0BACvC,OAAO,mBAAmB,EAAE,WAAW;4BACvC,OAAO,mBAAmB,EAAE,aAAa;yBACzC,OAAO,mBAAmB,EAAE,UAAU;4BACtC,OAAO,mBAAmB,EAAE,aAAa;iCACzC,OAAO,mBAAmB,EAAE,kBAAkB;8BAC9C,OAAO,mBAAmB,EAAE,eAAe;6BAC3C,OAAO,mBAAmB,EAAE,cAAc;iCAC1C,OAAO,mBAAmB,EAAE,kBAAkB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/indexes.d.ts b/web/node_modules/hyparquet/types/indexes.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..5cb041af997bafd8f71d075b2d59c453987695e0 --- /dev/null +++ b/web/node_modules/hyparquet/types/indexes.d.ts @@ -0,0 +1,18 @@ +/** + * @param {DataReader} reader + * @param {SchemaElement} schema + * @param {ParquetParsers | undefined} parsers + * @returns {ColumnIndex} + */ +export function readColumnIndex(reader: DataReader, schema: SchemaElement, parsers?: ParquetParsers | undefined): ColumnIndex; +/** + * @param {DataReader} reader + * @returns {OffsetIndex} + */ +export function readOffsetIndex(reader: DataReader): OffsetIndex; +import type { DataReader } from '../src/types.js'; +import type { SchemaElement } from '../src/types.js'; +import type { ParquetParsers } from '../src/types.js'; +import type { ColumnIndex } from '../src/types.js'; +import type { OffsetIndex } from '../src/types.js'; +//# sourceMappingURL=indexes.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/indexes.d.ts.map b/web/node_modules/hyparquet/types/indexes.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..eb321048d58870f1ede5d31d5786a8a8e0f0f7e8 --- /dev/null +++ b/web/node_modules/hyparquet/types/indexes.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"indexes.d.ts","sourceRoot":"","sources":["../src/indexes.js"],"names":[],"mappings":"AASA;;;;;GAKG;AACH,wCALW,UAAU,UACV,aAAa,YACb,cAAc,GAAG,SAAS,GACxB,WAAW,CAevB;AAED;;;GAGG;AACH,wCAHW,UAAU,GACR,WAAW,CAavB;gCA5CmG,iBAAiB;mCAAjB,iBAAiB;oCAAjB,iBAAiB;iCAAjB,iBAAiB;iCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/metadata.d.ts b/web/node_modules/hyparquet/types/metadata.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..5ea7c4f4600f864576a611973a86194951d53431 --- /dev/null +++ b/web/node_modules/hyparquet/types/metadata.d.ts @@ -0,0 +1,63 @@ +/** + * Read parquet metadata from an async buffer. + * + * An AsyncBuffer is like an ArrayBuffer, but the slices are loaded + * asynchronously, possibly over the network. + * + * You must provide the byteLength of the buffer, typically from a HEAD request. + * + * In theory, you could use suffix-range requests to fetch the end of the file, + * and save a round trip. But in practice, this doesn't work because chrome + * deems suffix-range requests as a not-safe-listed header, and will require + * a pre-flight. So the byteLength is required. + * + * To make this efficient, we initially request the last 512kb of the file, + * which is likely to contain the metadata. If the metadata length exceeds the + * initial fetch, 512kb, we request the rest of the metadata from the AsyncBuffer. + * + * This ensures that we either make one 512kb initial request for the metadata, + * or a second request for up to the metadata size. + * + * @param {AsyncBuffer} asyncBuffer parquet file contents + * @param {MetadataOptions & { initialFetchSize?: number }} options initial fetch size in bytes (default 512kb) + * @returns {Promise} parquet metadata object + */ +export function parquetMetadataAsync(asyncBuffer: AsyncBuffer, { parsers, initialFetchSize, geoparquet }?: MetadataOptions & { + initialFetchSize?: number; +}): Promise; +/** + * Read parquet metadata from a buffer synchronously. + * + * @param {ArrayBuffer} arrayBuffer parquet file footer + * @param {MetadataOptions} options metadata parsing options + * @returns {FileMetaData} parquet metadata object + */ +export function parquetMetadata(arrayBuffer: ArrayBuffer, { parsers, geoparquet }?: MetadataOptions): FileMetaData; +/** + * Return a tree of schema elements from parquet metadata. + * + * @param {{schema: SchemaElement[]}} metadata parquet metadata object + * @returns {SchemaTree} tree of schema elements + */ +export function parquetSchema({ schema }: { + schema: SchemaElement[]; +}): SchemaTree; +/** + * @param {Uint8Array | undefined} value + * @param {SchemaElement} schema + * @param {ParquetParsers} parsers + * @returns {MinMaxType | undefined} + */ +export function convertMetadata(value: Uint8Array | undefined, schema: SchemaElement, parsers: ParquetParsers): MinMaxType | undefined; +/** + * @import {AsyncBuffer, FileMetaData, KeyValue, LogicalType, MetadataOptions, MinMaxType, ParquetParsers, SchemaElement, SchemaTree, Statistics, TimeUnit} from '../src/types.js' + */ +export const defaultInitialFetchSize: number; +import type { AsyncBuffer } from '../src/types.js'; +import type { MetadataOptions } from '../src/types.js'; +import type { FileMetaData } from '../src/types.js'; +import type { SchemaElement } from '../src/types.js'; +import type { SchemaTree } from '../src/types.js'; +import type { ParquetParsers } from '../src/types.js'; +import type { MinMaxType } from '../src/types.js'; +//# sourceMappingURL=metadata.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/metadata.d.ts.map b/web/node_modules/hyparquet/types/metadata.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..38e60521bf1c3a03606c7947ccfd885827803786 --- /dev/null +++ b/web/node_modules/hyparquet/types/metadata.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"metadata.d.ts","sourceRoot":"","sources":["../src/metadata.js"],"names":[],"mappings":"AAiBA;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,kDAJW,WAAW,8CACX,eAAe,GAAG;IAAE,gBAAgB,CAAC,EAAE,MAAM,CAAA;CAAE,GAC7C,OAAO,CAAC,YAAY,CAAC,CAqCjC;AAED;;;;;;GAMG;AACH,6CAJW,WAAW,4BACX,eAAe,GACb,YAAY,CAoIxB;AAED;;;;;GAKG;AACH,0CAHW;IAAC,MAAM,EAAE,aAAa,EAAE,CAAA;CAAC,GACvB,UAAU,CAItB;AAqFD;;;;;GAKG;AACH,uCALW,UAAU,GAAG,SAAS,UACtB,aAAa,WACb,cAAc,GACZ,UAAU,GAAG,SAAS,CAwBlC;AA5UD;;GAEG;AAEH,6CAA8C;iCAHmH,iBAAiB;qCAAjB,iBAAiB;kCAAjB,iBAAiB;mCAAjB,iBAAiB;gCAAjB,iBAAiB;oCAAjB,iBAAiB;gCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/node.d.ts b/web/node_modules/hyparquet/types/node.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..d098dc653a1e1346a0f617caa91b178478460525 --- /dev/null +++ b/web/node_modules/hyparquet/types/node.d.ts @@ -0,0 +1,10 @@ +/** + * Construct an AsyncBuffer for a local file using node fs package. + * + * @param {string} filename + * @returns {Promise} + */ +export function asyncBufferFromFile(filename: string): Promise; +export * from "./index.js"; +import type { AsyncBuffer } from '../src/types.js'; +//# sourceMappingURL=node.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/node.d.ts.map b/web/node_modules/hyparquet/types/node.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..4a4e185fb1f39c3e9ea02680c1102e413c0b282c --- /dev/null +++ b/web/node_modules/hyparquet/types/node.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"node.d.ts","sourceRoot":"","sources":["../src/node.js"],"names":[],"mappings":"AAQA;;;;;GAKG;AACH,8CAHW,MAAM,GACJ,OAAO,CAAC,WAAW,CAAC,CAqBhC;;iCAhC6B,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/plain.d.ts b/web/node_modules/hyparquet/types/plain.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..1c19067a8eb9dbc67c35c4f2e3eb560ccc755495 --- /dev/null +++ b/web/node_modules/hyparquet/types/plain.d.ts @@ -0,0 +1,14 @@ +/** + * Read `count` values of the given type from the reader.view. + * + * @param {DataReader} reader - buffer to read data from + * @param {ParquetType} type - parquet type of the data + * @param {number} count - number of values to read + * @param {number | undefined} fixedLength - length of each fixed length byte array + * @returns {DecodedArray} array of values + */ +export function readPlain(reader: DataReader, type: ParquetType, count: number, fixedLength: number | undefined): DecodedArray; +import type { DataReader } from '../src/types.js'; +import type { ParquetType } from '../src/types.js'; +import type { DecodedArray } from '../src/types.js'; +//# sourceMappingURL=plain.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/plain.d.ts.map b/web/node_modules/hyparquet/types/plain.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..51f44012b2d97cff583d6e6c68994b116e6e987e --- /dev/null +++ b/web/node_modules/hyparquet/types/plain.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"plain.d.ts","sourceRoot":"","sources":["../src/plain.js"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AACH,kCANW,UAAU,QACV,WAAW,SACX,MAAM,eACN,MAAM,GAAG,SAAS,GAChB,YAAY,CAwBxB;gCA0IuD,iBAAiB;iCAAjB,iBAAiB;kCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/plan.d.ts b/web/node_modules/hyparquet/types/plan.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..46da72fe3644cd83676f09cd503aa21e05ba5a6c --- /dev/null +++ b/web/node_modules/hyparquet/types/plan.d.ts @@ -0,0 +1,47 @@ +/** + * Plan which byte ranges to read to satisfy a read request. + * Metadata must be non-null. + * + * @param {ParquetReadOptions & { bloomFiltersByGroup?: Record[], schemaElements?: Record }} options + * @returns {QueryPlan} + */ +export function parquetPlan({ metadata, rowStart, rowEnd, columns, filter, filterStrict, useOffsetIndex, bloomFiltersByGroup, schemaElements }: ParquetReadOptions & { + bloomFiltersByGroup?: Record[]; + schemaElements?: Record; +}): QueryPlan; +/** + * Fetch bloom filters for $eq / $in columns of row groups not already provably + * skippable by statistics alone. Returns an array indexed by row-group ordinal; + * each entry maps top-level column name → BloomFilter for any chunk whose + * bloom filter we were able to parse. Adds one round-trip when at least one + * bloom filter is fetched; otherwise returns synchronously. + * + * @param {object} options + * @param {AsyncBuffer} options.file + * @param {FileMetaData} options.metadata + * @param {ParquetQueryFilter} options.filter + * @param {boolean} [options.filterStrict] + * @returns {Promise[]>} + */ +export function prefetchBloomFilters({ file, metadata, filter, filterStrict }: { + file: AsyncBuffer; + metadata: FileMetaData; + filter: ParquetQueryFilter; + filterStrict?: boolean | undefined; +}): Promise[]>; +/** + * Prefetch byte ranges from an AsyncBuffer. + * + * @param {AsyncBuffer} file + * @param {QueryPlan} plan + * @returns {AsyncBuffer} + */ +export function prefetchAsyncBuffer(file: AsyncBuffer, { fetches }: QueryPlan): AsyncBuffer; +import type { ParquetReadOptions } from '../src/types.js'; +import type { BloomFilter } from '../src/types.js'; +import type { SchemaElement } from '../src/types.js'; +import type { QueryPlan } from '../src/types.js'; +import type { AsyncBuffer } from '../src/types.js'; +import type { FileMetaData } from '../src/types.js'; +import type { ParquetQueryFilter } from '../src/types.js'; +//# sourceMappingURL=plan.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/plan.d.ts.map b/web/node_modules/hyparquet/types/plan.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..7ec9346759e20a82419b4badb3859d2ac2dc9b68 --- /dev/null +++ b/web/node_modules/hyparquet/types/plan.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"plan.d.ts","sourceRoot":"","sources":["../src/plan.js"],"names":[],"mappings":"AAYA;;;;;;GAMG;AACH,gJAHW,kBAAkB,GAAG;IAAE,mBAAmB,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,WAAW,CAAC,EAAE,CAAC;IAAC,cAAc,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,CAAA;CAAE,GAC1H,SAAS,CA6FrB;AAED;;;;;;;;;;;;;GAaG;AACH,+EANG;IAA6B,IAAI,EAAzB,WAAW;IACW,QAAQ,EAA9B,YAAY;IACgB,MAAM,EAAlC,kBAAkB;IACA,YAAY;CACtC,GAAU,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,WAAW,CAAC,EAAE,CAAC,CA6BlD;AAED;;;;;;GAMG;AACH,0CAJW,WAAW,eACX,SAAS,GACP,WAAW,CA4BvB;wCAtL2J,iBAAiB;iCAAjB,iBAAiB;mCAAjB,iBAAiB;+BAAjB,iBAAiB;iCAAjB,iBAAiB;kCAAjB,iBAAiB;wCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/query.d.ts b/web/node_modules/hyparquet/types/query.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..f80d76ab11f7698bb2f3979be9bbcc93b3ebaa72 --- /dev/null +++ b/web/node_modules/hyparquet/types/query.d.ts @@ -0,0 +1,14 @@ +/** + * Wraps parquetRead with orderBy support. + * This is a parquet-aware query engine that can read a subset of rows and columns. + * Accepts optional orderBy column name to sort the results. + * Note that using orderBy may SIGNIFICANTLY increase the query time. + * + * @param {BaseParquetReadOptions & { orderBy?: string }} options + * @returns {Promise[]>} resolves when all requested rows and columns are parsed + */ +export function parquetQuery(options: BaseParquetReadOptions & { + orderBy?: string; +}): Promise[]>; +import type { BaseParquetReadOptions } from '../src/types.js'; +//# sourceMappingURL=query.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/query.d.ts.map b/web/node_modules/hyparquet/types/query.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..0f7df304da416ad63d4b9b0b580960f63c51daab --- /dev/null +++ b/web/node_modules/hyparquet/types/query.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"query.d.ts","sourceRoot":"","sources":["../src/query.js"],"names":[],"mappings":"AAOA;;;;;;;;GAQG;AACH,sCAHW,sBAAsB,GAAG;IAAE,OAAO,CAAC,EAAE,MAAM,CAAA;CAAE,GAC3C,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC,CAkF1C;4CA/FwC,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/read.d.ts b/web/node_modules/hyparquet/types/read.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..57de7d3f36049c758a284867b9df10d9abc5bce2 --- /dev/null +++ b/web/node_modules/hyparquet/types/read.d.ts @@ -0,0 +1,38 @@ +/** + * Read parquet data rows from a file-like object. + * Reads the minimal number of row groups and columns to satisfy the request. + * + * Returns a void promise when complete. + * Errors are thrown on the returned promise. + * Data is returned in callbacks onComplete, onChunk, onPage, NOT the return promise. + * See parquetReadObjects for a more convenient API. + * + * @param {ParquetReadOptions} options read options + * @returns {Promise} resolves when all requested rows and columns are parsed, all errors are thrown here + */ +export function parquetRead(options: ParquetReadOptions): Promise; +/** + * @param {ParquetReadOptions} options read options + * @returns {AsyncRowGroup[]} + */ +export function parquetReadAsync(options: ParquetReadOptions): AsyncRowGroup[]; +/** + * Reads a single column from a parquet file. + * + * @param {BaseParquetReadOptions} options + * @returns {Promise} + */ +export function parquetReadColumn(options: BaseParquetReadOptions): Promise; +/** + * This is a helper function to read parquet row data as a promise. + * It is a wrapper around the more configurable parquetRead function. + * + * @param {Omit} options + * @returns {Promise[]>} resolves when all requested rows and columns are parsed + */ +export function parquetReadObjects(options: Omit): Promise[]>; +import type { ParquetReadOptions } from '../src/types.js'; +import type { AsyncRowGroup } from '../src/types.js'; +import type { BaseParquetReadOptions } from '../src/types.js'; +import type { DecodedArray } from '../src/types.js'; +//# sourceMappingURL=read.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/read.d.ts.map b/web/node_modules/hyparquet/types/read.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..0a562ec2a9921cd760fb65e035dd7e9a97822ac5 --- /dev/null +++ b/web/node_modules/hyparquet/types/read.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"read.d.ts","sourceRoot":"","sources":["../src/read.js"],"names":[],"mappings":"AAUA;;;;;;;;;;;GAWG;AACH,qCAHW,kBAAkB,GAChB,OAAO,CAAC,IAAI,CAAC,CAyGzB;AAgBD;;;GAGG;AACH,0CAHW,kBAAkB,GAChB,aAAa,EAAE,CAY3B;AAED;;;;;GAKG;AACH,2CAHW,sBAAsB,GACpB,OAAO,CAAC,YAAY,CAAC,CAyBjC;AA4BD;;;;;;GAMG;AACH,4CAHW,IAAI,CAAC,kBAAkB,EAAE,YAAY,CAAC,GACpC,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC,CAU1C;wCApOwG,iBAAiB;mCAAjB,iBAAiB;4CAAjB,iBAAiB;kCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/rowgroup.d.ts b/web/node_modules/hyparquet/types/rowgroup.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..1e182c904c516b744ac50c324da8dcfe55ba46ca --- /dev/null +++ b/web/node_modules/hyparquet/types/rowgroup.d.ts @@ -0,0 +1,45 @@ +/** + * Read a row group from a file-like object. + * + * @param {ParquetReadOptions} options + * @param {QueryPlan} plan + * @param {GroupPlan} groupPlan + * @returns {AsyncRowGroup} resolves to column data + */ +export function readRowGroup(options: ParquetReadOptions, { metadata }: QueryPlan, groupPlan: GroupPlan): AsyncRowGroup; +/** + * @overload + * @param {AsyncRowGroup} asyncGroup + * @param {number} selectStart + * @param {number} selectEnd + * @param {string[] | undefined} columns + * @param {'object'} rowFormat + * @returns {Promise[]>} resolves to row data + */ +export function asyncGroupToRows(asyncGroup: AsyncRowGroup, selectStart: number, selectEnd: number, columns: string[] | undefined, rowFormat: "object"): Promise[]>; +/** + * @overload + * @param {AsyncRowGroup} asyncGroup + * @param {number} selectStart + * @param {number} selectEnd + * @param {string[] | undefined} columns + * @param {'array'} [rowFormat] + * @returns {Promise} resolves to row data + */ +export function asyncGroupToRows(asyncGroup: AsyncRowGroup, selectStart: number, selectEnd: number, columns: string[] | undefined, rowFormat?: "array" | undefined): Promise; +/** + * Assemble physical columns into top-level columns asynchronously. + * + * @param {AsyncRowGroup} asyncRowGroup + * @param {SchemaTree} schemaTree + * @param {ParquetParsers} [parsers] + * @returns {AsyncRowGroup} + */ +export function assembleAsync(asyncRowGroup: AsyncRowGroup, schemaTree: SchemaTree, parsers?: ParquetParsers): AsyncRowGroup; +import type { ParquetReadOptions } from '../src/types.js'; +import type { QueryPlan } from '../src/types.js'; +import type { GroupPlan } from '../src/types.js'; +import type { AsyncRowGroup } from '../src/types.js'; +import type { SchemaTree } from '../src/types.js'; +import type { ParquetParsers } from '../src/types.js'; +//# sourceMappingURL=rowgroup.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/rowgroup.d.ts.map b/web/node_modules/hyparquet/types/rowgroup.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..59a2f3b065442d08674a59657fd4bb0fd40b4192 --- /dev/null +++ b/web/node_modules/hyparquet/types/rowgroup.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"rowgroup.d.ts","sourceRoot":"","sources":["../src/rowgroup.js"],"names":[],"mappings":"AAWA;;;;;;;GAOG;AACH,sCALW,kBAAkB,gBAClB,SAAS,aACT,SAAS,GACP,aAAa,CAgFzB;;;;;;;;;;AAGE,6CACQ,aAAa,eACb,MAAM,aACN,MAAM,WACN,MAAM,EAAE,GAAG,SAAS,aACpB,QAAQ,GACN,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC,CAC1C;;;;;;;;;;AAEE,6CACQ,aAAa,eACb,MAAM,aACN,MAAM,WACN,MAAM,EAAE,GAAG,SAAS,oCAElB,OAAO,CAAC,GAAG,EAAE,EAAE,CAAC,CAC5B;AAwDD;;;;;;;GAOG;AACH,6CALW,aAAa,cACb,UAAU,YACV,cAAc,GACZ,aAAa,CA8CzB;wCA/NgI,iBAAiB;+BAAjB,iBAAiB;+BAAjB,iBAAiB;mCAAjB,iBAAiB;gCAAjB,iBAAiB;oCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/schema.d.ts b/web/node_modules/hyparquet/types/schema.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..3507ed1d1a967bdaecdbea9f626beb007ba3b084 --- /dev/null +++ b/web/node_modules/hyparquet/types/schema.d.ts @@ -0,0 +1,53 @@ +/** + * Get schema elements from the root to the given element name. + * + * @param {SchemaElement[]} schema + * @param {string[]} name path to the element + * @returns {SchemaTree[]} list of schema elements + */ +export function getSchemaPath(schema: SchemaElement[], name: string[]): SchemaTree[]; +/** + * Get all physical (leaf) column names. + * + * @param {SchemaTree} schemaTree + * @returns {string[]} list of physical column names + */ +export function getPhysicalColumns(schemaTree: SchemaTree): string[]; +/** + * Get the max repetition level for a given schema path. + * + * @param {SchemaTree[]} schemaPath + * @returns {number} max repetition level + */ +export function getMaxRepetitionLevel(schemaPath: SchemaTree[]): number; +/** + * Get the max definition level for a given schema path. + * + * @param {SchemaTree[]} schemaPath + * @returns {number} max definition level + */ +export function getMaxDefinitionLevel(schemaPath: SchemaTree[]): number; +/** + * Check if a column is list-like. + * + * @param {SchemaTree} schema + * @returns {boolean} true if list-like + */ +export function isListLike(schema: SchemaTree): boolean; +/** + * Check if a column is map-like. + * + * @param {SchemaTree} schema + * @returns {boolean} true if map-like + */ +export function isMapLike(schema: SchemaTree): boolean; +/** + * Returns true if a column is non-nested. + * + * @param {SchemaTree[]} schemaPath + * @returns {boolean} + */ +export function isFlatColumn(schemaPath: SchemaTree[]): boolean; +import type { SchemaElement } from '../src/types.js'; +import type { SchemaTree } from '../src/types.js'; +//# sourceMappingURL=schema.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/schema.d.ts.map b/web/node_modules/hyparquet/types/schema.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..5d23fbd049fcb445de8be10f16f92e6336bb4131 --- /dev/null +++ b/web/node_modules/hyparquet/types/schema.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"schema.d.ts","sourceRoot":"","sources":["../src/schema.js"],"names":[],"mappings":"AA0BA;;;;;;GAMG;AACH,sCAJW,aAAa,EAAE,QACf,MAAM,EAAE,GACN,UAAU,EAAE,CAYxB;AAED;;;;;GAKG;AACH,+CAHW,UAAU,GACR,MAAM,EAAE,CAiBpB;AAED;;;;;GAKG;AACH,kDAHW,UAAU,EAAE,GACV,MAAM,CAUlB;AAED;;;;;GAKG;AACH,kDAHW,UAAU,EAAE,GACV,MAAM,CAUlB;AAED;;;;;GAKG;AACH,mCAHW,UAAU,GACR,OAAO,CAYnB;AAED;;;;;GAKG;AACH,kCAHW,UAAU,GACR,OAAO,CAkBnB;AAED;;;;;GAKG;AACH,yCAHW,UAAU,EAAE,GACV,OAAO,CAQnB;mCAG2C,iBAAiB;gCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/snappy.d.ts b/web/node_modules/hyparquet/types/snappy.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..191d6d72f36f0599b3cf0b1e993afbef7a7647e7 --- /dev/null +++ b/web/node_modules/hyparquet/types/snappy.d.ts @@ -0,0 +1,9 @@ +/** + * Decompress snappy data. + * Accepts an output buffer to avoid allocating a new buffer for each call. + * + * @param {Uint8Array} input compressed data + * @param {Uint8Array} output output buffer + */ +export function snappyUncompress(input: Uint8Array, output: Uint8Array): void; +//# sourceMappingURL=snappy.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/snappy.d.ts.map b/web/node_modules/hyparquet/types/snappy.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..0f1d7f4b4ac91064357048b0bf801da87e5dfe62 --- /dev/null +++ b/web/node_modules/hyparquet/types/snappy.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"snappy.d.ts","sourceRoot":"","sources":["../src/snappy.js"],"names":[],"mappings":"AAuBA;;;;;;GAMG;AACH,wCAHW,UAAU,UACV,UAAU,QAkGpB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/thrift.d.ts b/web/node_modules/hyparquet/types/thrift.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..447350fc23365d78ab40a947ad1cfa0687a75f15 --- /dev/null +++ b/web/node_modules/hyparquet/types/thrift.d.ts @@ -0,0 +1,33 @@ +/** + * Parse TCompactProtocol + * + * @param {DataReader} reader + * @returns {{ [key: `field_${number}`]: any }} + */ +export function deserializeTCompactProtocol(reader: DataReader): { + [key: `field_${number}`]: any; +}; +/** + * Read varint aka Unsigned LEB128. + * + * @param {DataReader} reader + * @returns {number} + */ +export function readVarInt(reader: DataReader): number; +/** + * Read a zigzag number. + * Zigzag folds positive and negative numbers into the positive number space. + * + * @param {DataReader} reader + * @returns {number} + */ +export function readZigZag(reader: DataReader): number; +/** + * Read a zigzag bigint. + * + * @param {DataReader} reader + * @returns {bigint} + */ +export function readZigZagBigInt(reader: DataReader): bigint; +import type { DataReader } from '../src/types.js'; +//# sourceMappingURL=thrift.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/thrift.d.ts.map b/web/node_modules/hyparquet/types/thrift.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..f439b9cd34fea1030520d0ce6b2d03307fb9854f --- /dev/null +++ b/web/node_modules/hyparquet/types/thrift.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"thrift.d.ts","sourceRoot":"","sources":["../src/thrift.js"],"names":[],"mappings":"AAiBA;;;;;GAKG;AACH,oDAHW,UAAU,GACR;IAAE,CAAC,GAAG,EAAE,SAAS,MAAM,EAAE,GAAG,GAAG,CAAA;CAAE,CAkB7C;AAwDD;;;;;GAKG;AACH,mCAHW,UAAU,GACR,MAAM,CAclB;AAqBD;;;;;;GAMG;AACH,mCAHW,UAAU,GACR,MAAM,CAKlB;AAED;;;;;GAKG;AACH,yCAHW,UAAU,GACR,MAAM,CAKlB;gCA1JsD,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/utils.d.ts b/web/node_modules/hyparquet/types/utils.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..aec5100490bb0f12886066b9a0a0d0371f235c35 --- /dev/null +++ b/web/node_modules/hyparquet/types/utils.d.ts @@ -0,0 +1,75 @@ +/** + * Replace bigint, date, etc with legal JSON types. + * + * @param {any} obj object to convert + * @returns {unknown} converted object + */ +export function toJson(obj: any): unknown; +/** + * Concatenate two arrays fast. + * + * @param {any[]} aaa + * @param {DecodedArray} bbb + */ +export function concat(aaa: any[], bbb: DecodedArray): void; +/** + * Deep equality. + * + * @param {any} a + * @param {any} b + * @param {boolean} [strict] + * @returns {boolean} + */ +export function equals(a: any, b: any, strict?: boolean): boolean; +/** + * Get the byte length of a URL using a HEAD request. + * If HEAD fails with 403 (e.g., with signed S3 URLs), falls back to a ranged GET request. + * If HEAD succeeds but Content-Length is missing, falls back to GET with range. + * If requestInit is provided, it will be passed to fetch. + * + * @param {string} url + * @param {RequestInit} [requestInit] fetch options + * @param {typeof globalThis.fetch} [customFetch] fetch function to use + * @returns {Promise} + */ +export function byteLengthFromUrl(url: string, requestInit?: RequestInit, customFetch?: typeof globalThis.fetch): Promise; +/** + * Construct an AsyncBuffer for a URL. + * If byteLength is not provided, will make a HEAD request to get the file size. + * If fetch is provided, it will be used instead of the global fetch. + * If requestInit is provided, it will be passed to fetch. + * + * @param {object} options + * @param {string} options.url + * @param {number} [options.byteLength] + * @param {typeof globalThis.fetch} [options.fetch] fetch function to use + * @param {RequestInit} [options.requestInit] + * @returns {Promise} + */ +export function asyncBufferFromUrl({ url, byteLength, requestInit, fetch: customFetch }: { + url: string; + byteLength?: number | undefined; + fetch?: typeof fetch | undefined; + requestInit?: RequestInit | undefined; +}): Promise; +/** + * Returns a cached layer on top of an AsyncBuffer. For caching slices of a file + * that are read multiple times, possibly over a network. + * + * @param {AsyncBuffer} file file-like object to cache + * @param {{ minSize?: number }} [options] + * @returns {AsyncBuffer} cached file-like object + */ +export function cachedAsyncBuffer({ byteLength, slice }: AsyncBuffer, { minSize }?: { + minSize?: number; +}): AsyncBuffer; +/** + * Flatten a list of lists into a single list. + * + * @param {DecodedArray[]} [chunks] + * @returns {DecodedArray} + */ +export function flatten(chunks?: DecodedArray[]): DecodedArray; +import type { DecodedArray } from '../src/types.js'; +import type { AsyncBuffer } from '../src/types.js'; +//# sourceMappingURL=utils.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/utils.d.ts.map b/web/node_modules/hyparquet/types/utils.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..172b91370b1928675a3fb1b1599485554668066e --- /dev/null +++ b/web/node_modules/hyparquet/types/utils.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"utils.d.ts","sourceRoot":"","sources":["../src/utils.js"],"names":[],"mappings":"AAMA;;;;;GAKG;AACH,4BAHW,GAAG,GACD,OAAO,CAmBnB;AAED;;;;;GAKG;AACH,4BAHW,GAAG,EAAE,OACL,YAAY,QAOtB;AAED;;;;;;;GAOG;AACH,0BALW,GAAG,KACH,GAAG,WACH,OAAO,GACL,OAAO,CA4BnB;AAiDD;;;;;;;;;;GAUG;AACH,uCALW,MAAM,gBACN,WAAW,gBACX,OAAO,UAAU,CAAC,KAAK,GACrB,OAAO,CAAC,MAAM,CAAC,CAkB3B;AAED;;;;;;;;;;;;GAYG;AACH,yFANG;IAAwB,GAAG,EAAnB,MAAM;IACW,UAAU;IACO,KAAK;IACjB,WAAW;CACzC,GAAU,OAAO,CAAC,WAAW,CAAC,CAyChC;AAED;;;;;;;GAOG;AACH,yDAJW,WAAW,gBACX;IAAE,OAAO,CAAC,EAAE,MAAM,CAAA;CAAE,GAClB,WAAW,CA+BvB;AA2BD;;;;;GAKG;AACH,iCAHW,YAAY,EAAE,GACZ,YAAY,CAWxB;kCAhSsD,iBAAiB;iCAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/variant.d.ts b/web/node_modules/hyparquet/types/variant.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..e5f04d6b8f2768957d97fc1c788831e22179e7f9 --- /dev/null +++ b/web/node_modules/hyparquet/types/variant.d.ts @@ -0,0 +1,10 @@ +/** + * Recursively decode variant structs into native values. + * + * @param {any} value + * @param {ParquetParsers} [parsers] + * @returns {any} + */ +export function decodeVariantColumn(value: any, parsers?: ParquetParsers): any; +import type { ParquetParsers } from '../src/types.js'; +//# sourceMappingURL=variant.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/variant.d.ts.map b/web/node_modules/hyparquet/types/variant.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..49a4d0f1f1c289c7f07d1ca952c676af5ad3bf88 --- /dev/null +++ b/web/node_modules/hyparquet/types/variant.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"variant.d.ts","sourceRoot":"","sources":["../src/variant.js"],"names":[],"mappings":"AAUA;;;;;;GAMG;AACH,2CAJW,GAAG,YACH,cAAc,GACZ,GAAG,CAyBf;oCArC6D,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/wkb.d.ts b/web/node_modules/hyparquet/types/wkb.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..c905caba429daa89bec51c782ed7e26963586e64 --- /dev/null +++ b/web/node_modules/hyparquet/types/wkb.d.ts @@ -0,0 +1,16 @@ +/** + * WKB (Well-Known Binary) decoder for geometry objects. + * + * @param {DataReader} reader + * @returns {Geometry} geometry object + */ +export function wkbToGeojson(reader: DataReader): Geometry; +export type WkbFlags = { + littleEndian: boolean; + type: number; + dim: number; + count: number; +}; +import type { DataReader } from '../src/types.js'; +import type { Geometry } from '../src/types.js'; +//# sourceMappingURL=wkb.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/wkb.d.ts.map b/web/node_modules/hyparquet/types/wkb.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..5f7c6a13e5b65a226ac7ab4f8e888be35276f15b --- /dev/null +++ b/web/node_modules/hyparquet/types/wkb.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"wkb.d.ts","sourceRoot":"","sources":["../src/wkb.js"],"names":[],"mappings":"AAAA;;;;;GAKG;AACH,qCAHW,UAAU,GACR,QAAQ,CAsCpB;;kBA6Ea,OAAO;UACP,MAAM;SACN,MAAM;WACN,MAAM;;gCAImB,iBAAiB;8BAAjB,iBAAiB"} \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/xxhash.d.ts b/web/node_modules/hyparquet/types/xxhash.d.ts new file mode 100644 index 0000000000000000000000000000000000000000..10b7bcdc3adc8a5aff49ea0cd892f6ea63d9215a --- /dev/null +++ b/web/node_modules/hyparquet/types/xxhash.d.ts @@ -0,0 +1,9 @@ +/** + * Compute the 64-bit xxHash of a byte buffer. + * + * @param {Uint8Array} input + * @param {bigint} [seed] + * @returns {bigint} 64-bit hash + */ +export function xxhash64(input: Uint8Array, seed?: bigint): bigint; +//# sourceMappingURL=xxhash.d.ts.map \ No newline at end of file diff --git a/web/node_modules/hyparquet/types/xxhash.d.ts.map b/web/node_modules/hyparquet/types/xxhash.d.ts.map new file mode 100644 index 0000000000000000000000000000000000000000..0a13a573365a17741fab64eed40287fa848ac29b --- /dev/null +++ b/web/node_modules/hyparquet/types/xxhash.d.ts.map @@ -0,0 +1 @@ +{"version":3,"file":"xxhash.d.ts","sourceRoot":"","sources":["../src/xxhash.js"],"names":[],"mappings":"AAwCA;;;;;;GAMG;AACH,gCAJW,UAAU,SACV,MAAM,GACJ,MAAM,CAwDlB"} \ No newline at end of file diff --git a/web/node_modules/ws/LICENSE b/web/node_modules/ws/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..1da5b96a11ac2c83160e3085188f04eba7e3dd7d --- /dev/null +++ b/web/node_modules/ws/LICENSE @@ -0,0 +1,20 @@ +Copyright (c) 2011 Einar Otto Stangvik +Copyright (c) 2013 Arnout Kazemier and contributors +Copyright (c) 2016 Luigi Pinca and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/web/node_modules/ws/README.md b/web/node_modules/ws/README.md new file mode 100644 index 0000000000000000000000000000000000000000..21f10df108e11042e02b9b9cb81528d2951e18e8 --- /dev/null +++ b/web/node_modules/ws/README.md @@ -0,0 +1,548 @@ +# ws: a Node.js WebSocket library + +[![Version npm](https://img.shields.io/npm/v/ws.svg?logo=npm)](https://www.npmjs.com/package/ws) +[![CI](https://img.shields.io/github/actions/workflow/status/websockets/ws/ci.yml?branch=master&label=CI&logo=github)](https://github.com/websockets/ws/actions?query=workflow%3ACI+branch%3Amaster) +[![Coverage Status](https://img.shields.io/coveralls/websockets/ws/master.svg?logo=coveralls)](https://coveralls.io/github/websockets/ws) + +ws is a simple to use, blazing fast, and thoroughly tested WebSocket client and +server implementation. + +Passes the quite extensive Autobahn test suite: [server][server-report], +[client][client-report]. + +**Note**: This module does not work in the browser. The client in the docs is a +reference to a backend with the role of a client in the WebSocket communication. +Browser clients must use the native +[`WebSocket`](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) +object. To make the same code work seamlessly on Node.js and the browser, you +can use one of the many wrappers available on npm, like +[isomorphic-ws](https://github.com/heineiuo/isomorphic-ws). + +## Table of Contents + +- [Protocol support](#protocol-support) +- [Installing](#installing) + - [Opt-in for performance](#opt-in-for-performance) + - [Legacy opt-in for performance](#legacy-opt-in-for-performance) +- [API docs](#api-docs) +- [WebSocket compression](#websocket-compression) +- [Usage examples](#usage-examples) + - [Sending and receiving text data](#sending-and-receiving-text-data) + - [Sending binary data](#sending-binary-data) + - [Simple server](#simple-server) + - [External HTTP/S server](#external-https-server) + - [Multiple servers sharing a single HTTP/S server](#multiple-servers-sharing-a-single-https-server) + - [Client authentication](#client-authentication) + - [Server broadcast](#server-broadcast) + - [Round-trip time](#round-trip-time) + - [Use the Node.js streams API](#use-the-nodejs-streams-api) + - [Other examples](#other-examples) +- [FAQ](#faq) + - [How to get the IP address of the client?](#how-to-get-the-ip-address-of-the-client) + - [How to detect and close broken connections?](#how-to-detect-and-close-broken-connections) + - [How to connect via a proxy?](#how-to-connect-via-a-proxy) +- [Changelog](#changelog) +- [License](#license) + +## Protocol support + +- **HyBi drafts 07-12** (Use the option `protocolVersion: 8`) +- **HyBi drafts 13-17** (Current default, alternatively option + `protocolVersion: 13`) + +## Installing + +``` +npm install ws +``` + +### Opt-in for performance + +[bufferutil][] is an optional module that can be installed alongside the ws +module: + +``` +npm install --save-optional bufferutil +``` + +This is a binary addon that improves the performance of certain operations such +as masking and unmasking the data payload of the WebSocket frames. Prebuilt +binaries are available for the most popular platforms, so you don't necessarily +need to have a C++ compiler installed on your machine. + +To force ws to not use bufferutil, use the +[`WS_NO_BUFFER_UTIL`](./doc/ws.md#ws_no_buffer_util) environment variable. This +can be useful to enhance security in systems where a user can put a package in +the package search path of an application of another user, due to how the +Node.js resolver algorithm works. + +#### Legacy opt-in for performance + +If you are running on an old version of Node.js (prior to v18.14.0), ws also +supports the [utf-8-validate][] module: + +``` +npm install --save-optional utf-8-validate +``` + +This contains a binary polyfill for [`buffer.isUtf8()`][]. + +To force ws not to use utf-8-validate, use the +[`WS_NO_UTF_8_VALIDATE`](./doc/ws.md#ws_no_utf_8_validate) environment variable. + +## API docs + +See [`/doc/ws.md`](./doc/ws.md) for Node.js-like documentation of ws classes and +utility functions. + +## WebSocket compression + +ws supports the [permessage-deflate extension][permessage-deflate] which enables +the client and server to negotiate a compression algorithm and its parameters, +and then selectively apply it to the data payloads of each WebSocket message. + +The extension is disabled by default on the server and enabled by default on the +client. It adds a significant overhead in terms of performance and memory +consumption so we suggest to enable it only if it is really needed. + +Note that Node.js has a variety of issues with high-performance compression, +where increased concurrency, especially on Linux, can lead to [catastrophic +memory fragmentation][node-zlib-bug] and slow performance. If you intend to use +permessage-deflate in production, it is worthwhile to set up a test +representative of your workload and ensure Node.js/zlib will handle it with +acceptable performance and memory usage. + +Tuning of permessage-deflate can be done via the options defined below. You can +also use `zlibDeflateOptions` and `zlibInflateOptions`, which is passed directly +into the creation of [raw deflate/inflate streams][node-zlib-deflaterawdocs]. + +See [the docs][ws-server-options] for more options. + +```js +import WebSocket, { WebSocketServer } from 'ws'; + +const wss = new WebSocketServer({ + port: 8080, + perMessageDeflate: { + zlibDeflateOptions: { + // See zlib defaults. + chunkSize: 1024, + memLevel: 7, + level: 3 + }, + zlibInflateOptions: { + chunkSize: 10 * 1024 + }, + // Other options settable: + clientNoContextTakeover: true, // Defaults to negotiated value. + serverNoContextTakeover: true, // Defaults to negotiated value. + serverMaxWindowBits: 10, // Defaults to negotiated value. + // Below options specified as default values. + concurrencyLimit: 10, // Limits zlib concurrency for perf. + threshold: 1024 // Size (in bytes) below which messages + // should not be compressed if context takeover is disabled. + } +}); +``` + +The client will only use the extension if it is supported and enabled on the +server. To always disable the extension on the client, set the +`perMessageDeflate` option to `false`. + +```js +import WebSocket from 'ws'; + +const ws = new WebSocket('ws://www.host.com/path', { + perMessageDeflate: false +}); +``` + +## Usage examples + +### Sending and receiving text data + +```js +import WebSocket from 'ws'; + +const ws = new WebSocket('ws://www.host.com/path'); + +ws.on('error', console.error); + +ws.on('open', function open() { + ws.send('something'); +}); + +ws.on('message', function message(data) { + console.log('received: %s', data); +}); +``` + +### Sending binary data + +```js +import WebSocket from 'ws'; + +const ws = new WebSocket('ws://www.host.com/path'); + +ws.on('error', console.error); + +ws.on('open', function open() { + const array = new Float32Array(5); + + for (var i = 0; i < array.length; ++i) { + array[i] = i / 2; + } + + ws.send(array); +}); +``` + +### Simple server + +```js +import { WebSocketServer } from 'ws'; + +const wss = new WebSocketServer({ port: 8080 }); + +wss.on('connection', function connection(ws) { + ws.on('error', console.error); + + ws.on('message', function message(data) { + console.log('received: %s', data); + }); + + ws.send('something'); +}); +``` + +### External HTTP/S server + +```js +import { createServer } from 'https'; +import { readFileSync } from 'fs'; +import { WebSocketServer } from 'ws'; + +const server = createServer({ + cert: readFileSync('/path/to/cert.pem'), + key: readFileSync('/path/to/key.pem') +}); +const wss = new WebSocketServer({ server }); + +wss.on('connection', function connection(ws) { + ws.on('error', console.error); + + ws.on('message', function message(data) { + console.log('received: %s', data); + }); + + ws.send('something'); +}); + +server.listen(8080); +``` + +### Multiple servers sharing a single HTTP/S server + +```js +import { createServer } from 'http'; +import { WebSocketServer } from 'ws'; + +const server = createServer(); +const wss1 = new WebSocketServer({ noServer: true }); +const wss2 = new WebSocketServer({ noServer: true }); + +wss1.on('connection', function connection(ws) { + ws.on('error', console.error); + + // ... +}); + +wss2.on('connection', function connection(ws) { + ws.on('error', console.error); + + // ... +}); + +server.on('upgrade', function upgrade(request, socket, head) { + const { pathname } = new URL(request.url, 'wss://base.url'); + + if (pathname === '/foo') { + wss1.handleUpgrade(request, socket, head, function done(ws) { + wss1.emit('connection', ws, request); + }); + } else if (pathname === '/bar') { + wss2.handleUpgrade(request, socket, head, function done(ws) { + wss2.emit('connection', ws, request); + }); + } else { + socket.destroy(); + } +}); + +server.listen(8080); +``` + +### Client authentication + +```js +import { createServer } from 'http'; +import { WebSocketServer } from 'ws'; + +function onSocketError(err) { + console.error(err); +} + +const server = createServer(); +const wss = new WebSocketServer({ noServer: true }); + +wss.on('connection', function connection(ws, request, client) { + ws.on('error', console.error); + + ws.on('message', function message(data) { + console.log(`Received message ${data} from user ${client}`); + }); +}); + +server.on('upgrade', function upgrade(request, socket, head) { + socket.on('error', onSocketError); + + // This function is not defined on purpose. Implement it with your own logic. + authenticate(request, function next(err, client) { + if (err || !client) { + socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n'); + socket.destroy(); + return; + } + + socket.removeListener('error', onSocketError); + + wss.handleUpgrade(request, socket, head, function done(ws) { + wss.emit('connection', ws, request, client); + }); + }); +}); + +server.listen(8080); +``` + +Also see the provided [example][session-parse-example] using `express-session`. + +### Server broadcast + +A client WebSocket broadcasting to all connected WebSocket clients, including +itself. + +```js +import WebSocket, { WebSocketServer } from 'ws'; + +const wss = new WebSocketServer({ port: 8080 }); + +wss.on('connection', function connection(ws) { + ws.on('error', console.error); + + ws.on('message', function message(data, isBinary) { + wss.clients.forEach(function each(client) { + if (client.readyState === WebSocket.OPEN) { + client.send(data, { binary: isBinary }); + } + }); + }); +}); +``` + +A client WebSocket broadcasting to every other connected WebSocket clients, +excluding itself. + +```js +import WebSocket, { WebSocketServer } from 'ws'; + +const wss = new WebSocketServer({ port: 8080 }); + +wss.on('connection', function connection(ws) { + ws.on('error', console.error); + + ws.on('message', function message(data, isBinary) { + wss.clients.forEach(function each(client) { + if (client !== ws && client.readyState === WebSocket.OPEN) { + client.send(data, { binary: isBinary }); + } + }); + }); +}); +``` + +### Round-trip time + +```js +import WebSocket from 'ws'; + +const ws = new WebSocket('wss://websocket-echo.com/'); + +ws.on('error', console.error); + +ws.on('open', function open() { + console.log('connected'); + ws.send(Date.now()); +}); + +ws.on('close', function close() { + console.log('disconnected'); +}); + +ws.on('message', function message(data) { + console.log(`Round-trip time: ${Date.now() - data} ms`); + + setTimeout(function timeout() { + ws.send(Date.now()); + }, 500); +}); +``` + +### Use the Node.js streams API + +```js +import WebSocket, { createWebSocketStream } from 'ws'; + +const ws = new WebSocket('wss://websocket-echo.com/'); + +const duplex = createWebSocketStream(ws, { encoding: 'utf8' }); + +duplex.on('error', console.error); + +duplex.pipe(process.stdout); +process.stdin.pipe(duplex); +``` + +### Other examples + +For a full example with a browser client communicating with a ws server, see the +examples folder. + +Otherwise, see the test cases. + +## FAQ + +### How to get the IP address of the client? + +The remote IP address can be obtained from the raw socket. + +```js +import { WebSocketServer } from 'ws'; + +const wss = new WebSocketServer({ port: 8080 }); + +wss.on('connection', function connection(ws, req) { + const ip = req.socket.remoteAddress; + + ws.on('error', console.error); +}); +``` + +When the server runs behind a proxy like NGINX, the de-facto standard is to use +the `X-Forwarded-For` header. + +```js +wss.on('connection', function connection(ws, req) { + const ip = req.headers['x-forwarded-for'].split(',')[0].trim(); + + ws.on('error', console.error); +}); +``` + +### How to detect and close broken connections? + +Sometimes, the link between the server and the client can be interrupted in a +way that keeps both the server and the client unaware of the broken state of the +connection (e.g. when pulling the cord). + +In these cases, ping messages can be used as a means to verify that the remote +endpoint is still responsive. + +```js +import { WebSocketServer } from 'ws'; + +function heartbeat() { + this.isAlive = true; +} + +const wss = new WebSocketServer({ port: 8080 }); + +wss.on('connection', function connection(ws) { + ws.isAlive = true; + ws.on('error', console.error); + ws.on('pong', heartbeat); +}); + +const interval = setInterval(function ping() { + wss.clients.forEach(function each(ws) { + if (ws.isAlive === false) return ws.terminate(); + + ws.isAlive = false; + ws.ping(); + }); +}, 30000); + +wss.on('close', function close() { + clearInterval(interval); +}); +``` + +Pong messages are automatically sent in response to ping messages as required by +the spec. + +Just like the server example above, your clients might as well lose connection +without knowing it. You might want to add a ping listener on your clients to +prevent that. A simple implementation would be: + +```js +import WebSocket from 'ws'; + +function heartbeat() { + clearTimeout(this.pingTimeout); + + // Use `WebSocket#terminate()`, which immediately destroys the connection, + // instead of `WebSocket#close()`, which waits for the close timer. + // Delay should be equal to the interval at which your server + // sends out pings plus a conservative assumption of the latency. + this.pingTimeout = setTimeout(() => { + this.terminate(); + }, 30000 + 1000); +} + +const client = new WebSocket('wss://websocket-echo.com/'); + +client.on('error', console.error); +client.on('open', heartbeat); +client.on('ping', heartbeat); +client.on('close', function clear() { + clearTimeout(this.pingTimeout); +}); +``` + +### How to connect via a proxy? + +Use a custom `http.Agent` implementation like [https-proxy-agent][] or +[socks-proxy-agent][]. + +## Changelog + +We're using the GitHub [releases][changelog] for changelog entries. + +## License + +[MIT](LICENSE) + +[`buffer.isutf8()`]: https://nodejs.org/api/buffer.html#bufferisutf8input +[bufferutil]: https://github.com/websockets/bufferutil +[changelog]: https://github.com/websockets/ws/releases +[client-report]: http://websockets.github.io/ws/autobahn/clients/ +[https-proxy-agent]: https://github.com/TooTallNate/node-https-proxy-agent +[node-zlib-bug]: https://github.com/nodejs/node/issues/8871 +[node-zlib-deflaterawdocs]: + https://nodejs.org/api/zlib.html#zlib_zlib_createdeflateraw_options +[permessage-deflate]: https://tools.ietf.org/html/rfc7692 +[server-report]: http://websockets.github.io/ws/autobahn/servers/ +[session-parse-example]: ./examples/express-session-parse +[socks-proxy-agent]: https://github.com/TooTallNate/node-socks-proxy-agent +[utf-8-validate]: https://github.com/websockets/utf-8-validate +[ws-server-options]: ./doc/ws.md#new-websocketserveroptions-callback diff --git a/web/node_modules/ws/browser.js b/web/node_modules/ws/browser.js new file mode 100644 index 0000000000000000000000000000000000000000..ca4f628ac17c80e5b1d40910d1b18c605490deda --- /dev/null +++ b/web/node_modules/ws/browser.js @@ -0,0 +1,8 @@ +'use strict'; + +module.exports = function () { + throw new Error( + 'ws does not work in the browser. Browser clients must use the native ' + + 'WebSocket object' + ); +}; diff --git a/web/node_modules/ws/index.js b/web/node_modules/ws/index.js new file mode 100644 index 0000000000000000000000000000000000000000..3fdb7b21030c13e27777d6349ab09ed2d9910c01 --- /dev/null +++ b/web/node_modules/ws/index.js @@ -0,0 +1,22 @@ +'use strict'; + +const createWebSocketStream = require('./lib/stream'); +const extension = require('./lib/extension'); +const PerMessageDeflate = require('./lib/permessage-deflate'); +const Receiver = require('./lib/receiver'); +const Sender = require('./lib/sender'); +const subprotocol = require('./lib/subprotocol'); +const WebSocket = require('./lib/websocket'); +const WebSocketServer = require('./lib/websocket-server'); + +WebSocket.createWebSocketStream = createWebSocketStream; +WebSocket.extension = extension; +WebSocket.PerMessageDeflate = PerMessageDeflate; +WebSocket.Receiver = Receiver; +WebSocket.Sender = Sender; +WebSocket.Server = WebSocketServer; +WebSocket.subprotocol = subprotocol; +WebSocket.WebSocket = WebSocket; +WebSocket.WebSocketServer = WebSocketServer; + +module.exports = WebSocket; diff --git a/web/node_modules/ws/lib/buffer-util.js b/web/node_modules/ws/lib/buffer-util.js new file mode 100644 index 0000000000000000000000000000000000000000..f7536e28efa570277e30e9ea001200ee11595fee --- /dev/null +++ b/web/node_modules/ws/lib/buffer-util.js @@ -0,0 +1,131 @@ +'use strict'; + +const { EMPTY_BUFFER } = require('./constants'); + +const FastBuffer = Buffer[Symbol.species]; + +/** + * Merges an array of buffers into a new buffer. + * + * @param {Buffer[]} list The array of buffers to concat + * @param {Number} totalLength The total length of buffers in the list + * @return {Buffer} The resulting buffer + * @public + */ +function concat(list, totalLength) { + if (list.length === 0) return EMPTY_BUFFER; + if (list.length === 1) return list[0]; + + const target = Buffer.allocUnsafe(totalLength); + let offset = 0; + + for (let i = 0; i < list.length; i++) { + const buf = list[i]; + target.set(buf, offset); + offset += buf.length; + } + + if (offset < totalLength) { + return new FastBuffer(target.buffer, target.byteOffset, offset); + } + + return target; +} + +/** + * Masks a buffer using the given mask. + * + * @param {Buffer} source The buffer to mask + * @param {Buffer} mask The mask to use + * @param {Buffer} output The buffer where to store the result + * @param {Number} offset The offset at which to start writing + * @param {Number} length The number of bytes to mask. + * @public + */ +function _mask(source, mask, output, offset, length) { + for (let i = 0; i < length; i++) { + output[offset + i] = source[i] ^ mask[i & 3]; + } +} + +/** + * Unmasks a buffer using the given mask. + * + * @param {Buffer} buffer The buffer to unmask + * @param {Buffer} mask The mask to use + * @public + */ +function _unmask(buffer, mask) { + for (let i = 0; i < buffer.length; i++) { + buffer[i] ^= mask[i & 3]; + } +} + +/** + * Converts a buffer to an `ArrayBuffer`. + * + * @param {Buffer} buf The buffer to convert + * @return {ArrayBuffer} Converted buffer + * @public + */ +function toArrayBuffer(buf) { + if (buf.length === buf.buffer.byteLength) { + return buf.buffer; + } + + return buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.length); +} + +/** + * Converts `data` to a `Buffer`. + * + * @param {*} data The data to convert + * @return {Buffer} The buffer + * @throws {TypeError} + * @public + */ +function toBuffer(data) { + toBuffer.readOnly = true; + + if (Buffer.isBuffer(data)) return data; + + let buf; + + if (data instanceof ArrayBuffer) { + buf = new FastBuffer(data); + } else if (ArrayBuffer.isView(data)) { + buf = new FastBuffer(data.buffer, data.byteOffset, data.byteLength); + } else { + buf = Buffer.from(data); + toBuffer.readOnly = false; + } + + return buf; +} + +module.exports = { + concat, + mask: _mask, + toArrayBuffer, + toBuffer, + unmask: _unmask +}; + +/* istanbul ignore else */ +if (!process.env.WS_NO_BUFFER_UTIL) { + try { + const bufferUtil = require('bufferutil'); + + module.exports.mask = function (source, mask, output, offset, length) { + if (length < 48) _mask(source, mask, output, offset, length); + else bufferUtil.mask(source, mask, output, offset, length); + }; + + module.exports.unmask = function (buffer, mask) { + if (buffer.length < 32) _unmask(buffer, mask); + else bufferUtil.unmask(buffer, mask); + }; + } catch (e) { + // Continue regardless of the error. + } +} diff --git a/web/node_modules/ws/lib/constants.js b/web/node_modules/ws/lib/constants.js new file mode 100644 index 0000000000000000000000000000000000000000..69b2fe3c43f5fb4323b7540ea1a4f1d3ddfc2d5c --- /dev/null +++ b/web/node_modules/ws/lib/constants.js @@ -0,0 +1,19 @@ +'use strict'; + +const BINARY_TYPES = ['nodebuffer', 'arraybuffer', 'fragments']; +const hasBlob = typeof Blob !== 'undefined'; + +if (hasBlob) BINARY_TYPES.push('blob'); + +module.exports = { + BINARY_TYPES, + CLOSE_TIMEOUT: 30000, + EMPTY_BUFFER: Buffer.alloc(0), + GUID: '258EAFA5-E914-47DA-95CA-C5AB0DC85B11', + hasBlob, + kForOnEventAttribute: Symbol('kIsForOnEventAttribute'), + kListener: Symbol('kListener'), + kStatusCode: Symbol('status-code'), + kWebSocket: Symbol('websocket'), + NOOP: () => {} +}; diff --git a/web/node_modules/ws/lib/event-target.js b/web/node_modules/ws/lib/event-target.js new file mode 100644 index 0000000000000000000000000000000000000000..fea4cbc52c3299d0bd5fea32245360594812b5a5 --- /dev/null +++ b/web/node_modules/ws/lib/event-target.js @@ -0,0 +1,292 @@ +'use strict'; + +const { kForOnEventAttribute, kListener } = require('./constants'); + +const kCode = Symbol('kCode'); +const kData = Symbol('kData'); +const kError = Symbol('kError'); +const kMessage = Symbol('kMessage'); +const kReason = Symbol('kReason'); +const kTarget = Symbol('kTarget'); +const kType = Symbol('kType'); +const kWasClean = Symbol('kWasClean'); + +/** + * Class representing an event. + */ +class Event { + /** + * Create a new `Event`. + * + * @param {String} type The name of the event + * @throws {TypeError} If the `type` argument is not specified + */ + constructor(type) { + this[kTarget] = null; + this[kType] = type; + } + + /** + * @type {*} + */ + get target() { + return this[kTarget]; + } + + /** + * @type {String} + */ + get type() { + return this[kType]; + } +} + +Object.defineProperty(Event.prototype, 'target', { enumerable: true }); +Object.defineProperty(Event.prototype, 'type', { enumerable: true }); + +/** + * Class representing a close event. + * + * @extends Event + */ +class CloseEvent extends Event { + /** + * Create a new `CloseEvent`. + * + * @param {String} type The name of the event + * @param {Object} [options] A dictionary object that allows for setting + * attributes via object members of the same name + * @param {Number} [options.code=0] The status code explaining why the + * connection was closed + * @param {String} [options.reason=''] A human-readable string explaining why + * the connection was closed + * @param {Boolean} [options.wasClean=false] Indicates whether or not the + * connection was cleanly closed + */ + constructor(type, options = {}) { + super(type); + + this[kCode] = options.code === undefined ? 0 : options.code; + this[kReason] = options.reason === undefined ? '' : options.reason; + this[kWasClean] = options.wasClean === undefined ? false : options.wasClean; + } + + /** + * @type {Number} + */ + get code() { + return this[kCode]; + } + + /** + * @type {String} + */ + get reason() { + return this[kReason]; + } + + /** + * @type {Boolean} + */ + get wasClean() { + return this[kWasClean]; + } +} + +Object.defineProperty(CloseEvent.prototype, 'code', { enumerable: true }); +Object.defineProperty(CloseEvent.prototype, 'reason', { enumerable: true }); +Object.defineProperty(CloseEvent.prototype, 'wasClean', { enumerable: true }); + +/** + * Class representing an error event. + * + * @extends Event + */ +class ErrorEvent extends Event { + /** + * Create a new `ErrorEvent`. + * + * @param {String} type The name of the event + * @param {Object} [options] A dictionary object that allows for setting + * attributes via object members of the same name + * @param {*} [options.error=null] The error that generated this event + * @param {String} [options.message=''] The error message + */ + constructor(type, options = {}) { + super(type); + + this[kError] = options.error === undefined ? null : options.error; + this[kMessage] = options.message === undefined ? '' : options.message; + } + + /** + * @type {*} + */ + get error() { + return this[kError]; + } + + /** + * @type {String} + */ + get message() { + return this[kMessage]; + } +} + +Object.defineProperty(ErrorEvent.prototype, 'error', { enumerable: true }); +Object.defineProperty(ErrorEvent.prototype, 'message', { enumerable: true }); + +/** + * Class representing a message event. + * + * @extends Event + */ +class MessageEvent extends Event { + /** + * Create a new `MessageEvent`. + * + * @param {String} type The name of the event + * @param {Object} [options] A dictionary object that allows for setting + * attributes via object members of the same name + * @param {*} [options.data=null] The message content + */ + constructor(type, options = {}) { + super(type); + + this[kData] = options.data === undefined ? null : options.data; + } + + /** + * @type {*} + */ + get data() { + return this[kData]; + } +} + +Object.defineProperty(MessageEvent.prototype, 'data', { enumerable: true }); + +/** + * This provides methods for emulating the `EventTarget` interface. It's not + * meant to be used directly. + * + * @mixin + */ +const EventTarget = { + /** + * Register an event listener. + * + * @param {String} type A string representing the event type to listen for + * @param {(Function|Object)} handler The listener to add + * @param {Object} [options] An options object specifies characteristics about + * the event listener + * @param {Boolean} [options.once=false] A `Boolean` indicating that the + * listener should be invoked at most once after being added. If `true`, + * the listener would be automatically removed when invoked. + * @public + */ + addEventListener(type, handler, options = {}) { + for (const listener of this.listeners(type)) { + if ( + !options[kForOnEventAttribute] && + listener[kListener] === handler && + !listener[kForOnEventAttribute] + ) { + return; + } + } + + let wrapper; + + if (type === 'message') { + wrapper = function onMessage(data, isBinary) { + const event = new MessageEvent('message', { + data: isBinary ? data : data.toString() + }); + + event[kTarget] = this; + callListener(handler, this, event); + }; + } else if (type === 'close') { + wrapper = function onClose(code, message) { + const event = new CloseEvent('close', { + code, + reason: message.toString(), + wasClean: this._closeFrameReceived && this._closeFrameSent + }); + + event[kTarget] = this; + callListener(handler, this, event); + }; + } else if (type === 'error') { + wrapper = function onError(error) { + const event = new ErrorEvent('error', { + error, + message: error.message + }); + + event[kTarget] = this; + callListener(handler, this, event); + }; + } else if (type === 'open') { + wrapper = function onOpen() { + const event = new Event('open'); + + event[kTarget] = this; + callListener(handler, this, event); + }; + } else { + return; + } + + wrapper[kForOnEventAttribute] = !!options[kForOnEventAttribute]; + wrapper[kListener] = handler; + + if (options.once) { + this.once(type, wrapper); + } else { + this.on(type, wrapper); + } + }, + + /** + * Remove an event listener. + * + * @param {String} type A string representing the event type to remove + * @param {(Function|Object)} handler The listener to remove + * @public + */ + removeEventListener(type, handler) { + for (const listener of this.listeners(type)) { + if (listener[kListener] === handler && !listener[kForOnEventAttribute]) { + this.removeListener(type, listener); + break; + } + } + } +}; + +module.exports = { + CloseEvent, + ErrorEvent, + Event, + EventTarget, + MessageEvent +}; + +/** + * Call an event listener + * + * @param {(Function|Object)} listener The listener to call + * @param {*} thisArg The value to use as `this`` when calling the listener + * @param {Event} event The event to pass to the listener + * @private + */ +function callListener(listener, thisArg, event) { + if (typeof listener === 'object' && listener.handleEvent) { + listener.handleEvent.call(listener, event); + } else { + listener.call(thisArg, event); + } +} diff --git a/web/node_modules/ws/lib/extension.js b/web/node_modules/ws/lib/extension.js new file mode 100644 index 0000000000000000000000000000000000000000..3d7895c1b0608d4bef7073a6acea9be39a4f8b74 --- /dev/null +++ b/web/node_modules/ws/lib/extension.js @@ -0,0 +1,203 @@ +'use strict'; + +const { tokenChars } = require('./validation'); + +/** + * Adds an offer to the map of extension offers or a parameter to the map of + * parameters. + * + * @param {Object} dest The map of extension offers or parameters + * @param {String} name The extension or parameter name + * @param {(Object|Boolean|String)} elem The extension parameters or the + * parameter value + * @private + */ +function push(dest, name, elem) { + if (dest[name] === undefined) dest[name] = [elem]; + else dest[name].push(elem); +} + +/** + * Parses the `Sec-WebSocket-Extensions` header into an object. + * + * @param {String} header The field value of the header + * @return {Object} The parsed object + * @public + */ +function parse(header) { + const offers = Object.create(null); + let params = Object.create(null); + let mustUnescape = false; + let isEscaping = false; + let inQuotes = false; + let extensionName; + let paramName; + let start = -1; + let code = -1; + let end = -1; + let i = 0; + + for (; i < header.length; i++) { + code = header.charCodeAt(i); + + if (extensionName === undefined) { + if (end === -1 && tokenChars[code] === 1) { + if (start === -1) start = i; + } else if ( + i !== 0 && + (code === 0x20 /* ' ' */ || code === 0x09) /* '\t' */ + ) { + if (end === -1 && start !== -1) end = i; + } else if (code === 0x3b /* ';' */ || code === 0x2c /* ',' */) { + if (start === -1) { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + + if (end === -1) end = i; + const name = header.slice(start, end); + if (code === 0x2c) { + push(offers, name, params); + params = Object.create(null); + } else { + extensionName = name; + } + + start = end = -1; + } else { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + } else if (paramName === undefined) { + if (end === -1 && tokenChars[code] === 1) { + if (start === -1) start = i; + } else if (code === 0x20 || code === 0x09) { + if (end === -1 && start !== -1) end = i; + } else if (code === 0x3b || code === 0x2c) { + if (start === -1) { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + + if (end === -1) end = i; + push(params, header.slice(start, end), true); + if (code === 0x2c) { + push(offers, extensionName, params); + params = Object.create(null); + extensionName = undefined; + } + + start = end = -1; + } else if (code === 0x3d /* '=' */ && start !== -1 && end === -1) { + paramName = header.slice(start, i); + start = end = -1; + } else { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + } else { + // + // The value of a quoted-string after unescaping must conform to the + // token ABNF, so only token characters are valid. + // Ref: https://tools.ietf.org/html/rfc6455#section-9.1 + // + if (isEscaping) { + if (tokenChars[code] !== 1) { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + if (start === -1) start = i; + else if (!mustUnescape) mustUnescape = true; + isEscaping = false; + } else if (inQuotes) { + if (tokenChars[code] === 1) { + if (start === -1) start = i; + } else if (code === 0x22 /* '"' */ && start !== -1) { + inQuotes = false; + end = i; + } else if (code === 0x5c /* '\' */) { + isEscaping = true; + } else { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + } else if (code === 0x22 && header.charCodeAt(i - 1) === 0x3d) { + inQuotes = true; + } else if (end === -1 && tokenChars[code] === 1) { + if (start === -1) start = i; + } else if (start !== -1 && (code === 0x20 || code === 0x09)) { + if (end === -1) end = i; + } else if (code === 0x3b || code === 0x2c) { + if (start === -1) { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + + if (end === -1) end = i; + let value = header.slice(start, end); + if (mustUnescape) { + value = value.replace(/\\/g, ''); + mustUnescape = false; + } + push(params, paramName, value); + if (code === 0x2c) { + push(offers, extensionName, params); + params = Object.create(null); + extensionName = undefined; + } + + paramName = undefined; + start = end = -1; + } else { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + } + } + + if (start === -1 || inQuotes || code === 0x20 || code === 0x09) { + throw new SyntaxError('Unexpected end of input'); + } + + if (end === -1) end = i; + const token = header.slice(start, end); + if (extensionName === undefined) { + push(offers, token, params); + } else { + if (paramName === undefined) { + push(params, token, true); + } else if (mustUnescape) { + push(params, paramName, token.replace(/\\/g, '')); + } else { + push(params, paramName, token); + } + push(offers, extensionName, params); + } + + return offers; +} + +/** + * Builds the `Sec-WebSocket-Extensions` header field value. + * + * @param {Object} extensions The map of extensions and parameters to format + * @return {String} A string representing the given object + * @public + */ +function format(extensions) { + return Object.keys(extensions) + .map((extension) => { + let configurations = extensions[extension]; + if (!Array.isArray(configurations)) configurations = [configurations]; + return configurations + .map((params) => { + return [extension] + .concat( + Object.keys(params).map((k) => { + let values = params[k]; + if (!Array.isArray(values)) values = [values]; + return values + .map((v) => (v === true ? k : `${k}=${v}`)) + .join('; '); + }) + ) + .join('; '); + }) + .join(', '); + }) + .join(', '); +} + +module.exports = { format, parse }; diff --git a/web/node_modules/ws/lib/limiter.js b/web/node_modules/ws/lib/limiter.js new file mode 100644 index 0000000000000000000000000000000000000000..3fd35784ea9ea59cff8c112b6556a89cde7f7b6f --- /dev/null +++ b/web/node_modules/ws/lib/limiter.js @@ -0,0 +1,55 @@ +'use strict'; + +const kDone = Symbol('kDone'); +const kRun = Symbol('kRun'); + +/** + * A very simple job queue with adjustable concurrency. Adapted from + * https://github.com/STRML/async-limiter + */ +class Limiter { + /** + * Creates a new `Limiter`. + * + * @param {Number} [concurrency=Infinity] The maximum number of jobs allowed + * to run concurrently + */ + constructor(concurrency) { + this[kDone] = () => { + this.pending--; + this[kRun](); + }; + this.concurrency = concurrency || Infinity; + this.jobs = []; + this.pending = 0; + } + + /** + * Adds a job to the queue. + * + * @param {Function} job The job to run + * @public + */ + add(job) { + this.jobs.push(job); + this[kRun](); + } + + /** + * Removes a job from the queue and runs it if possible. + * + * @private + */ + [kRun]() { + if (this.pending === this.concurrency) return; + + if (this.jobs.length) { + const job = this.jobs.shift(); + + this.pending++; + job(this[kDone]); + } + } +} + +module.exports = Limiter; diff --git a/web/node_modules/ws/lib/permessage-deflate.js b/web/node_modules/ws/lib/permessage-deflate.js new file mode 100644 index 0000000000000000000000000000000000000000..aa5db761dfa56144c1dc2d5d8c74b36101607f12 --- /dev/null +++ b/web/node_modules/ws/lib/permessage-deflate.js @@ -0,0 +1,528 @@ +'use strict'; + +const zlib = require('zlib'); + +const bufferUtil = require('./buffer-util'); +const Limiter = require('./limiter'); +const { kStatusCode } = require('./constants'); + +const FastBuffer = Buffer[Symbol.species]; +const TRAILER = Buffer.from([0x00, 0x00, 0xff, 0xff]); +const kPerMessageDeflate = Symbol('permessage-deflate'); +const kTotalLength = Symbol('total-length'); +const kCallback = Symbol('callback'); +const kBuffers = Symbol('buffers'); +const kError = Symbol('error'); + +// +// We limit zlib concurrency, which prevents severe memory fragmentation +// as documented in https://github.com/nodejs/node/issues/8871#issuecomment-250915913 +// and https://github.com/websockets/ws/issues/1202 +// +// Intentionally global; it's the global thread pool that's an issue. +// +let zlibLimiter; + +/** + * permessage-deflate implementation. + */ +class PerMessageDeflate { + /** + * Creates a PerMessageDeflate instance. + * + * @param {Object} [options] Configuration options + * @param {(Boolean|Number)} [options.clientMaxWindowBits] Advertise support + * for, or request, a custom client window size + * @param {Boolean} [options.clientNoContextTakeover=false] Advertise/ + * acknowledge disabling of client context takeover + * @param {Number} [options.concurrencyLimit=10] The number of concurrent + * calls to zlib + * @param {Boolean} [options.isServer=false] Create the instance in either + * server or client mode + * @param {Number} [options.maxPayload=0] The maximum allowed message length + * @param {(Boolean|Number)} [options.serverMaxWindowBits] Request/confirm the + * use of a custom server window size + * @param {Boolean} [options.serverNoContextTakeover=false] Request/accept + * disabling of server context takeover + * @param {Number} [options.threshold=1024] Size (in bytes) below which + * messages should not be compressed if context takeover is disabled + * @param {Object} [options.zlibDeflateOptions] Options to pass to zlib on + * deflate + * @param {Object} [options.zlibInflateOptions] Options to pass to zlib on + * inflate + */ + constructor(options) { + this._options = options || {}; + this._threshold = + this._options.threshold !== undefined ? this._options.threshold : 1024; + this._maxPayload = this._options.maxPayload | 0; + this._isServer = !!this._options.isServer; + this._deflate = null; + this._inflate = null; + + this.params = null; + + if (!zlibLimiter) { + const concurrency = + this._options.concurrencyLimit !== undefined + ? this._options.concurrencyLimit + : 10; + zlibLimiter = new Limiter(concurrency); + } + } + + /** + * @type {String} + */ + static get extensionName() { + return 'permessage-deflate'; + } + + /** + * Create an extension negotiation offer. + * + * @return {Object} Extension parameters + * @public + */ + offer() { + const params = {}; + + if (this._options.serverNoContextTakeover) { + params.server_no_context_takeover = true; + } + if (this._options.clientNoContextTakeover) { + params.client_no_context_takeover = true; + } + if (this._options.serverMaxWindowBits) { + params.server_max_window_bits = this._options.serverMaxWindowBits; + } + if (this._options.clientMaxWindowBits) { + params.client_max_window_bits = this._options.clientMaxWindowBits; + } else if (this._options.clientMaxWindowBits == null) { + params.client_max_window_bits = true; + } + + return params; + } + + /** + * Accept an extension negotiation offer/response. + * + * @param {Array} configurations The extension negotiation offers/reponse + * @return {Object} Accepted configuration + * @public + */ + accept(configurations) { + configurations = this.normalizeParams(configurations); + + this.params = this._isServer + ? this.acceptAsServer(configurations) + : this.acceptAsClient(configurations); + + return this.params; + } + + /** + * Releases all resources used by the extension. + * + * @public + */ + cleanup() { + if (this._inflate) { + this._inflate.close(); + this._inflate = null; + } + + if (this._deflate) { + const callback = this._deflate[kCallback]; + + this._deflate.close(); + this._deflate = null; + + if (callback) { + callback( + new Error( + 'The deflate stream was closed while data was being processed' + ) + ); + } + } + } + + /** + * Accept an extension negotiation offer. + * + * @param {Array} offers The extension negotiation offers + * @return {Object} Accepted configuration + * @private + */ + acceptAsServer(offers) { + const opts = this._options; + const accepted = offers.find((params) => { + if ( + (opts.serverNoContextTakeover === false && + params.server_no_context_takeover) || + (params.server_max_window_bits && + (opts.serverMaxWindowBits === false || + (typeof opts.serverMaxWindowBits === 'number' && + opts.serverMaxWindowBits > params.server_max_window_bits))) || + (typeof opts.clientMaxWindowBits === 'number' && + !params.client_max_window_bits) + ) { + return false; + } + + return true; + }); + + if (!accepted) { + throw new Error('None of the extension offers can be accepted'); + } + + if (opts.serverNoContextTakeover) { + accepted.server_no_context_takeover = true; + } + if (opts.clientNoContextTakeover) { + accepted.client_no_context_takeover = true; + } + if (typeof opts.serverMaxWindowBits === 'number') { + accepted.server_max_window_bits = opts.serverMaxWindowBits; + } + if (typeof opts.clientMaxWindowBits === 'number') { + accepted.client_max_window_bits = opts.clientMaxWindowBits; + } else if ( + accepted.client_max_window_bits === true || + opts.clientMaxWindowBits === false + ) { + delete accepted.client_max_window_bits; + } + + return accepted; + } + + /** + * Accept the extension negotiation response. + * + * @param {Array} response The extension negotiation response + * @return {Object} Accepted configuration + * @private + */ + acceptAsClient(response) { + const params = response[0]; + + if ( + this._options.clientNoContextTakeover === false && + params.client_no_context_takeover + ) { + throw new Error('Unexpected parameter "client_no_context_takeover"'); + } + + if (!params.client_max_window_bits) { + if (typeof this._options.clientMaxWindowBits === 'number') { + params.client_max_window_bits = this._options.clientMaxWindowBits; + } + } else if ( + this._options.clientMaxWindowBits === false || + (typeof this._options.clientMaxWindowBits === 'number' && + params.client_max_window_bits > this._options.clientMaxWindowBits) + ) { + throw new Error( + 'Unexpected or invalid parameter "client_max_window_bits"' + ); + } + + return params; + } + + /** + * Normalize parameters. + * + * @param {Array} configurations The extension negotiation offers/reponse + * @return {Array} The offers/response with normalized parameters + * @private + */ + normalizeParams(configurations) { + configurations.forEach((params) => { + Object.keys(params).forEach((key) => { + let value = params[key]; + + if (value.length > 1) { + throw new Error(`Parameter "${key}" must have only a single value`); + } + + value = value[0]; + + if (key === 'client_max_window_bits') { + if (value !== true) { + const num = +value; + if (!Number.isInteger(num) || num < 8 || num > 15) { + throw new TypeError( + `Invalid value for parameter "${key}": ${value}` + ); + } + value = num; + } else if (!this._isServer) { + throw new TypeError( + `Invalid value for parameter "${key}": ${value}` + ); + } + } else if (key === 'server_max_window_bits') { + const num = +value; + if (!Number.isInteger(num) || num < 8 || num > 15) { + throw new TypeError( + `Invalid value for parameter "${key}": ${value}` + ); + } + value = num; + } else if ( + key === 'client_no_context_takeover' || + key === 'server_no_context_takeover' + ) { + if (value !== true) { + throw new TypeError( + `Invalid value for parameter "${key}": ${value}` + ); + } + } else { + throw new Error(`Unknown parameter "${key}"`); + } + + params[key] = value; + }); + }); + + return configurations; + } + + /** + * Decompress data. Concurrency limited. + * + * @param {Buffer} data Compressed data + * @param {Boolean} fin Specifies whether or not this is the last fragment + * @param {Function} callback Callback + * @public + */ + decompress(data, fin, callback) { + zlibLimiter.add((done) => { + this._decompress(data, fin, (err, result) => { + done(); + callback(err, result); + }); + }); + } + + /** + * Compress data. Concurrency limited. + * + * @param {(Buffer|String)} data Data to compress + * @param {Boolean} fin Specifies whether or not this is the last fragment + * @param {Function} callback Callback + * @public + */ + compress(data, fin, callback) { + zlibLimiter.add((done) => { + this._compress(data, fin, (err, result) => { + done(); + callback(err, result); + }); + }); + } + + /** + * Decompress data. + * + * @param {Buffer} data Compressed data + * @param {Boolean} fin Specifies whether or not this is the last fragment + * @param {Function} callback Callback + * @private + */ + _decompress(data, fin, callback) { + const endpoint = this._isServer ? 'client' : 'server'; + + if (!this._inflate) { + const key = `${endpoint}_max_window_bits`; + const windowBits = + typeof this.params[key] !== 'number' + ? zlib.Z_DEFAULT_WINDOWBITS + : this.params[key]; + + this._inflate = zlib.createInflateRaw({ + ...this._options.zlibInflateOptions, + windowBits + }); + this._inflate[kPerMessageDeflate] = this; + this._inflate[kTotalLength] = 0; + this._inflate[kBuffers] = []; + this._inflate.on('error', inflateOnError); + this._inflate.on('data', inflateOnData); + } + + this._inflate[kCallback] = callback; + + this._inflate.write(data); + if (fin) this._inflate.write(TRAILER); + + this._inflate.flush(() => { + const err = this._inflate[kError]; + + if (err) { + this._inflate.close(); + this._inflate = null; + callback(err); + return; + } + + const data = bufferUtil.concat( + this._inflate[kBuffers], + this._inflate[kTotalLength] + ); + + if (this._inflate._readableState.endEmitted) { + this._inflate.close(); + this._inflate = null; + } else { + this._inflate[kTotalLength] = 0; + this._inflate[kBuffers] = []; + + if (fin && this.params[`${endpoint}_no_context_takeover`]) { + this._inflate.reset(); + } + } + + callback(null, data); + }); + } + + /** + * Compress data. + * + * @param {(Buffer|String)} data Data to compress + * @param {Boolean} fin Specifies whether or not this is the last fragment + * @param {Function} callback Callback + * @private + */ + _compress(data, fin, callback) { + const endpoint = this._isServer ? 'server' : 'client'; + + if (!this._deflate) { + const key = `${endpoint}_max_window_bits`; + const windowBits = + typeof this.params[key] !== 'number' + ? zlib.Z_DEFAULT_WINDOWBITS + : this.params[key]; + + this._deflate = zlib.createDeflateRaw({ + ...this._options.zlibDeflateOptions, + windowBits + }); + + this._deflate[kTotalLength] = 0; + this._deflate[kBuffers] = []; + + this._deflate.on('data', deflateOnData); + } + + this._deflate[kCallback] = callback; + + this._deflate.write(data); + this._deflate.flush(zlib.Z_SYNC_FLUSH, () => { + if (!this._deflate) { + // + // The deflate stream was closed while data was being processed. + // + return; + } + + let data = bufferUtil.concat( + this._deflate[kBuffers], + this._deflate[kTotalLength] + ); + + if (fin) { + data = new FastBuffer(data.buffer, data.byteOffset, data.length - 4); + } + + // + // Ensure that the callback will not be called again in + // `PerMessageDeflate#cleanup()`. + // + this._deflate[kCallback] = null; + + this._deflate[kTotalLength] = 0; + this._deflate[kBuffers] = []; + + if (fin && this.params[`${endpoint}_no_context_takeover`]) { + this._deflate.reset(); + } + + callback(null, data); + }); + } +} + +module.exports = PerMessageDeflate; + +/** + * The listener of the `zlib.DeflateRaw` stream `'data'` event. + * + * @param {Buffer} chunk A chunk of data + * @private + */ +function deflateOnData(chunk) { + this[kBuffers].push(chunk); + this[kTotalLength] += chunk.length; +} + +/** + * The listener of the `zlib.InflateRaw` stream `'data'` event. + * + * @param {Buffer} chunk A chunk of data + * @private + */ +function inflateOnData(chunk) { + this[kTotalLength] += chunk.length; + + if ( + this[kPerMessageDeflate]._maxPayload < 1 || + this[kTotalLength] <= this[kPerMessageDeflate]._maxPayload + ) { + this[kBuffers].push(chunk); + return; + } + + this[kError] = new RangeError('Max payload size exceeded'); + this[kError].code = 'WS_ERR_UNSUPPORTED_MESSAGE_LENGTH'; + this[kError][kStatusCode] = 1009; + this.removeListener('data', inflateOnData); + + // + // The choice to employ `zlib.reset()` over `zlib.close()` is dictated by the + // fact that in Node.js versions prior to 13.10.0, the callback for + // `zlib.flush()` is not called if `zlib.close()` is used. Utilizing + // `zlib.reset()` ensures that either the callback is invoked or an error is + // emitted. + // + this.reset(); +} + +/** + * The listener of the `zlib.InflateRaw` stream `'error'` event. + * + * @param {Error} err The emitted error + * @private + */ +function inflateOnError(err) { + // + // There is no need to call `Zlib#close()` as the handle is automatically + // closed when an error is emitted. + // + this[kPerMessageDeflate]._inflate = null; + + if (this[kError]) { + this[kCallback](this[kError]); + return; + } + + err[kStatusCode] = 1007; + this[kCallback](err); +} diff --git a/web/node_modules/ws/lib/receiver.js b/web/node_modules/ws/lib/receiver.js new file mode 100644 index 0000000000000000000000000000000000000000..1f8f0286cd460f83a49f21adbb3df8671af2e3d2 --- /dev/null +++ b/web/node_modules/ws/lib/receiver.js @@ -0,0 +1,760 @@ +'use strict'; + +const { Writable } = require('stream'); + +const PerMessageDeflate = require('./permessage-deflate'); +const { + BINARY_TYPES, + EMPTY_BUFFER, + kStatusCode, + kWebSocket +} = require('./constants'); +const { concat, toArrayBuffer, unmask } = require('./buffer-util'); +const { isValidStatusCode, isValidUTF8 } = require('./validation'); + +const FastBuffer = Buffer[Symbol.species]; + +const GET_INFO = 0; +const GET_PAYLOAD_LENGTH_16 = 1; +const GET_PAYLOAD_LENGTH_64 = 2; +const GET_MASK = 3; +const GET_DATA = 4; +const INFLATING = 5; +const DEFER_EVENT = 6; + +/** + * HyBi Receiver implementation. + * + * @extends Writable + */ +class Receiver extends Writable { + /** + * Creates a Receiver instance. + * + * @param {Object} [options] Options object + * @param {Boolean} [options.allowSynchronousEvents=true] Specifies whether + * any of the `'message'`, `'ping'`, and `'pong'` events can be emitted + * multiple times in the same tick + * @param {String} [options.binaryType=nodebuffer] The type for binary data + * @param {Object} [options.extensions] An object containing the negotiated + * extensions + * @param {Boolean} [options.isServer=false] Specifies whether to operate in + * client or server mode + * @param {Number} [options.maxBufferedChunks=0] The maximum number of + * buffered data chunks + * @param {Number} [options.maxFragments=0] The maximum number of message + * fragments + * @param {Number} [options.maxPayload=0] The maximum allowed message length + * @param {Boolean} [options.skipUTF8Validation=false] Specifies whether or + * not to skip UTF-8 validation for text and close messages + */ + constructor(options = {}) { + super(); + + this._allowSynchronousEvents = + options.allowSynchronousEvents !== undefined + ? options.allowSynchronousEvents + : true; + this._binaryType = options.binaryType || BINARY_TYPES[0]; + this._extensions = options.extensions || {}; + this._isServer = !!options.isServer; + this._maxBufferedChunks = options.maxBufferedChunks | 0; + this._maxFragments = options.maxFragments | 0; + this._maxPayload = options.maxPayload | 0; + this._skipUTF8Validation = !!options.skipUTF8Validation; + this[kWebSocket] = undefined; + + this._bufferedBytes = 0; + this._buffers = []; + + this._compressed = false; + this._payloadLength = 0; + this._mask = undefined; + this._fragmented = 0; + this._masked = false; + this._fin = false; + this._opcode = 0; + + this._totalPayloadLength = 0; + this._messageLength = 0; + this._fragments = []; + + this._errored = false; + this._loop = false; + this._state = GET_INFO; + } + + /** + * Implements `Writable.prototype._write()`. + * + * @param {Buffer} chunk The chunk of data to write + * @param {String} encoding The character encoding of `chunk` + * @param {Function} cb Callback + * @private + */ + _write(chunk, encoding, cb) { + if (this._opcode === 0x08 && this._state == GET_INFO) return cb(); + + if ( + this._maxBufferedChunks > 0 && + this._buffers.length >= this._maxBufferedChunks + ) { + cb( + this.createError( + RangeError, + 'Too many buffered chunks', + false, + 1008, + 'WS_ERR_TOO_MANY_BUFFERED_PARTS' + ) + ); + return; + } + + this._bufferedBytes += chunk.length; + this._buffers.push(chunk); + this.startLoop(cb); + } + + /** + * Consumes `n` bytes from the buffered data. + * + * @param {Number} n The number of bytes to consume + * @return {Buffer} The consumed bytes + * @private + */ + consume(n) { + this._bufferedBytes -= n; + + if (n === this._buffers[0].length) return this._buffers.shift(); + + if (n < this._buffers[0].length) { + const buf = this._buffers[0]; + this._buffers[0] = new FastBuffer( + buf.buffer, + buf.byteOffset + n, + buf.length - n + ); + + return new FastBuffer(buf.buffer, buf.byteOffset, n); + } + + const dst = Buffer.allocUnsafe(n); + + do { + const buf = this._buffers[0]; + const offset = dst.length - n; + + if (n >= buf.length) { + dst.set(this._buffers.shift(), offset); + } else { + dst.set(new Uint8Array(buf.buffer, buf.byteOffset, n), offset); + this._buffers[0] = new FastBuffer( + buf.buffer, + buf.byteOffset + n, + buf.length - n + ); + } + + n -= buf.length; + } while (n > 0); + + return dst; + } + + /** + * Starts the parsing loop. + * + * @param {Function} cb Callback + * @private + */ + startLoop(cb) { + this._loop = true; + + do { + switch (this._state) { + case GET_INFO: + this.getInfo(cb); + break; + case GET_PAYLOAD_LENGTH_16: + this.getPayloadLength16(cb); + break; + case GET_PAYLOAD_LENGTH_64: + this.getPayloadLength64(cb); + break; + case GET_MASK: + this.getMask(); + break; + case GET_DATA: + this.getData(cb); + break; + case INFLATING: + case DEFER_EVENT: + this._loop = false; + return; + } + } while (this._loop); + + if (!this._errored) cb(); + } + + /** + * Reads the first two bytes of a frame. + * + * @param {Function} cb Callback + * @private + */ + getInfo(cb) { + if (this._bufferedBytes < 2) { + this._loop = false; + return; + } + + const buf = this.consume(2); + + if ((buf[0] & 0x30) !== 0x00) { + const error = this.createError( + RangeError, + 'RSV2 and RSV3 must be clear', + true, + 1002, + 'WS_ERR_UNEXPECTED_RSV_2_3' + ); + + cb(error); + return; + } + + const compressed = (buf[0] & 0x40) === 0x40; + + if (compressed && !this._extensions[PerMessageDeflate.extensionName]) { + const error = this.createError( + RangeError, + 'RSV1 must be clear', + true, + 1002, + 'WS_ERR_UNEXPECTED_RSV_1' + ); + + cb(error); + return; + } + + this._fin = (buf[0] & 0x80) === 0x80; + this._opcode = buf[0] & 0x0f; + this._payloadLength = buf[1] & 0x7f; + + if (this._opcode === 0x00) { + if (compressed) { + const error = this.createError( + RangeError, + 'RSV1 must be clear', + true, + 1002, + 'WS_ERR_UNEXPECTED_RSV_1' + ); + + cb(error); + return; + } + + if (!this._fragmented) { + const error = this.createError( + RangeError, + 'invalid opcode 0', + true, + 1002, + 'WS_ERR_INVALID_OPCODE' + ); + + cb(error); + return; + } + + this._opcode = this._fragmented; + } else if (this._opcode === 0x01 || this._opcode === 0x02) { + if (this._fragmented) { + const error = this.createError( + RangeError, + `invalid opcode ${this._opcode}`, + true, + 1002, + 'WS_ERR_INVALID_OPCODE' + ); + + cb(error); + return; + } + + this._compressed = compressed; + } else if (this._opcode > 0x07 && this._opcode < 0x0b) { + if (!this._fin) { + const error = this.createError( + RangeError, + 'FIN must be set', + true, + 1002, + 'WS_ERR_EXPECTED_FIN' + ); + + cb(error); + return; + } + + if (compressed) { + const error = this.createError( + RangeError, + 'RSV1 must be clear', + true, + 1002, + 'WS_ERR_UNEXPECTED_RSV_1' + ); + + cb(error); + return; + } + + if ( + this._payloadLength > 0x7d || + (this._opcode === 0x08 && this._payloadLength === 1) + ) { + const error = this.createError( + RangeError, + `invalid payload length ${this._payloadLength}`, + true, + 1002, + 'WS_ERR_INVALID_CONTROL_PAYLOAD_LENGTH' + ); + + cb(error); + return; + } + } else { + const error = this.createError( + RangeError, + `invalid opcode ${this._opcode}`, + true, + 1002, + 'WS_ERR_INVALID_OPCODE' + ); + + cb(error); + return; + } + + if (!this._fin && !this._fragmented) this._fragmented = this._opcode; + this._masked = (buf[1] & 0x80) === 0x80; + + if (this._isServer) { + if (!this._masked) { + const error = this.createError( + RangeError, + 'MASK must be set', + true, + 1002, + 'WS_ERR_EXPECTED_MASK' + ); + + cb(error); + return; + } + } else if (this._masked) { + const error = this.createError( + RangeError, + 'MASK must be clear', + true, + 1002, + 'WS_ERR_UNEXPECTED_MASK' + ); + + cb(error); + return; + } + + if (this._payloadLength === 126) this._state = GET_PAYLOAD_LENGTH_16; + else if (this._payloadLength === 127) this._state = GET_PAYLOAD_LENGTH_64; + else this.haveLength(cb); + } + + /** + * Gets extended payload length (7+16). + * + * @param {Function} cb Callback + * @private + */ + getPayloadLength16(cb) { + if (this._bufferedBytes < 2) { + this._loop = false; + return; + } + + this._payloadLength = this.consume(2).readUInt16BE(0); + this.haveLength(cb); + } + + /** + * Gets extended payload length (7+64). + * + * @param {Function} cb Callback + * @private + */ + getPayloadLength64(cb) { + if (this._bufferedBytes < 8) { + this._loop = false; + return; + } + + const buf = this.consume(8); + const num = buf.readUInt32BE(0); + + // + // The maximum safe integer in JavaScript is 2^53 - 1. An error is returned + // if payload length is greater than this number. + // + if (num > Math.pow(2, 53 - 32) - 1) { + const error = this.createError( + RangeError, + 'Unsupported WebSocket frame: payload length > 2^53 - 1', + false, + 1009, + 'WS_ERR_UNSUPPORTED_DATA_PAYLOAD_LENGTH' + ); + + cb(error); + return; + } + + this._payloadLength = num * Math.pow(2, 32) + buf.readUInt32BE(4); + this.haveLength(cb); + } + + /** + * Payload length has been read. + * + * @param {Function} cb Callback + * @private + */ + haveLength(cb) { + if (this._payloadLength && this._opcode < 0x08) { + this._totalPayloadLength += this._payloadLength; + if (this._totalPayloadLength > this._maxPayload && this._maxPayload > 0) { + const error = this.createError( + RangeError, + 'Max payload size exceeded', + false, + 1009, + 'WS_ERR_UNSUPPORTED_MESSAGE_LENGTH' + ); + + cb(error); + return; + } + } + + if (this._masked) this._state = GET_MASK; + else this._state = GET_DATA; + } + + /** + * Reads mask bytes. + * + * @private + */ + getMask() { + if (this._bufferedBytes < 4) { + this._loop = false; + return; + } + + this._mask = this.consume(4); + this._state = GET_DATA; + } + + /** + * Reads data bytes. + * + * @param {Function} cb Callback + * @private + */ + getData(cb) { + let data = EMPTY_BUFFER; + + if (this._payloadLength) { + if (this._bufferedBytes < this._payloadLength) { + this._loop = false; + return; + } + + data = this.consume(this._payloadLength); + + if ( + this._masked && + (this._mask[0] | this._mask[1] | this._mask[2] | this._mask[3]) !== 0 + ) { + unmask(data, this._mask); + } + } + + if (this._opcode > 0x07) { + this.controlMessage(data, cb); + return; + } + + if (this._compressed) { + this._state = INFLATING; + this.decompress(data, cb); + return; + } + + if (data.length) { + if ( + this._maxFragments > 0 && + this._fragments.length >= this._maxFragments + ) { + const error = this.createError( + RangeError, + 'Too many message fragments', + false, + 1008, + 'WS_ERR_TOO_MANY_BUFFERED_PARTS' + ); + + cb(error); + return; + } + + // + // This message is not compressed so its length is the sum of the payload + // length of all fragments. + // + this._messageLength = this._totalPayloadLength; + this._fragments.push(data); + } + + this.dataMessage(cb); + } + + /** + * Decompresses data. + * + * @param {Buffer} data Compressed data + * @param {Function} cb Callback + * @private + */ + decompress(data, cb) { + const perMessageDeflate = this._extensions[PerMessageDeflate.extensionName]; + + perMessageDeflate.decompress(data, this._fin, (err, buf) => { + if (err) return cb(err); + + if (buf.length) { + this._messageLength += buf.length; + if (this._messageLength > this._maxPayload && this._maxPayload > 0) { + const error = this.createError( + RangeError, + 'Max payload size exceeded', + false, + 1009, + 'WS_ERR_UNSUPPORTED_MESSAGE_LENGTH' + ); + + cb(error); + return; + } + + if ( + this._maxFragments > 0 && + this._fragments.length >= this._maxFragments + ) { + const error = this.createError( + RangeError, + 'Too many message fragments', + false, + 1008, + 'WS_ERR_TOO_MANY_BUFFERED_PARTS' + ); + + cb(error); + return; + } + + this._fragments.push(buf); + } + + this.dataMessage(cb); + if (this._state === GET_INFO) this.startLoop(cb); + }); + } + + /** + * Handles a data message. + * + * @param {Function} cb Callback + * @private + */ + dataMessage(cb) { + if (!this._fin) { + this._state = GET_INFO; + return; + } + + const messageLength = this._messageLength; + const fragments = this._fragments; + + this._totalPayloadLength = 0; + this._messageLength = 0; + this._fragmented = 0; + this._fragments = []; + + if (this._opcode === 2) { + let data; + + if (this._binaryType === 'nodebuffer') { + data = concat(fragments, messageLength); + } else if (this._binaryType === 'arraybuffer') { + data = toArrayBuffer(concat(fragments, messageLength)); + } else if (this._binaryType === 'blob') { + data = new Blob(fragments); + } else { + data = fragments; + } + + if (this._allowSynchronousEvents) { + this.emit('message', data, true); + this._state = GET_INFO; + } else { + this._state = DEFER_EVENT; + setImmediate(() => { + this.emit('message', data, true); + this._state = GET_INFO; + this.startLoop(cb); + }); + } + } else { + const buf = concat(fragments, messageLength); + + if (!this._skipUTF8Validation && !isValidUTF8(buf)) { + const error = this.createError( + Error, + 'invalid UTF-8 sequence', + true, + 1007, + 'WS_ERR_INVALID_UTF8' + ); + + cb(error); + return; + } + + if (this._state === INFLATING || this._allowSynchronousEvents) { + this.emit('message', buf, false); + this._state = GET_INFO; + } else { + this._state = DEFER_EVENT; + setImmediate(() => { + this.emit('message', buf, false); + this._state = GET_INFO; + this.startLoop(cb); + }); + } + } + } + + /** + * Handles a control message. + * + * @param {Buffer} data Data to handle + * @return {(Error|RangeError|undefined)} A possible error + * @private + */ + controlMessage(data, cb) { + if (this._opcode === 0x08) { + if (data.length === 0) { + this._loop = false; + this.emit('conclude', 1005, EMPTY_BUFFER); + this.end(); + } else { + const code = data.readUInt16BE(0); + + if (!isValidStatusCode(code)) { + const error = this.createError( + RangeError, + `invalid status code ${code}`, + true, + 1002, + 'WS_ERR_INVALID_CLOSE_CODE' + ); + + cb(error); + return; + } + + const buf = new FastBuffer( + data.buffer, + data.byteOffset + 2, + data.length - 2 + ); + + if (!this._skipUTF8Validation && !isValidUTF8(buf)) { + const error = this.createError( + Error, + 'invalid UTF-8 sequence', + true, + 1007, + 'WS_ERR_INVALID_UTF8' + ); + + cb(error); + return; + } + + this._loop = false; + this.emit('conclude', code, buf); + this.end(); + } + + this._state = GET_INFO; + return; + } + + if (this._allowSynchronousEvents) { + this.emit(this._opcode === 0x09 ? 'ping' : 'pong', data); + this._state = GET_INFO; + } else { + this._state = DEFER_EVENT; + setImmediate(() => { + this.emit(this._opcode === 0x09 ? 'ping' : 'pong', data); + this._state = GET_INFO; + this.startLoop(cb); + }); + } + } + + /** + * Builds an error object. + * + * @param {function(new:Error|RangeError)} ErrorCtor The error constructor + * @param {String} message The error message + * @param {Boolean} prefix Specifies whether or not to add a default prefix to + * `message` + * @param {Number} statusCode The status code + * @param {String} errorCode The exposed error code + * @return {(Error|RangeError)} The error + * @private + */ + createError(ErrorCtor, message, prefix, statusCode, errorCode) { + this._loop = false; + this._errored = true; + + const err = new ErrorCtor( + prefix ? `Invalid WebSocket frame: ${message}` : message + ); + + Error.captureStackTrace(err, this.createError); + err.code = errorCode; + err[kStatusCode] = statusCode; + return err; + } +} + +module.exports = Receiver; diff --git a/web/node_modules/ws/lib/sender.js b/web/node_modules/ws/lib/sender.js new file mode 100644 index 0000000000000000000000000000000000000000..2c767e2c62a7fb50a987454f0c148ec5b4a0a523 --- /dev/null +++ b/web/node_modules/ws/lib/sender.js @@ -0,0 +1,607 @@ +/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "^Duplex" }] */ + +'use strict'; + +const { Duplex } = require('stream'); +const { randomFillSync } = require('crypto'); +const { + types: { isUint8Array } +} = require('util'); + +const PerMessageDeflate = require('./permessage-deflate'); +const { EMPTY_BUFFER, kWebSocket, NOOP } = require('./constants'); +const { isBlob, isValidStatusCode } = require('./validation'); +const { mask: applyMask, toBuffer } = require('./buffer-util'); + +const kByteLength = Symbol('kByteLength'); +const maskBuffer = Buffer.alloc(4); +const RANDOM_POOL_SIZE = 8 * 1024; +let randomPool; +let randomPoolPointer = RANDOM_POOL_SIZE; + +const DEFAULT = 0; +const DEFLATING = 1; +const GET_BLOB_DATA = 2; + +/** + * HyBi Sender implementation. + */ +class Sender { + /** + * Creates a Sender instance. + * + * @param {Duplex} socket The connection socket + * @param {Object} [extensions] An object containing the negotiated extensions + * @param {Function} [generateMask] The function used to generate the masking + * key + */ + constructor(socket, extensions, generateMask) { + this._extensions = extensions || {}; + + if (generateMask) { + this._generateMask = generateMask; + this._maskBuffer = Buffer.alloc(4); + } + + this._socket = socket; + + this._firstFragment = true; + this._compress = false; + + this._bufferedBytes = 0; + this._queue = []; + this._state = DEFAULT; + this.onerror = NOOP; + this[kWebSocket] = undefined; + } + + /** + * Frames a piece of data according to the HyBi WebSocket protocol. + * + * @param {(Buffer|String)} data The data to frame + * @param {Object} options Options object + * @param {Boolean} [options.fin=false] Specifies whether or not to set the + * FIN bit + * @param {Function} [options.generateMask] The function used to generate the + * masking key + * @param {Boolean} [options.mask=false] Specifies whether or not to mask + * `data` + * @param {Buffer} [options.maskBuffer] The buffer used to store the masking + * key + * @param {Number} options.opcode The opcode + * @param {Boolean} [options.readOnly=false] Specifies whether `data` can be + * modified + * @param {Boolean} [options.rsv1=false] Specifies whether or not to set the + * RSV1 bit + * @return {(Buffer|String)[]} The framed data + * @public + */ + static frame(data, options) { + let mask; + let merge = false; + let offset = 2; + let skipMasking = false; + + if (options.mask) { + mask = options.maskBuffer || maskBuffer; + + if (options.generateMask) { + options.generateMask(mask); + } else { + if (randomPoolPointer === RANDOM_POOL_SIZE) { + /* istanbul ignore else */ + if (randomPool === undefined) { + // + // This is lazily initialized because server-sent frames must not + // be masked so it may never be used. + // + randomPool = Buffer.alloc(RANDOM_POOL_SIZE); + } + + randomFillSync(randomPool, 0, RANDOM_POOL_SIZE); + randomPoolPointer = 0; + } + + mask[0] = randomPool[randomPoolPointer++]; + mask[1] = randomPool[randomPoolPointer++]; + mask[2] = randomPool[randomPoolPointer++]; + mask[3] = randomPool[randomPoolPointer++]; + } + + skipMasking = (mask[0] | mask[1] | mask[2] | mask[3]) === 0; + offset = 6; + } + + let dataLength; + + if (typeof data === 'string') { + if ( + (!options.mask || skipMasking) && + options[kByteLength] !== undefined + ) { + dataLength = options[kByteLength]; + } else { + data = Buffer.from(data); + dataLength = data.length; + } + } else { + dataLength = data.length; + merge = options.mask && options.readOnly && !skipMasking; + } + + let payloadLength = dataLength; + + if (dataLength >= 65536) { + offset += 8; + payloadLength = 127; + } else if (dataLength > 125) { + offset += 2; + payloadLength = 126; + } + + const target = Buffer.allocUnsafe(merge ? dataLength + offset : offset); + + target[0] = options.fin ? options.opcode | 0x80 : options.opcode; + if (options.rsv1) target[0] |= 0x40; + + target[1] = payloadLength; + + if (payloadLength === 126) { + target.writeUInt16BE(dataLength, 2); + } else if (payloadLength === 127) { + target[2] = target[3] = 0; + target.writeUIntBE(dataLength, 4, 6); + } + + if (!options.mask) return [target, data]; + + target[1] |= 0x80; + target[offset - 4] = mask[0]; + target[offset - 3] = mask[1]; + target[offset - 2] = mask[2]; + target[offset - 1] = mask[3]; + + if (skipMasking) return [target, data]; + + if (merge) { + applyMask(data, mask, target, offset, dataLength); + return [target]; + } + + applyMask(data, mask, data, 0, dataLength); + return [target, data]; + } + + /** + * Sends a close message to the other peer. + * + * @param {Number} [code] The status code component of the body + * @param {(String|Buffer)} [data] The message component of the body + * @param {Boolean} [mask=false] Specifies whether or not to mask the message + * @param {Function} [cb] Callback + * @public + */ + close(code, data, mask, cb) { + let buf; + + if (code === undefined) { + buf = EMPTY_BUFFER; + } else if (typeof code !== 'number' || !isValidStatusCode(code)) { + throw new TypeError('First argument must be a valid error code number'); + } else if (data === undefined || !data.length) { + buf = Buffer.allocUnsafe(2); + buf.writeUInt16BE(code, 0); + } else { + const length = Buffer.byteLength(data); + + if (length > 123) { + throw new RangeError('The message must not be greater than 123 bytes'); + } + + buf = Buffer.allocUnsafe(2 + length); + buf.writeUInt16BE(code, 0); + + if (typeof data === 'string') { + buf.write(data, 2); + } else if (isUint8Array(data)) { + buf.set(data, 2); + } else { + throw new TypeError('Second argument must be a string or a Uint8Array'); + } + } + + const options = { + [kByteLength]: buf.length, + fin: true, + generateMask: this._generateMask, + mask, + maskBuffer: this._maskBuffer, + opcode: 0x08, + readOnly: false, + rsv1: false + }; + + if (this._state !== DEFAULT) { + this.enqueue([this.dispatch, buf, false, options, cb]); + } else { + this.sendFrame(Sender.frame(buf, options), cb); + } + } + + /** + * Sends a ping message to the other peer. + * + * @param {*} data The message to send + * @param {Boolean} [mask=false] Specifies whether or not to mask `data` + * @param {Function} [cb] Callback + * @public + */ + ping(data, mask, cb) { + let byteLength; + let readOnly; + + if (typeof data === 'string') { + byteLength = Buffer.byteLength(data); + readOnly = false; + } else if (isBlob(data)) { + byteLength = data.size; + readOnly = false; + } else { + data = toBuffer(data); + byteLength = data.length; + readOnly = toBuffer.readOnly; + } + + if (byteLength > 125) { + throw new RangeError('The data size must not be greater than 125 bytes'); + } + + const options = { + [kByteLength]: byteLength, + fin: true, + generateMask: this._generateMask, + mask, + maskBuffer: this._maskBuffer, + opcode: 0x09, + readOnly, + rsv1: false + }; + + if (isBlob(data)) { + if (this._state !== DEFAULT) { + this.enqueue([this.getBlobData, data, false, options, cb]); + } else { + this.getBlobData(data, false, options, cb); + } + } else if (this._state !== DEFAULT) { + this.enqueue([this.dispatch, data, false, options, cb]); + } else { + this.sendFrame(Sender.frame(data, options), cb); + } + } + + /** + * Sends a pong message to the other peer. + * + * @param {*} data The message to send + * @param {Boolean} [mask=false] Specifies whether or not to mask `data` + * @param {Function} [cb] Callback + * @public + */ + pong(data, mask, cb) { + let byteLength; + let readOnly; + + if (typeof data === 'string') { + byteLength = Buffer.byteLength(data); + readOnly = false; + } else if (isBlob(data)) { + byteLength = data.size; + readOnly = false; + } else { + data = toBuffer(data); + byteLength = data.length; + readOnly = toBuffer.readOnly; + } + + if (byteLength > 125) { + throw new RangeError('The data size must not be greater than 125 bytes'); + } + + const options = { + [kByteLength]: byteLength, + fin: true, + generateMask: this._generateMask, + mask, + maskBuffer: this._maskBuffer, + opcode: 0x0a, + readOnly, + rsv1: false + }; + + if (isBlob(data)) { + if (this._state !== DEFAULT) { + this.enqueue([this.getBlobData, data, false, options, cb]); + } else { + this.getBlobData(data, false, options, cb); + } + } else if (this._state !== DEFAULT) { + this.enqueue([this.dispatch, data, false, options, cb]); + } else { + this.sendFrame(Sender.frame(data, options), cb); + } + } + + /** + * Sends a data message to the other peer. + * + * @param {*} data The message to send + * @param {Object} options Options object + * @param {Boolean} [options.binary=false] Specifies whether `data` is binary + * or text + * @param {Boolean} [options.compress=false] Specifies whether or not to + * compress `data` + * @param {Boolean} [options.fin=false] Specifies whether the fragment is the + * last one + * @param {Boolean} [options.mask=false] Specifies whether or not to mask + * `data` + * @param {Function} [cb] Callback + * @public + */ + send(data, options, cb) { + const perMessageDeflate = this._extensions[PerMessageDeflate.extensionName]; + let opcode = options.binary ? 2 : 1; + let rsv1 = options.compress; + + let byteLength; + let readOnly; + + if (typeof data === 'string') { + byteLength = Buffer.byteLength(data); + readOnly = false; + } else if (isBlob(data)) { + byteLength = data.size; + readOnly = false; + } else { + data = toBuffer(data); + byteLength = data.length; + readOnly = toBuffer.readOnly; + } + + if (this._firstFragment) { + this._firstFragment = false; + if ( + rsv1 && + perMessageDeflate && + perMessageDeflate.params[ + perMessageDeflate._isServer + ? 'server_no_context_takeover' + : 'client_no_context_takeover' + ] + ) { + rsv1 = byteLength >= perMessageDeflate._threshold; + } + this._compress = rsv1; + } else { + rsv1 = false; + opcode = 0; + } + + if (options.fin) this._firstFragment = true; + + const opts = { + [kByteLength]: byteLength, + fin: options.fin, + generateMask: this._generateMask, + mask: options.mask, + maskBuffer: this._maskBuffer, + opcode, + readOnly, + rsv1 + }; + + if (isBlob(data)) { + if (this._state !== DEFAULT) { + this.enqueue([this.getBlobData, data, this._compress, opts, cb]); + } else { + this.getBlobData(data, this._compress, opts, cb); + } + } else if (this._state !== DEFAULT) { + this.enqueue([this.dispatch, data, this._compress, opts, cb]); + } else { + this.dispatch(data, this._compress, opts, cb); + } + } + + /** + * Gets the contents of a blob as binary data. + * + * @param {Blob} blob The blob + * @param {Boolean} [compress=false] Specifies whether or not to compress + * the data + * @param {Object} options Options object + * @param {Boolean} [options.fin=false] Specifies whether or not to set the + * FIN bit + * @param {Function} [options.generateMask] The function used to generate the + * masking key + * @param {Boolean} [options.mask=false] Specifies whether or not to mask + * `data` + * @param {Buffer} [options.maskBuffer] The buffer used to store the masking + * key + * @param {Number} options.opcode The opcode + * @param {Boolean} [options.readOnly=false] Specifies whether `data` can be + * modified + * @param {Boolean} [options.rsv1=false] Specifies whether or not to set the + * RSV1 bit + * @param {Function} [cb] Callback + * @private + */ + getBlobData(blob, compress, options, cb) { + this._bufferedBytes += options[kByteLength]; + this._state = GET_BLOB_DATA; + + blob + .arrayBuffer() + .then((arrayBuffer) => { + if (this._socket.destroyed) { + const err = new Error( + 'The socket was closed while the blob was being read' + ); + + // + // `callCallbacks` is called in the next tick to ensure that errors + // that might be thrown in the callbacks behave like errors thrown + // outside the promise chain. + // + process.nextTick(callCallbacks, this, err, cb); + return; + } + + this._bufferedBytes -= options[kByteLength]; + const data = toBuffer(arrayBuffer); + + if (!compress) { + this._state = DEFAULT; + this.sendFrame(Sender.frame(data, options), cb); + this.dequeue(); + } else { + this.dispatch(data, compress, options, cb); + } + }) + .catch((err) => { + // + // `onError` is called in the next tick for the same reason that + // `callCallbacks` above is. + // + process.nextTick(onError, this, err, cb); + }); + } + + /** + * Dispatches a message. + * + * @param {(Buffer|String)} data The message to send + * @param {Boolean} [compress=false] Specifies whether or not to compress + * `data` + * @param {Object} options Options object + * @param {Boolean} [options.fin=false] Specifies whether or not to set the + * FIN bit + * @param {Function} [options.generateMask] The function used to generate the + * masking key + * @param {Boolean} [options.mask=false] Specifies whether or not to mask + * `data` + * @param {Buffer} [options.maskBuffer] The buffer used to store the masking + * key + * @param {Number} options.opcode The opcode + * @param {Boolean} [options.readOnly=false] Specifies whether `data` can be + * modified + * @param {Boolean} [options.rsv1=false] Specifies whether or not to set the + * RSV1 bit + * @param {Function} [cb] Callback + * @private + */ + dispatch(data, compress, options, cb) { + if (!compress) { + this.sendFrame(Sender.frame(data, options), cb); + return; + } + + const perMessageDeflate = this._extensions[PerMessageDeflate.extensionName]; + + this._bufferedBytes += options[kByteLength]; + this._state = DEFLATING; + perMessageDeflate.compress(data, options.fin, (_, buf) => { + if (this._socket.destroyed) { + const err = new Error( + 'The socket was closed while data was being compressed' + ); + + callCallbacks(this, err, cb); + return; + } + + this._bufferedBytes -= options[kByteLength]; + this._state = DEFAULT; + options.readOnly = false; + this.sendFrame(Sender.frame(buf, options), cb); + this.dequeue(); + }); + } + + /** + * Executes queued send operations. + * + * @private + */ + dequeue() { + while (this._state === DEFAULT && this._queue.length) { + const params = this._queue.shift(); + + this._bufferedBytes -= params[3][kByteLength]; + Reflect.apply(params[0], this, params.slice(1)); + } + } + + /** + * Enqueues a send operation. + * + * @param {Array} params Send operation parameters. + * @private + */ + enqueue(params) { + this._bufferedBytes += params[3][kByteLength]; + this._queue.push(params); + } + + /** + * Sends a frame. + * + * @param {(Buffer | String)[]} list The frame to send + * @param {Function} [cb] Callback + * @private + */ + sendFrame(list, cb) { + if (list.length === 2) { + this._socket.cork(); + this._socket.write(list[0]); + this._socket.write(list[1], cb); + this._socket.uncork(); + } else { + this._socket.write(list[0], cb); + } + } +} + +module.exports = Sender; + +/** + * Calls queued callbacks with an error. + * + * @param {Sender} sender The `Sender` instance + * @param {Error} err The error to call the callbacks with + * @param {Function} [cb] The first callback + * @private + */ +function callCallbacks(sender, err, cb) { + if (typeof cb === 'function') cb(err); + + for (let i = 0; i < sender._queue.length; i++) { + const params = sender._queue[i]; + const callback = params[params.length - 1]; + + if (typeof callback === 'function') callback(err); + } +} + +/** + * Handles a `Sender` error. + * + * @param {Sender} sender The `Sender` instance + * @param {Error} err The error + * @param {Function} [cb] The first pending callback + * @private + */ +function onError(sender, err, cb) { + callCallbacks(sender, err, cb); + sender.onerror(err); +} diff --git a/web/node_modules/ws/lib/stream.js b/web/node_modules/ws/lib/stream.js new file mode 100644 index 0000000000000000000000000000000000000000..4c58c911bc3da49c9b2610da77ff61b71f111810 --- /dev/null +++ b/web/node_modules/ws/lib/stream.js @@ -0,0 +1,161 @@ +/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "^WebSocket$" }] */ +'use strict'; + +const WebSocket = require('./websocket'); +const { Duplex } = require('stream'); + +/** + * Emits the `'close'` event on a stream. + * + * @param {Duplex} stream The stream. + * @private + */ +function emitClose(stream) { + stream.emit('close'); +} + +/** + * The listener of the `'end'` event. + * + * @private + */ +function duplexOnEnd() { + if (!this.destroyed && this._writableState.finished) { + this.destroy(); + } +} + +/** + * The listener of the `'error'` event. + * + * @param {Error} err The error + * @private + */ +function duplexOnError(err) { + this.removeListener('error', duplexOnError); + this.destroy(); + if (this.listenerCount('error') === 0) { + // Do not suppress the throwing behavior. + this.emit('error', err); + } +} + +/** + * Wraps a `WebSocket` in a duplex stream. + * + * @param {WebSocket} ws The `WebSocket` to wrap + * @param {Object} [options] The options for the `Duplex` constructor + * @return {Duplex} The duplex stream + * @public + */ +function createWebSocketStream(ws, options) { + let terminateOnDestroy = true; + + const duplex = new Duplex({ + ...options, + autoDestroy: false, + emitClose: false, + objectMode: false, + writableObjectMode: false + }); + + ws.on('message', function message(msg, isBinary) { + const data = + !isBinary && duplex._readableState.objectMode ? msg.toString() : msg; + + if (!duplex.push(data)) ws.pause(); + }); + + ws.once('error', function error(err) { + if (duplex.destroyed) return; + + // Prevent `ws.terminate()` from being called by `duplex._destroy()`. + // + // - If the `'error'` event is emitted before the `'open'` event, then + // `ws.terminate()` is a noop as no socket is assigned. + // - Otherwise, the error is re-emitted by the listener of the `'error'` + // event of the `Receiver` object. The listener already closes the + // connection by calling `ws.close()`. This allows a close frame to be + // sent to the other peer. If `ws.terminate()` is called right after this, + // then the close frame might not be sent. + terminateOnDestroy = false; + duplex.destroy(err); + }); + + ws.once('close', function close() { + if (duplex.destroyed) return; + + duplex.push(null); + }); + + duplex._destroy = function (err, callback) { + if (ws.readyState === ws.CLOSED) { + callback(err); + process.nextTick(emitClose, duplex); + return; + } + + let called = false; + + ws.once('error', function error(err) { + called = true; + callback(err); + }); + + ws.once('close', function close() { + if (!called) callback(err); + process.nextTick(emitClose, duplex); + }); + + if (terminateOnDestroy) ws.terminate(); + }; + + duplex._final = function (callback) { + if (ws.readyState === ws.CONNECTING) { + ws.once('open', function open() { + duplex._final(callback); + }); + return; + } + + // If the value of the `_socket` property is `null` it means that `ws` is a + // client websocket and the handshake failed. In fact, when this happens, a + // socket is never assigned to the websocket. Wait for the `'error'` event + // that will be emitted by the websocket. + if (ws._socket === null) return; + + if (ws._socket._writableState.finished) { + callback(); + if (duplex._readableState.endEmitted) duplex.destroy(); + } else { + ws._socket.once('finish', function finish() { + // `duplex` is not destroyed here because the `'end'` event will be + // emitted on `duplex` after this `'finish'` event. The EOF signaling + // `null` chunk is, in fact, pushed when the websocket emits `'close'`. + callback(); + }); + ws.close(); + } + }; + + duplex._read = function () { + if (ws.isPaused) ws.resume(); + }; + + duplex._write = function (chunk, encoding, callback) { + if (ws.readyState === ws.CONNECTING) { + ws.once('open', function open() { + duplex._write(chunk, encoding, callback); + }); + return; + } + + ws.send(chunk, callback); + }; + + duplex.on('end', duplexOnEnd); + duplex.on('error', duplexOnError); + return duplex; +} + +module.exports = createWebSocketStream; diff --git a/web/node_modules/ws/lib/subprotocol.js b/web/node_modules/ws/lib/subprotocol.js new file mode 100644 index 0000000000000000000000000000000000000000..d4381e8864fd055e81dd75eeb4b852d9da864673 --- /dev/null +++ b/web/node_modules/ws/lib/subprotocol.js @@ -0,0 +1,62 @@ +'use strict'; + +const { tokenChars } = require('./validation'); + +/** + * Parses the `Sec-WebSocket-Protocol` header into a set of subprotocol names. + * + * @param {String} header The field value of the header + * @return {Set} The subprotocol names + * @public + */ +function parse(header) { + const protocols = new Set(); + let start = -1; + let end = -1; + let i = 0; + + for (i; i < header.length; i++) { + const code = header.charCodeAt(i); + + if (end === -1 && tokenChars[code] === 1) { + if (start === -1) start = i; + } else if ( + i !== 0 && + (code === 0x20 /* ' ' */ || code === 0x09) /* '\t' */ + ) { + if (end === -1 && start !== -1) end = i; + } else if (code === 0x2c /* ',' */) { + if (start === -1) { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + + if (end === -1) end = i; + + const protocol = header.slice(start, end); + + if (protocols.has(protocol)) { + throw new SyntaxError(`The "${protocol}" subprotocol is duplicated`); + } + + protocols.add(protocol); + start = end = -1; + } else { + throw new SyntaxError(`Unexpected character at index ${i}`); + } + } + + if (start === -1 || end !== -1) { + throw new SyntaxError('Unexpected end of input'); + } + + const protocol = header.slice(start, i); + + if (protocols.has(protocol)) { + throw new SyntaxError(`The "${protocol}" subprotocol is duplicated`); + } + + protocols.add(protocol); + return protocols; +} + +module.exports = { parse }; diff --git a/web/node_modules/ws/lib/validation.js b/web/node_modules/ws/lib/validation.js new file mode 100644 index 0000000000000000000000000000000000000000..4a2e68d5127279431a74b6961e8d35161d3276e2 --- /dev/null +++ b/web/node_modules/ws/lib/validation.js @@ -0,0 +1,152 @@ +'use strict'; + +const { isUtf8 } = require('buffer'); + +const { hasBlob } = require('./constants'); + +// +// Allowed token characters: +// +// '!', '#', '$', '%', '&', ''', '*', '+', '-', +// '.', 0-9, A-Z, '^', '_', '`', a-z, '|', '~' +// +// tokenChars[32] === 0 // ' ' +// tokenChars[33] === 1 // '!' +// tokenChars[34] === 0 // '"' +// ... +// +// prettier-ignore +const tokenChars = [ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0 - 15 + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 16 - 31 + 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, // 32 - 47 + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, // 48 - 63 + 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 64 - 79 + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, // 80 - 95 + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 96 - 111 + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0 // 112 - 127 +]; + +/** + * Checks if a status code is allowed in a close frame. + * + * @param {Number} code The status code + * @return {Boolean} `true` if the status code is valid, else `false` + * @public + */ +function isValidStatusCode(code) { + return ( + (code >= 1000 && + code <= 1014 && + code !== 1004 && + code !== 1005 && + code !== 1006) || + (code >= 3000 && code <= 4999) + ); +} + +/** + * Checks if a given buffer contains only correct UTF-8. + * Ported from https://www.cl.cam.ac.uk/%7Emgk25/ucs/utf8_check.c by + * Markus Kuhn. + * + * @param {Buffer} buf The buffer to check + * @return {Boolean} `true` if `buf` contains only correct UTF-8, else `false` + * @public + */ +function _isValidUTF8(buf) { + const len = buf.length; + let i = 0; + + while (i < len) { + if ((buf[i] & 0x80) === 0) { + // 0xxxxxxx + i++; + } else if ((buf[i] & 0xe0) === 0xc0) { + // 110xxxxx 10xxxxxx + if ( + i + 1 === len || + (buf[i + 1] & 0xc0) !== 0x80 || + (buf[i] & 0xfe) === 0xc0 // Overlong + ) { + return false; + } + + i += 2; + } else if ((buf[i] & 0xf0) === 0xe0) { + // 1110xxxx 10xxxxxx 10xxxxxx + if ( + i + 2 >= len || + (buf[i + 1] & 0xc0) !== 0x80 || + (buf[i + 2] & 0xc0) !== 0x80 || + (buf[i] === 0xe0 && (buf[i + 1] & 0xe0) === 0x80) || // Overlong + (buf[i] === 0xed && (buf[i + 1] & 0xe0) === 0xa0) // Surrogate (U+D800 - U+DFFF) + ) { + return false; + } + + i += 3; + } else if ((buf[i] & 0xf8) === 0xf0) { + // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx + if ( + i + 3 >= len || + (buf[i + 1] & 0xc0) !== 0x80 || + (buf[i + 2] & 0xc0) !== 0x80 || + (buf[i + 3] & 0xc0) !== 0x80 || + (buf[i] === 0xf0 && (buf[i + 1] & 0xf0) === 0x80) || // Overlong + (buf[i] === 0xf4 && buf[i + 1] > 0x8f) || + buf[i] > 0xf4 // > U+10FFFF + ) { + return false; + } + + i += 4; + } else { + return false; + } + } + + return true; +} + +/** + * Determines whether a value is a `Blob`. + * + * @param {*} value The value to be tested + * @return {Boolean} `true` if `value` is a `Blob`, else `false` + * @private + */ +function isBlob(value) { + return ( + hasBlob && + typeof value === 'object' && + typeof value.arrayBuffer === 'function' && + typeof value.type === 'string' && + typeof value.stream === 'function' && + (value[Symbol.toStringTag] === 'Blob' || + value[Symbol.toStringTag] === 'File') + ); +} + +module.exports = { + isBlob, + isValidStatusCode, + isValidUTF8: _isValidUTF8, + tokenChars +}; + +if (isUtf8) { + module.exports.isValidUTF8 = function (buf) { + return buf.length < 24 ? _isValidUTF8(buf) : isUtf8(buf); + }; +} /* istanbul ignore else */ else if (!process.env.WS_NO_UTF_8_VALIDATE) { + try { + const isValidUTF8 = require('utf-8-validate'); + + module.exports.isValidUTF8 = function (buf) { + return buf.length < 32 ? _isValidUTF8(buf) : isValidUTF8(buf); + }; + } catch (e) { + // Continue regardless of the error. + } +} diff --git a/web/node_modules/ws/lib/websocket-server.js b/web/node_modules/ws/lib/websocket-server.js new file mode 100644 index 0000000000000000000000000000000000000000..8ce9798465ba6241224496d1b76b959acda4e072 --- /dev/null +++ b/web/node_modules/ws/lib/websocket-server.js @@ -0,0 +1,562 @@ +/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "^Duplex$", "caughtErrors": "none" }] */ + +'use strict'; + +const EventEmitter = require('events'); +const http = require('http'); +const { Duplex } = require('stream'); +const { createHash } = require('crypto'); + +const extension = require('./extension'); +const PerMessageDeflate = require('./permessage-deflate'); +const subprotocol = require('./subprotocol'); +const WebSocket = require('./websocket'); +const { CLOSE_TIMEOUT, GUID, kWebSocket } = require('./constants'); + +const keyRegex = /^[+/0-9A-Za-z]{22}==$/; + +const RUNNING = 0; +const CLOSING = 1; +const CLOSED = 2; + +/** + * Class representing a WebSocket server. + * + * @extends EventEmitter + */ +class WebSocketServer extends EventEmitter { + /** + * Create a `WebSocketServer` instance. + * + * @param {Object} options Configuration options + * @param {Boolean} [options.allowSynchronousEvents=true] Specifies whether + * any of the `'message'`, `'ping'`, and `'pong'` events can be emitted + * multiple times in the same tick + * @param {Boolean} [options.autoPong=true] Specifies whether or not to + * automatically send a pong in response to a ping + * @param {Number} [options.backlog=511] The maximum length of the queue of + * pending connections + * @param {Boolean} [options.clientTracking=true] Specifies whether or not to + * track clients + * @param {Number} [options.closeTimeout=30000] Duration in milliseconds to + * wait for the closing handshake to finish after `websocket.close()` is + * called + * @param {Function} [options.handleProtocols] A hook to handle protocols + * @param {String} [options.host] The hostname where to bind the server + * @param {Number} [options.maxBufferedChunks=1048576] The maximum number of + * buffered data chunks + * @param {Number} [options.maxFragments=131072] The maximum number of message + * fragments + * @param {Number} [options.maxPayload=104857600] The maximum allowed message + * size + * @param {Boolean} [options.noServer=false] Enable no server mode + * @param {String} [options.path] Accept only connections matching this path + * @param {(Boolean|Object)} [options.perMessageDeflate=false] Enable/disable + * permessage-deflate + * @param {Number} [options.port] The port where to bind the server + * @param {(http.Server|https.Server)} [options.server] A pre-created HTTP/S + * server to use + * @param {Boolean} [options.skipUTF8Validation=false] Specifies whether or + * not to skip UTF-8 validation for text and close messages + * @param {Function} [options.verifyClient] A hook to reject connections + * @param {Function} [options.WebSocket=WebSocket] Specifies the `WebSocket` + * class to use. It must be the `WebSocket` class or class that extends it + * @param {Function} [callback] A listener for the `listening` event + */ + constructor(options, callback) { + super(); + + options = { + allowSynchronousEvents: true, + autoPong: true, + maxBufferedChunks: 1024 * 1024, + maxFragments: 128 * 1024, + maxPayload: 100 * 1024 * 1024, + skipUTF8Validation: false, + perMessageDeflate: false, + handleProtocols: null, + clientTracking: true, + closeTimeout: CLOSE_TIMEOUT, + verifyClient: null, + noServer: false, + backlog: null, // use default (511 as implemented in net.js) + server: null, + host: null, + path: null, + port: null, + WebSocket, + ...options + }; + + if ( + (options.port == null && !options.server && !options.noServer) || + (options.port != null && (options.server || options.noServer)) || + (options.server && options.noServer) + ) { + throw new TypeError( + 'One and only one of the "port", "server", or "noServer" options ' + + 'must be specified' + ); + } + + if (options.port != null) { + this._server = http.createServer((req, res) => { + const body = http.STATUS_CODES[426]; + + res.writeHead(426, { + 'Content-Length': body.length, + 'Content-Type': 'text/plain' + }); + res.end(body); + }); + this._server.listen( + options.port, + options.host, + options.backlog, + callback + ); + } else if (options.server) { + this._server = options.server; + } + + if (this._server) { + const emitConnection = this.emit.bind(this, 'connection'); + + this._removeListeners = addListeners(this._server, { + listening: this.emit.bind(this, 'listening'), + error: this.emit.bind(this, 'error'), + upgrade: (req, socket, head) => { + this.handleUpgrade(req, socket, head, emitConnection); + } + }); + } + + if (options.perMessageDeflate === true) options.perMessageDeflate = {}; + if (options.clientTracking) { + this.clients = new Set(); + this._shouldEmitClose = false; + } + + this.options = options; + this._state = RUNNING; + } + + /** + * Returns the bound address, the address family name, and port of the server + * as reported by the operating system if listening on an IP socket. + * If the server is listening on a pipe or UNIX domain socket, the name is + * returned as a string. + * + * @return {(Object|String|null)} The address of the server + * @public + */ + address() { + if (this.options.noServer) { + throw new Error('The server is operating in "noServer" mode'); + } + + if (!this._server) return null; + return this._server.address(); + } + + /** + * Stop the server from accepting new connections and emit the `'close'` event + * when all existing connections are closed. + * + * @param {Function} [cb] A one-time listener for the `'close'` event + * @public + */ + close(cb) { + if (this._state === CLOSED) { + if (cb) { + this.once('close', () => { + cb(new Error('The server is not running')); + }); + } + + process.nextTick(emitClose, this); + return; + } + + if (cb) this.once('close', cb); + + if (this._state === CLOSING) return; + this._state = CLOSING; + + if (this.options.noServer || this.options.server) { + if (this._server) { + this._removeListeners(); + this._removeListeners = this._server = null; + } + + if (this.clients) { + if (!this.clients.size) { + process.nextTick(emitClose, this); + } else { + this._shouldEmitClose = true; + } + } else { + process.nextTick(emitClose, this); + } + } else { + const server = this._server; + + this._removeListeners(); + this._removeListeners = this._server = null; + + // + // The HTTP/S server was created internally. Close it, and rely on its + // `'close'` event. + // + server.close(() => { + emitClose(this); + }); + } + } + + /** + * See if a given request should be handled by this server instance. + * + * @param {http.IncomingMessage} req Request object to inspect + * @return {Boolean} `true` if the request is valid, else `false` + * @public + */ + shouldHandle(req) { + if (this.options.path) { + const index = req.url.indexOf('?'); + const pathname = index !== -1 ? req.url.slice(0, index) : req.url; + + if (pathname !== this.options.path) return false; + } + + return true; + } + + /** + * Handle a HTTP Upgrade request. + * + * @param {http.IncomingMessage} req The request object + * @param {Duplex} socket The network socket between the server and client + * @param {Buffer} head The first packet of the upgraded stream + * @param {Function} cb Callback + * @public + */ + handleUpgrade(req, socket, head, cb) { + socket.on('error', socketOnError); + + const key = req.headers['sec-websocket-key']; + const upgrade = req.headers.upgrade; + const version = +req.headers['sec-websocket-version']; + + if (req.method !== 'GET') { + const message = 'Invalid HTTP method'; + abortHandshakeOrEmitwsClientError(this, req, socket, 405, message); + return; + } + + if (upgrade === undefined || upgrade.toLowerCase() !== 'websocket') { + const message = 'Invalid Upgrade header'; + abortHandshakeOrEmitwsClientError(this, req, socket, 400, message); + return; + } + + if (key === undefined || !keyRegex.test(key)) { + const message = 'Missing or invalid Sec-WebSocket-Key header'; + abortHandshakeOrEmitwsClientError(this, req, socket, 400, message); + return; + } + + if (version !== 13 && version !== 8) { + const message = 'Missing or invalid Sec-WebSocket-Version header'; + abortHandshakeOrEmitwsClientError(this, req, socket, 400, message, { + 'Sec-WebSocket-Version': '13, 8' + }); + return; + } + + if (!this.shouldHandle(req)) { + abortHandshake(socket, 400); + return; + } + + const secWebSocketProtocol = req.headers['sec-websocket-protocol']; + let protocols = new Set(); + + if (secWebSocketProtocol !== undefined) { + try { + protocols = subprotocol.parse(secWebSocketProtocol); + } catch (err) { + const message = 'Invalid Sec-WebSocket-Protocol header'; + abortHandshakeOrEmitwsClientError(this, req, socket, 400, message); + return; + } + } + + const secWebSocketExtensions = req.headers['sec-websocket-extensions']; + const extensions = {}; + + if ( + this.options.perMessageDeflate && + secWebSocketExtensions !== undefined + ) { + const perMessageDeflate = new PerMessageDeflate({ + ...this.options.perMessageDeflate, + isServer: true, + maxPayload: this.options.maxPayload + }); + + try { + const offers = extension.parse(secWebSocketExtensions); + + if (offers[PerMessageDeflate.extensionName]) { + perMessageDeflate.accept(offers[PerMessageDeflate.extensionName]); + extensions[PerMessageDeflate.extensionName] = perMessageDeflate; + } + } catch (err) { + const message = + 'Invalid or unacceptable Sec-WebSocket-Extensions header'; + abortHandshakeOrEmitwsClientError(this, req, socket, 400, message); + return; + } + } + + // + // Optionally call external client verification handler. + // + if (this.options.verifyClient) { + const info = { + origin: + req.headers[`${version === 8 ? 'sec-websocket-origin' : 'origin'}`], + secure: !!(req.socket.authorized || req.socket.encrypted), + req + }; + + if (this.options.verifyClient.length === 2) { + this.options.verifyClient(info, (verified, code, message, headers) => { + if (!verified) { + return abortHandshake(socket, code || 401, message, headers); + } + + this.completeUpgrade( + extensions, + key, + protocols, + req, + socket, + head, + cb + ); + }); + return; + } + + if (!this.options.verifyClient(info)) return abortHandshake(socket, 401); + } + + this.completeUpgrade(extensions, key, protocols, req, socket, head, cb); + } + + /** + * Upgrade the connection to WebSocket. + * + * @param {Object} extensions The accepted extensions + * @param {String} key The value of the `Sec-WebSocket-Key` header + * @param {Set} protocols The subprotocols + * @param {http.IncomingMessage} req The request object + * @param {Duplex} socket The network socket between the server and client + * @param {Buffer} head The first packet of the upgraded stream + * @param {Function} cb Callback + * @throws {Error} If called more than once with the same socket + * @private + */ + completeUpgrade(extensions, key, protocols, req, socket, head, cb) { + // + // Destroy the socket if the client has already sent a FIN packet. + // + if (!socket.readable || !socket.writable) return socket.destroy(); + + if (socket[kWebSocket]) { + throw new Error( + 'server.handleUpgrade() was called more than once with the same ' + + 'socket, possibly due to a misconfiguration' + ); + } + + if (this._state > RUNNING) return abortHandshake(socket, 503); + + const digest = createHash('sha1') + .update(key + GUID) + .digest('base64'); + + const headers = [ + 'HTTP/1.1 101 Switching Protocols', + 'Upgrade: websocket', + 'Connection: Upgrade', + `Sec-WebSocket-Accept: ${digest}` + ]; + + const ws = new this.options.WebSocket(null, undefined, this.options); + + if (protocols.size) { + // + // Optionally call external protocol selection handler. + // + const protocol = this.options.handleProtocols + ? this.options.handleProtocols(protocols, req) + : protocols.values().next().value; + + if (protocol) { + headers.push(`Sec-WebSocket-Protocol: ${protocol}`); + ws._protocol = protocol; + } + } + + if (extensions[PerMessageDeflate.extensionName]) { + const params = extensions[PerMessageDeflate.extensionName].params; + const value = extension.format({ + [PerMessageDeflate.extensionName]: [params] + }); + headers.push(`Sec-WebSocket-Extensions: ${value}`); + ws._extensions = extensions; + } + + // + // Allow external modification/inspection of handshake headers. + // + this.emit('headers', headers, req); + + socket.write(headers.concat('\r\n').join('\r\n')); + socket.removeListener('error', socketOnError); + + ws.setSocket(socket, head, { + allowSynchronousEvents: this.options.allowSynchronousEvents, + maxBufferedChunks: this.options.maxBufferedChunks, + maxFragments: this.options.maxFragments, + maxPayload: this.options.maxPayload, + skipUTF8Validation: this.options.skipUTF8Validation + }); + + if (this.clients) { + this.clients.add(ws); + ws.on('close', () => { + this.clients.delete(ws); + + if (this._shouldEmitClose && !this.clients.size) { + process.nextTick(emitClose, this); + } + }); + } + + cb(ws, req); + } +} + +module.exports = WebSocketServer; + +/** + * Add event listeners on an `EventEmitter` using a map of + * pairs. + * + * @param {EventEmitter} server The event emitter + * @param {Object.} map The listeners to add + * @return {Function} A function that will remove the added listeners when + * called + * @private + */ +function addListeners(server, map) { + for (const event of Object.keys(map)) server.on(event, map[event]); + + return function removeListeners() { + for (const event of Object.keys(map)) { + server.removeListener(event, map[event]); + } + }; +} + +/** + * Emit a `'close'` event on an `EventEmitter`. + * + * @param {EventEmitter} server The event emitter + * @private + */ +function emitClose(server) { + server._state = CLOSED; + server.emit('close'); +} + +/** + * Handle socket errors. + * + * @private + */ +function socketOnError() { + this.destroy(); +} + +/** + * Close the connection when preconditions are not fulfilled. + * + * @param {Duplex} socket The socket of the upgrade request + * @param {Number} code The HTTP response status code + * @param {String} [message] The HTTP response body + * @param {Object} [headers] Additional HTTP response headers + * @private + */ +function abortHandshake(socket, code, message, headers) { + // + // The socket is writable unless the user destroyed or ended it before calling + // `server.handleUpgrade()` or in the `verifyClient` function, which is a user + // error. Handling this does not make much sense as the worst that can happen + // is that some of the data written by the user might be discarded due to the + // call to `socket.end()` below, which triggers an `'error'` event that in + // turn causes the socket to be destroyed. + // + message = message || http.STATUS_CODES[code]; + headers = { + Connection: 'close', + 'Content-Type': 'text/html', + 'Content-Length': Buffer.byteLength(message), + ...headers + }; + + socket.once('finish', socket.destroy); + + socket.end( + `HTTP/1.1 ${code} ${http.STATUS_CODES[code]}\r\n` + + Object.keys(headers) + .map((h) => `${h}: ${headers[h]}`) + .join('\r\n') + + '\r\n\r\n' + + message + ); +} + +/** + * Emit a `'wsClientError'` event on a `WebSocketServer` if there is at least + * one listener for it, otherwise call `abortHandshake()`. + * + * @param {WebSocketServer} server The WebSocket server + * @param {http.IncomingMessage} req The request object + * @param {Duplex} socket The socket of the upgrade request + * @param {Number} code The HTTP response status code + * @param {String} message The HTTP response body + * @param {Object} [headers] The HTTP response headers + * @private + */ +function abortHandshakeOrEmitwsClientError( + server, + req, + socket, + code, + message, + headers +) { + if (server.listenerCount('wsClientError')) { + const err = new Error(message); + Error.captureStackTrace(err, abortHandshakeOrEmitwsClientError); + + server.emit('wsClientError', err, socket, req); + } else { + abortHandshake(socket, code, message, headers); + } +} diff --git a/web/node_modules/ws/lib/websocket.js b/web/node_modules/ws/lib/websocket.js new file mode 100644 index 0000000000000000000000000000000000000000..ed3735ea48cab527a00f9245700159ceff28889c --- /dev/null +++ b/web/node_modules/ws/lib/websocket.js @@ -0,0 +1,1407 @@ +/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "^Duplex|Readable$", "caughtErrors": "none" }] */ + +'use strict'; + +const EventEmitter = require('events'); +const https = require('https'); +const http = require('http'); +const net = require('net'); +const tls = require('tls'); +const { randomBytes, createHash } = require('crypto'); +const { Duplex, Readable } = require('stream'); +const { URL } = require('url'); + +const PerMessageDeflate = require('./permessage-deflate'); +const Receiver = require('./receiver'); +const Sender = require('./sender'); +const { isBlob } = require('./validation'); + +const { + BINARY_TYPES, + CLOSE_TIMEOUT, + EMPTY_BUFFER, + GUID, + kForOnEventAttribute, + kListener, + kStatusCode, + kWebSocket, + NOOP +} = require('./constants'); +const { + EventTarget: { addEventListener, removeEventListener } +} = require('./event-target'); +const { format, parse } = require('./extension'); +const { toBuffer } = require('./buffer-util'); + +const kAborted = Symbol('kAborted'); +const protocolVersions = [8, 13]; +const readyStates = ['CONNECTING', 'OPEN', 'CLOSING', 'CLOSED']; +const subprotocolRegex = /^[!#$%&'*+\-.0-9A-Z^_`|a-z~]+$/; + +/** + * Class representing a WebSocket. + * + * @extends EventEmitter + */ +class WebSocket extends EventEmitter { + /** + * Create a new `WebSocket`. + * + * @param {(String|URL)} address The URL to which to connect + * @param {(String|String[])} [protocols] The subprotocols + * @param {Object} [options] Connection options + */ + constructor(address, protocols, options) { + super(); + + this._binaryType = BINARY_TYPES[0]; + this._closeCode = 1006; + this._closeFrameReceived = false; + this._closeFrameSent = false; + this._closeMessage = EMPTY_BUFFER; + this._closeTimer = null; + this._errorEmitted = false; + this._extensions = {}; + this._paused = false; + this._protocol = ''; + this._readyState = WebSocket.CONNECTING; + this._receiver = null; + this._sender = null; + this._socket = null; + + if (address !== null) { + this._bufferedAmount = 0; + this._isServer = false; + this._redirects = 0; + + if (protocols === undefined) { + protocols = []; + } else if (!Array.isArray(protocols)) { + if (typeof protocols === 'object' && protocols !== null) { + options = protocols; + protocols = []; + } else { + protocols = [protocols]; + } + } + + initAsClient(this, address, protocols, options); + } else { + this._autoPong = options.autoPong; + this._closeTimeout = options.closeTimeout; + this._isServer = true; + } + } + + /** + * For historical reasons, the custom "nodebuffer" type is used by the default + * instead of "blob". + * + * @type {String} + */ + get binaryType() { + return this._binaryType; + } + + set binaryType(type) { + if (!BINARY_TYPES.includes(type)) return; + + this._binaryType = type; + + // + // Allow to change `binaryType` on the fly. + // + if (this._receiver) this._receiver._binaryType = type; + } + + /** + * @type {Number} + */ + get bufferedAmount() { + if (!this._socket) return this._bufferedAmount; + + return this._socket._writableState.length + this._sender._bufferedBytes; + } + + /** + * @type {String} + */ + get extensions() { + return Object.keys(this._extensions).join(); + } + + /** + * @type {Boolean} + */ + get isPaused() { + return this._paused; + } + + /** + * @type {Function} + */ + /* istanbul ignore next */ + get onclose() { + return null; + } + + /** + * @type {Function} + */ + /* istanbul ignore next */ + get onerror() { + return null; + } + + /** + * @type {Function} + */ + /* istanbul ignore next */ + get onopen() { + return null; + } + + /** + * @type {Function} + */ + /* istanbul ignore next */ + get onmessage() { + return null; + } + + /** + * @type {String} + */ + get protocol() { + return this._protocol; + } + + /** + * @type {Number} + */ + get readyState() { + return this._readyState; + } + + /** + * @type {String} + */ + get url() { + return this._url; + } + + /** + * Set up the socket and the internal resources. + * + * @param {Duplex} socket The network socket between the server and client + * @param {Buffer} head The first packet of the upgraded stream + * @param {Object} options Options object + * @param {Boolean} [options.allowSynchronousEvents=false] Specifies whether + * any of the `'message'`, `'ping'`, and `'pong'` events can be emitted + * multiple times in the same tick + * @param {Function} [options.generateMask] The function used to generate the + * masking key + * @param {Number} [options.maxBufferedChunks=0] The maximum number of + * buffered data chunks + * @param {Number} [options.maxFragments=0] The maximum number of message + * fragments + * @param {Number} [options.maxPayload=0] The maximum allowed message size + * @param {Boolean} [options.skipUTF8Validation=false] Specifies whether or + * not to skip UTF-8 validation for text and close messages + * @private + */ + setSocket(socket, head, options) { + const receiver = new Receiver({ + allowSynchronousEvents: options.allowSynchronousEvents, + binaryType: this.binaryType, + extensions: this._extensions, + isServer: this._isServer, + maxBufferedChunks: options.maxBufferedChunks, + maxFragments: options.maxFragments, + maxPayload: options.maxPayload, + skipUTF8Validation: options.skipUTF8Validation + }); + + const sender = new Sender(socket, this._extensions, options.generateMask); + + this._receiver = receiver; + this._sender = sender; + this._socket = socket; + + receiver[kWebSocket] = this; + sender[kWebSocket] = this; + socket[kWebSocket] = this; + + receiver.on('conclude', receiverOnConclude); + receiver.on('drain', receiverOnDrain); + receiver.on('error', receiverOnError); + receiver.on('message', receiverOnMessage); + receiver.on('ping', receiverOnPing); + receiver.on('pong', receiverOnPong); + + sender.onerror = senderOnError; + + // + // These methods may not be available if `socket` is just a `Duplex`. + // + if (socket.setTimeout) socket.setTimeout(0); + if (socket.setNoDelay) socket.setNoDelay(); + + if (head.length > 0) socket.unshift(head); + + socket.on('close', socketOnClose); + socket.on('data', socketOnData); + socket.on('end', socketOnEnd); + socket.on('error', socketOnError); + + this._readyState = WebSocket.OPEN; + this.emit('open'); + } + + /** + * Emit the `'close'` event. + * + * @private + */ + emitClose() { + if (!this._socket) { + this._readyState = WebSocket.CLOSED; + this.emit('close', this._closeCode, this._closeMessage); + return; + } + + if (this._extensions[PerMessageDeflate.extensionName]) { + this._extensions[PerMessageDeflate.extensionName].cleanup(); + } + + this._receiver.removeAllListeners(); + this._readyState = WebSocket.CLOSED; + this.emit('close', this._closeCode, this._closeMessage); + } + + /** + * Start a closing handshake. + * + * +----------+ +-----------+ +----------+ + * - - -|ws.close()|-->|close frame|-->|ws.close()|- - - + * | +----------+ +-----------+ +----------+ | + * +----------+ +-----------+ | + * CLOSING |ws.close()|<--|close frame|<--+-----+ CLOSING + * +----------+ +-----------+ | + * | | | +---+ | + * +------------------------+-->|fin| - - - - + * | +---+ | +---+ + * - - - - -|fin|<---------------------+ + * +---+ + * + * @param {Number} [code] Status code explaining why the connection is closing + * @param {(String|Buffer)} [data] The reason why the connection is + * closing + * @public + */ + close(code, data) { + if (this.readyState === WebSocket.CLOSED) return; + if (this.readyState === WebSocket.CONNECTING) { + const msg = 'WebSocket was closed before the connection was established'; + abortHandshake(this, this._req, msg); + return; + } + + if (this.readyState === WebSocket.CLOSING) { + if ( + this._closeFrameSent && + (this._closeFrameReceived || this._receiver._writableState.errorEmitted) + ) { + this._socket.end(); + } + + return; + } + + this._readyState = WebSocket.CLOSING; + this._sender.close(code, data, !this._isServer, (err) => { + // + // This error is handled by the `'error'` listener on the socket. We only + // want to know if the close frame has been sent here. + // + if (err) return; + + this._closeFrameSent = true; + + if ( + this._closeFrameReceived || + this._receiver._writableState.errorEmitted + ) { + this._socket.end(); + } + }); + + setCloseTimer(this); + } + + /** + * Pause the socket. + * + * @public + */ + pause() { + if ( + this.readyState === WebSocket.CONNECTING || + this.readyState === WebSocket.CLOSED + ) { + return; + } + + this._paused = true; + this._socket.pause(); + } + + /** + * Send a ping. + * + * @param {*} [data] The data to send + * @param {Boolean} [mask] Indicates whether or not to mask `data` + * @param {Function} [cb] Callback which is executed when the ping is sent + * @public + */ + ping(data, mask, cb) { + if (this.readyState === WebSocket.CONNECTING) { + throw new Error('WebSocket is not open: readyState 0 (CONNECTING)'); + } + + if (typeof data === 'function') { + cb = data; + data = mask = undefined; + } else if (typeof mask === 'function') { + cb = mask; + mask = undefined; + } + + if (typeof data === 'number') data = data.toString(); + + if (this.readyState !== WebSocket.OPEN) { + sendAfterClose(this, data, cb); + return; + } + + if (mask === undefined) mask = !this._isServer; + this._sender.ping(data || EMPTY_BUFFER, mask, cb); + } + + /** + * Send a pong. + * + * @param {*} [data] The data to send + * @param {Boolean} [mask] Indicates whether or not to mask `data` + * @param {Function} [cb] Callback which is executed when the pong is sent + * @public + */ + pong(data, mask, cb) { + if (this.readyState === WebSocket.CONNECTING) { + throw new Error('WebSocket is not open: readyState 0 (CONNECTING)'); + } + + if (typeof data === 'function') { + cb = data; + data = mask = undefined; + } else if (typeof mask === 'function') { + cb = mask; + mask = undefined; + } + + if (typeof data === 'number') data = data.toString(); + + if (this.readyState !== WebSocket.OPEN) { + sendAfterClose(this, data, cb); + return; + } + + if (mask === undefined) mask = !this._isServer; + this._sender.pong(data || EMPTY_BUFFER, mask, cb); + } + + /** + * Resume the socket. + * + * @public + */ + resume() { + if ( + this.readyState === WebSocket.CONNECTING || + this.readyState === WebSocket.CLOSED + ) { + return; + } + + this._paused = false; + if (!this._receiver._writableState.needDrain) this._socket.resume(); + } + + /** + * Send a data message. + * + * @param {*} data The message to send + * @param {Object} [options] Options object + * @param {Boolean} [options.binary] Specifies whether `data` is binary or + * text + * @param {Boolean} [options.compress] Specifies whether or not to compress + * `data` + * @param {Boolean} [options.fin=true] Specifies whether the fragment is the + * last one + * @param {Boolean} [options.mask] Specifies whether or not to mask `data` + * @param {Function} [cb] Callback which is executed when data is written out + * @public + */ + send(data, options, cb) { + if (this.readyState === WebSocket.CONNECTING) { + throw new Error('WebSocket is not open: readyState 0 (CONNECTING)'); + } + + if (typeof options === 'function') { + cb = options; + options = {}; + } + + if (typeof data === 'number') data = data.toString(); + + if (this.readyState !== WebSocket.OPEN) { + sendAfterClose(this, data, cb); + return; + } + + const opts = { + binary: typeof data !== 'string', + mask: !this._isServer, + compress: true, + fin: true, + ...options + }; + + if (!this._extensions[PerMessageDeflate.extensionName]) { + opts.compress = false; + } + + this._sender.send(data || EMPTY_BUFFER, opts, cb); + } + + /** + * Forcibly close the connection. + * + * @public + */ + terminate() { + if (this.readyState === WebSocket.CLOSED) return; + if (this.readyState === WebSocket.CONNECTING) { + const msg = 'WebSocket was closed before the connection was established'; + abortHandshake(this, this._req, msg); + return; + } + + if (this._socket) { + this._readyState = WebSocket.CLOSING; + this._socket.destroy(); + } + } +} + +/** + * @constant {Number} CONNECTING + * @memberof WebSocket + */ +Object.defineProperty(WebSocket, 'CONNECTING', { + enumerable: true, + value: readyStates.indexOf('CONNECTING') +}); + +/** + * @constant {Number} CONNECTING + * @memberof WebSocket.prototype + */ +Object.defineProperty(WebSocket.prototype, 'CONNECTING', { + enumerable: true, + value: readyStates.indexOf('CONNECTING') +}); + +/** + * @constant {Number} OPEN + * @memberof WebSocket + */ +Object.defineProperty(WebSocket, 'OPEN', { + enumerable: true, + value: readyStates.indexOf('OPEN') +}); + +/** + * @constant {Number} OPEN + * @memberof WebSocket.prototype + */ +Object.defineProperty(WebSocket.prototype, 'OPEN', { + enumerable: true, + value: readyStates.indexOf('OPEN') +}); + +/** + * @constant {Number} CLOSING + * @memberof WebSocket + */ +Object.defineProperty(WebSocket, 'CLOSING', { + enumerable: true, + value: readyStates.indexOf('CLOSING') +}); + +/** + * @constant {Number} CLOSING + * @memberof WebSocket.prototype + */ +Object.defineProperty(WebSocket.prototype, 'CLOSING', { + enumerable: true, + value: readyStates.indexOf('CLOSING') +}); + +/** + * @constant {Number} CLOSED + * @memberof WebSocket + */ +Object.defineProperty(WebSocket, 'CLOSED', { + enumerable: true, + value: readyStates.indexOf('CLOSED') +}); + +/** + * @constant {Number} CLOSED + * @memberof WebSocket.prototype + */ +Object.defineProperty(WebSocket.prototype, 'CLOSED', { + enumerable: true, + value: readyStates.indexOf('CLOSED') +}); + +[ + 'binaryType', + 'bufferedAmount', + 'extensions', + 'isPaused', + 'protocol', + 'readyState', + 'url' +].forEach((property) => { + Object.defineProperty(WebSocket.prototype, property, { enumerable: true }); +}); + +// +// Add the `onopen`, `onerror`, `onclose`, and `onmessage` attributes. +// See https://html.spec.whatwg.org/multipage/comms.html#the-websocket-interface +// +['open', 'error', 'close', 'message'].forEach((method) => { + Object.defineProperty(WebSocket.prototype, `on${method}`, { + enumerable: true, + get() { + for (const listener of this.listeners(method)) { + if (listener[kForOnEventAttribute]) return listener[kListener]; + } + + return null; + }, + set(handler) { + for (const listener of this.listeners(method)) { + if (listener[kForOnEventAttribute]) { + this.removeListener(method, listener); + break; + } + } + + if (typeof handler !== 'function') return; + + this.addEventListener(method, handler, { + [kForOnEventAttribute]: true + }); + } + }); +}); + +WebSocket.prototype.addEventListener = addEventListener; +WebSocket.prototype.removeEventListener = removeEventListener; + +module.exports = WebSocket; + +/** + * Initialize a WebSocket client. + * + * @param {WebSocket} websocket The client to initialize + * @param {(String|URL)} address The URL to which to connect + * @param {Array} protocols The subprotocols + * @param {Object} [options] Connection options + * @param {Boolean} [options.allowSynchronousEvents=true] Specifies whether any + * of the `'message'`, `'ping'`, and `'pong'` events can be emitted multiple + * times in the same tick + * @param {Boolean} [options.autoPong=true] Specifies whether or not to + * automatically send a pong in response to a ping + * @param {Number} [options.closeTimeout=30000] Duration in milliseconds to wait + * for the closing handshake to finish after `websocket.close()` is called + * @param {Function} [options.finishRequest] A function which can be used to + * customize the headers of each http request before it is sent + * @param {Boolean} [options.followRedirects=false] Whether or not to follow + * redirects + * @param {Function} [options.generateMask] The function used to generate the + * masking key + * @param {Number} [options.handshakeTimeout] Timeout in milliseconds for the + * handshake request + * @param {Number} [options.maxBufferedChunks=1048576] The maximum number of + * buffered data chunks + * @param {Number} [options.maxFragments=131072] The maximum number of message + * fragments + * @param {Number} [options.maxPayload=104857600] The maximum allowed message + * size + * @param {Number} [options.maxRedirects=10] The maximum number of redirects + * allowed + * @param {String} [options.origin] Value of the `Origin` or + * `Sec-WebSocket-Origin` header + * @param {(Boolean|Object)} [options.perMessageDeflate=true] Enable/disable + * permessage-deflate + * @param {Number} [options.protocolVersion=13] Value of the + * `Sec-WebSocket-Version` header + * @param {Boolean} [options.skipUTF8Validation=false] Specifies whether or + * not to skip UTF-8 validation for text and close messages + * @private + */ +function initAsClient(websocket, address, protocols, options) { + const opts = { + allowSynchronousEvents: true, + autoPong: true, + closeTimeout: CLOSE_TIMEOUT, + protocolVersion: protocolVersions[1], + maxBufferedChunks: 1024 * 1024, + maxFragments: 128 * 1024, + maxPayload: 100 * 1024 * 1024, + skipUTF8Validation: false, + perMessageDeflate: true, + followRedirects: false, + maxRedirects: 10, + ...options, + socketPath: undefined, + hostname: undefined, + protocol: undefined, + timeout: undefined, + method: 'GET', + host: undefined, + path: undefined, + port: undefined + }; + + websocket._autoPong = opts.autoPong; + websocket._closeTimeout = opts.closeTimeout; + + if (!protocolVersions.includes(opts.protocolVersion)) { + throw new RangeError( + `Unsupported protocol version: ${opts.protocolVersion} ` + + `(supported versions: ${protocolVersions.join(', ')})` + ); + } + + let parsedUrl; + + if (address instanceof URL) { + parsedUrl = address; + } else { + try { + parsedUrl = new URL(address); + } catch { + throw new SyntaxError(`Invalid URL: ${address}`); + } + } + + if (parsedUrl.protocol === 'http:') { + parsedUrl.protocol = 'ws:'; + } else if (parsedUrl.protocol === 'https:') { + parsedUrl.protocol = 'wss:'; + } + + websocket._url = parsedUrl.href; + + const isSecure = parsedUrl.protocol === 'wss:'; + const isIpcUrl = parsedUrl.protocol === 'ws+unix:'; + let invalidUrlMessage; + + if (parsedUrl.protocol !== 'ws:' && !isSecure && !isIpcUrl) { + invalidUrlMessage = + 'The URL\'s protocol must be one of "ws:", "wss:", ' + + '"http:", "https:", or "ws+unix:"'; + } else if (isIpcUrl && !parsedUrl.pathname) { + invalidUrlMessage = "The URL's pathname is empty"; + } else if (parsedUrl.hash) { + invalidUrlMessage = 'The URL contains a fragment identifier'; + } + + if (invalidUrlMessage) { + const err = new SyntaxError(invalidUrlMessage); + + if (websocket._redirects === 0) { + throw err; + } else { + emitErrorAndClose(websocket, err); + return; + } + } + + const defaultPort = isSecure ? 443 : 80; + const key = randomBytes(16).toString('base64'); + const request = isSecure ? https.request : http.request; + const protocolSet = new Set(); + let perMessageDeflate; + + opts.createConnection = + opts.createConnection || (isSecure ? tlsConnect : netConnect); + opts.defaultPort = opts.defaultPort || defaultPort; + opts.port = parsedUrl.port || defaultPort; + opts.host = parsedUrl.hostname.startsWith('[') + ? parsedUrl.hostname.slice(1, -1) + : parsedUrl.hostname; + opts.headers = { + ...opts.headers, + 'Sec-WebSocket-Version': opts.protocolVersion, + 'Sec-WebSocket-Key': key, + Connection: 'Upgrade', + Upgrade: 'websocket' + }; + opts.path = parsedUrl.pathname + parsedUrl.search; + opts.timeout = opts.handshakeTimeout; + + if (opts.perMessageDeflate) { + perMessageDeflate = new PerMessageDeflate({ + ...opts.perMessageDeflate, + isServer: false, + maxPayload: opts.maxPayload + }); + opts.headers['Sec-WebSocket-Extensions'] = format({ + [PerMessageDeflate.extensionName]: perMessageDeflate.offer() + }); + } + if (protocols.length) { + for (const protocol of protocols) { + if ( + typeof protocol !== 'string' || + !subprotocolRegex.test(protocol) || + protocolSet.has(protocol) + ) { + throw new SyntaxError( + 'An invalid or duplicated subprotocol was specified' + ); + } + + protocolSet.add(protocol); + } + + opts.headers['Sec-WebSocket-Protocol'] = protocols.join(','); + } + if (opts.origin) { + if (opts.protocolVersion < 13) { + opts.headers['Sec-WebSocket-Origin'] = opts.origin; + } else { + opts.headers.Origin = opts.origin; + } + } + if (parsedUrl.username || parsedUrl.password) { + opts.auth = `${parsedUrl.username}:${parsedUrl.password}`; + } + + if (isIpcUrl) { + const parts = opts.path.split(':'); + + opts.socketPath = parts[0]; + opts.path = parts[1]; + } + + let req; + + if (opts.followRedirects) { + if (websocket._redirects === 0) { + websocket._originalIpc = isIpcUrl; + websocket._originalSecure = isSecure; + websocket._originalHostOrSocketPath = isIpcUrl + ? opts.socketPath + : parsedUrl.host; + + const headers = options && options.headers; + + // + // Shallow copy the user provided options so that headers can be changed + // without mutating the original object. + // + options = { ...options, headers: {} }; + + if (headers) { + for (const [key, value] of Object.entries(headers)) { + options.headers[key.toLowerCase()] = value; + } + } + } else if (websocket.listenerCount('redirect') === 0) { + const isSameHost = isIpcUrl + ? websocket._originalIpc + ? opts.socketPath === websocket._originalHostOrSocketPath + : false + : websocket._originalIpc + ? false + : parsedUrl.host === websocket._originalHostOrSocketPath; + + if (!isSameHost || (websocket._originalSecure && !isSecure)) { + // + // Match curl 7.77.0 behavior and drop the following headers. These + // headers are also dropped when following a redirect to a subdomain. + // + delete opts.headers.authorization; + delete opts.headers.cookie; + + if (!isSameHost) delete opts.headers.host; + + opts.auth = undefined; + } + } + + // + // Match curl 7.77.0 behavior and make the first `Authorization` header win. + // If the `Authorization` header is set, then there is nothing to do as it + // will take precedence. + // + if (opts.auth && !options.headers.authorization) { + options.headers.authorization = + 'Basic ' + Buffer.from(opts.auth).toString('base64'); + } + + req = websocket._req = request(opts); + + if (websocket._redirects) { + // + // Unlike what is done for the `'upgrade'` event, no early exit is + // triggered here if the user calls `websocket.close()` or + // `websocket.terminate()` from a listener of the `'redirect'` event. This + // is because the user can also call `request.destroy()` with an error + // before calling `websocket.close()` or `websocket.terminate()` and this + // would result in an error being emitted on the `request` object with no + // `'error'` event listeners attached. + // + websocket.emit('redirect', websocket.url, req); + } + } else { + req = websocket._req = request(opts); + } + + if (opts.timeout) { + req.on('timeout', () => { + abortHandshake(websocket, req, 'Opening handshake has timed out'); + }); + } + + req.on('error', (err) => { + if (req === null || req[kAborted]) return; + + req = websocket._req = null; + emitErrorAndClose(websocket, err); + }); + + req.on('response', (res) => { + const location = res.headers.location; + const statusCode = res.statusCode; + + if ( + location && + opts.followRedirects && + statusCode >= 300 && + statusCode < 400 + ) { + if (++websocket._redirects > opts.maxRedirects) { + abortHandshake(websocket, req, 'Maximum redirects exceeded'); + return; + } + + req.abort(); + + let addr; + + try { + addr = new URL(location, address); + } catch (e) { + const err = new SyntaxError(`Invalid URL: ${location}`); + emitErrorAndClose(websocket, err); + return; + } + + initAsClient(websocket, addr, protocols, options); + } else if (!websocket.emit('unexpected-response', req, res)) { + abortHandshake( + websocket, + req, + `Unexpected server response: ${res.statusCode}` + ); + } + }); + + req.on('upgrade', (res, socket, head) => { + websocket.emit('upgrade', res); + + // + // The user may have closed the connection from a listener of the + // `'upgrade'` event. + // + if (websocket.readyState !== WebSocket.CONNECTING) return; + + req = websocket._req = null; + + const upgrade = res.headers.upgrade; + + if (upgrade === undefined || upgrade.toLowerCase() !== 'websocket') { + abortHandshake(websocket, socket, 'Invalid Upgrade header'); + return; + } + + const digest = createHash('sha1') + .update(key + GUID) + .digest('base64'); + + if (res.headers['sec-websocket-accept'] !== digest) { + abortHandshake(websocket, socket, 'Invalid Sec-WebSocket-Accept header'); + return; + } + + const serverProt = res.headers['sec-websocket-protocol']; + let protError; + + if (serverProt !== undefined) { + if (!protocolSet.size) { + protError = 'Server sent a subprotocol but none was requested'; + } else if (!protocolSet.has(serverProt)) { + protError = 'Server sent an invalid subprotocol'; + } + } else if (protocolSet.size) { + protError = 'Server sent no subprotocol'; + } + + if (protError) { + abortHandshake(websocket, socket, protError); + return; + } + + if (serverProt) websocket._protocol = serverProt; + + const secWebSocketExtensions = res.headers['sec-websocket-extensions']; + + if (secWebSocketExtensions !== undefined) { + if (!perMessageDeflate) { + const message = + 'Server sent a Sec-WebSocket-Extensions header but no extension ' + + 'was requested'; + abortHandshake(websocket, socket, message); + return; + } + + let extensions; + + try { + extensions = parse(secWebSocketExtensions); + } catch (err) { + const message = 'Invalid Sec-WebSocket-Extensions header'; + abortHandshake(websocket, socket, message); + return; + } + + const extensionNames = Object.keys(extensions); + + if ( + extensionNames.length !== 1 || + extensionNames[0] !== PerMessageDeflate.extensionName + ) { + const message = 'Server indicated an extension that was not requested'; + abortHandshake(websocket, socket, message); + return; + } + + try { + perMessageDeflate.accept(extensions[PerMessageDeflate.extensionName]); + } catch (err) { + const message = 'Invalid Sec-WebSocket-Extensions header'; + abortHandshake(websocket, socket, message); + return; + } + + websocket._extensions[PerMessageDeflate.extensionName] = + perMessageDeflate; + } + + websocket.setSocket(socket, head, { + allowSynchronousEvents: opts.allowSynchronousEvents, + generateMask: opts.generateMask, + maxBufferedChunks: opts.maxBufferedChunks, + maxFragments: opts.maxFragments, + maxPayload: opts.maxPayload, + skipUTF8Validation: opts.skipUTF8Validation + }); + }); + + if (opts.finishRequest) { + opts.finishRequest(req, websocket); + } else { + req.end(); + } +} + +/** + * Emit the `'error'` and `'close'` events. + * + * @param {WebSocket} websocket The WebSocket instance + * @param {Error} The error to emit + * @private + */ +function emitErrorAndClose(websocket, err) { + websocket._readyState = WebSocket.CLOSING; + // + // The following assignment is practically useless and is done only for + // consistency. + // + websocket._errorEmitted = true; + websocket.emit('error', err); + websocket.emitClose(); +} + +/** + * Create a `net.Socket` and initiate a connection. + * + * @param {Object} options Connection options + * @return {net.Socket} The newly created socket used to start the connection + * @private + */ +function netConnect(options) { + options.path = options.socketPath; + return net.connect(options); +} + +/** + * Create a `tls.TLSSocket` and initiate a connection. + * + * @param {Object} options Connection options + * @return {tls.TLSSocket} The newly created socket used to start the connection + * @private + */ +function tlsConnect(options) { + options.path = undefined; + + if (!options.servername && options.servername !== '') { + options.servername = net.isIP(options.host) ? '' : options.host; + } + + return tls.connect(options); +} + +/** + * Abort the handshake and emit an error. + * + * @param {WebSocket} websocket The WebSocket instance + * @param {(http.ClientRequest|net.Socket|tls.Socket)} stream The request to + * abort or the socket to destroy + * @param {String} message The error message + * @private + */ +function abortHandshake(websocket, stream, message) { + websocket._readyState = WebSocket.CLOSING; + + const err = new Error(message); + Error.captureStackTrace(err, abortHandshake); + + if (stream.setHeader) { + stream[kAborted] = true; + stream.abort(); + + if (stream.socket && !stream.socket.destroyed) { + // + // On Node.js >= 14.3.0 `request.abort()` does not destroy the socket if + // called after the request completed. See + // https://github.com/websockets/ws/issues/1869. + // + stream.socket.destroy(); + } + + process.nextTick(emitErrorAndClose, websocket, err); + } else { + stream.destroy(err); + stream.once('error', websocket.emit.bind(websocket, 'error')); + stream.once('close', websocket.emitClose.bind(websocket)); + } +} + +/** + * Handle cases where the `ping()`, `pong()`, or `send()` methods are called + * when the `readyState` attribute is `CLOSING` or `CLOSED`. + * + * @param {WebSocket} websocket The WebSocket instance + * @param {*} [data] The data to send + * @param {Function} [cb] Callback + * @private + */ +function sendAfterClose(websocket, data, cb) { + if (data) { + const length = isBlob(data) ? data.size : toBuffer(data).length; + + // + // The `_bufferedAmount` property is used only when the peer is a client and + // the opening handshake fails. Under these circumstances, in fact, the + // `setSocket()` method is not called, so the `_socket` and `_sender` + // properties are set to `null`. + // + if (websocket._socket) websocket._sender._bufferedBytes += length; + else websocket._bufferedAmount += length; + } + + if (cb) { + const err = new Error( + `WebSocket is not open: readyState ${websocket.readyState} ` + + `(${readyStates[websocket.readyState]})` + ); + process.nextTick(cb, err); + } +} + +/** + * The listener of the `Receiver` `'conclude'` event. + * + * @param {Number} code The status code + * @param {Buffer} reason The reason for closing + * @private + */ +function receiverOnConclude(code, reason) { + const websocket = this[kWebSocket]; + + websocket._closeFrameReceived = true; + websocket._closeMessage = reason; + websocket._closeCode = code; + + if (websocket._socket[kWebSocket] === undefined) return; + + websocket._socket.removeListener('data', socketOnData); + process.nextTick(resume, websocket._socket); + + if (code === 1005) websocket.close(); + else websocket.close(code, reason); +} + +/** + * The listener of the `Receiver` `'drain'` event. + * + * @private + */ +function receiverOnDrain() { + const websocket = this[kWebSocket]; + + if (!websocket.isPaused) websocket._socket.resume(); +} + +/** + * The listener of the `Receiver` `'error'` event. + * + * @param {(RangeError|Error)} err The emitted error + * @private + */ +function receiverOnError(err) { + const websocket = this[kWebSocket]; + + if (websocket._socket[kWebSocket] !== undefined) { + websocket._socket.removeListener('data', socketOnData); + + // + // On Node.js < 14.0.0 the `'error'` event is emitted synchronously. See + // https://github.com/websockets/ws/issues/1940. + // + process.nextTick(resume, websocket._socket); + + websocket.close(err[kStatusCode]); + } + + if (!websocket._errorEmitted) { + websocket._errorEmitted = true; + websocket.emit('error', err); + } +} + +/** + * The listener of the `Receiver` `'finish'` event. + * + * @private + */ +function receiverOnFinish() { + this[kWebSocket].emitClose(); +} + +/** + * The listener of the `Receiver` `'message'` event. + * + * @param {Buffer|ArrayBuffer|Buffer[])} data The message + * @param {Boolean} isBinary Specifies whether the message is binary or not + * @private + */ +function receiverOnMessage(data, isBinary) { + this[kWebSocket].emit('message', data, isBinary); +} + +/** + * The listener of the `Receiver` `'ping'` event. + * + * @param {Buffer} data The data included in the ping frame + * @private + */ +function receiverOnPing(data) { + const websocket = this[kWebSocket]; + + if (websocket._autoPong) websocket.pong(data, !this._isServer, NOOP); + websocket.emit('ping', data); +} + +/** + * The listener of the `Receiver` `'pong'` event. + * + * @param {Buffer} data The data included in the pong frame + * @private + */ +function receiverOnPong(data) { + this[kWebSocket].emit('pong', data); +} + +/** + * Resume a readable stream + * + * @param {Readable} stream The readable stream + * @private + */ +function resume(stream) { + stream.resume(); +} + +/** + * The `Sender` error event handler. + * + * @param {Error} The error + * @private + */ +function senderOnError(err) { + const websocket = this[kWebSocket]; + + if (websocket.readyState === WebSocket.CLOSED) return; + if (websocket.readyState === WebSocket.OPEN) { + websocket._readyState = WebSocket.CLOSING; + setCloseTimer(websocket); + } + + // + // `socket.end()` is used instead of `socket.destroy()` to allow the other + // peer to finish sending queued data. There is no need to set a timer here + // because `CLOSING` means that it is already set or not needed. + // + this._socket.end(); + + if (!websocket._errorEmitted) { + websocket._errorEmitted = true; + websocket.emit('error', err); + } +} + +/** + * Set a timer to destroy the underlying raw socket of a WebSocket. + * + * @param {WebSocket} websocket The WebSocket instance + * @private + */ +function setCloseTimer(websocket) { + websocket._closeTimer = setTimeout( + websocket._socket.destroy.bind(websocket._socket), + websocket._closeTimeout + ); +} + +/** + * The listener of the socket `'close'` event. + * + * @private + */ +function socketOnClose() { + const websocket = this[kWebSocket]; + + this.removeListener('close', socketOnClose); + this.removeListener('data', socketOnData); + this.removeListener('end', socketOnEnd); + + websocket._readyState = WebSocket.CLOSING; + + // + // The close frame might not have been received or the `'end'` event emitted, + // for example, if the socket was destroyed due to an error. Ensure that the + // `receiver` stream is closed after writing any remaining buffered data to + // it. If the readable side of the socket is in flowing mode then there is no + // buffered data as everything has been already written. If instead, the + // socket is paused, any possible buffered data will be read as a single + // chunk. + // + if ( + !this._readableState.endEmitted && + !websocket._closeFrameReceived && + !websocket._receiver._writableState.errorEmitted && + this._readableState.length !== 0 + ) { + const chunk = this.read(this._readableState.length); + + websocket._receiver.write(chunk); + } + + websocket._receiver.end(); + + this[kWebSocket] = undefined; + + clearTimeout(websocket._closeTimer); + + if ( + websocket._receiver._writableState.finished || + websocket._receiver._writableState.errorEmitted + ) { + websocket.emitClose(); + } else { + websocket._receiver.on('error', receiverOnFinish); + websocket._receiver.on('finish', receiverOnFinish); + } +} + +/** + * The listener of the socket `'data'` event. + * + * @param {Buffer} chunk A chunk of data + * @private + */ +function socketOnData(chunk) { + if (!this[kWebSocket]._receiver.write(chunk)) { + this.pause(); + } +} + +/** + * The listener of the socket `'end'` event. + * + * @private + */ +function socketOnEnd() { + const websocket = this[kWebSocket]; + + websocket._readyState = WebSocket.CLOSING; + websocket._receiver.end(); + this.end(); +} + +/** + * The listener of the socket `'error'` event. + * + * @private + */ +function socketOnError() { + const websocket = this[kWebSocket]; + + this.removeListener('error', socketOnError); + this.on('error', NOOP); + + if (websocket) { + websocket._readyState = WebSocket.CLOSING; + this.destroy(); + } +} diff --git a/web/node_modules/ws/package.json b/web/node_modules/ws/package.json new file mode 100644 index 0000000000000000000000000000000000000000..9116e663298ebdb1321f29b44463a25d76d4791b --- /dev/null +++ b/web/node_modules/ws/package.json @@ -0,0 +1,70 @@ +{ + "name": "ws", + "version": "8.21.0", + "description": "Simple to use, blazing fast and thoroughly tested websocket client and server for Node.js", + "keywords": [ + "HyBi", + "Push", + "RFC-6455", + "WebSocket", + "WebSockets", + "real-time" + ], + "homepage": "https://github.com/websockets/ws", + "bugs": "https://github.com/websockets/ws/issues", + "repository": { + "type": "git", + "url": "git+https://github.com/websockets/ws.git" + }, + "author": "Einar Otto Stangvik (http://2x.io)", + "license": "MIT", + "main": "index.js", + "exports": { + ".": { + "browser": "./browser.js", + "import": "./wrapper.mjs", + "require": "./index.js" + }, + "./package.json": "./package.json" + }, + "browser": "browser.js", + "engines": { + "node": ">=10.0.0" + }, + "files": [ + "browser.js", + "index.js", + "lib/*.js", + "wrapper.mjs" + ], + "scripts": { + "test": "nyc --reporter=lcov --reporter=text mocha --throw-deprecation test/*.test.js", + "integration": "mocha --throw-deprecation test/*.integration.js", + "lint": "eslint . && prettier --check --ignore-path .gitignore \"**/*.{json,md,yaml,yml}\"" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + }, + "devDependencies": { + "@eslint/js": "^10.0.1", + "benchmark": "^2.1.4", + "bufferutil": "^4.0.1", + "eslint": "^10.0.1", + "eslint-config-prettier": "^10.0.1", + "eslint-plugin-prettier": "^5.0.0", + "globals": "^17.0.0", + "mocha": "^8.4.0", + "nyc": "^15.0.0", + "prettier": "^3.0.0", + "utf-8-validate": "^6.0.0" + } +} diff --git a/web/node_modules/ws/wrapper.mjs b/web/node_modules/ws/wrapper.mjs new file mode 100644 index 0000000000000000000000000000000000000000..a8ffabbb0e7de8c578eeb056907b58ea31cc3b22 --- /dev/null +++ b/web/node_modules/ws/wrapper.mjs @@ -0,0 +1,21 @@ +import createWebSocketStream from './lib/stream.js'; +import extension from './lib/extension.js'; +import PerMessageDeflate from './lib/permessage-deflate.js'; +import Receiver from './lib/receiver.js'; +import Sender from './lib/sender.js'; +import subprotocol from './lib/subprotocol.js'; +import WebSocket from './lib/websocket.js'; +import WebSocketServer from './lib/websocket-server.js'; + +export { + createWebSocketStream, + extension, + PerMessageDeflate, + Receiver, + Sender, + subprotocol, + WebSocket, + WebSocketServer +}; + +export default WebSocket; diff --git a/web/package-lock.json b/web/package-lock.json index 1bb8919929489ffad52557b78a6db690d2c701e9..ea593cf2835baeb99b58a63ecdd6d3702813413d 100644 --- a/web/package-lock.json +++ b/web/package-lock.json @@ -9,9 +9,16 @@ "version": "0.1.0", "license": "MIT", "dependencies": { + "hyparquet": "^1.26.2", "ws": "^8.21.0" } }, + "node_modules/hyparquet": { + "version": "1.26.2", + "resolved": "https://registry.npmjs.org/hyparquet/-/hyparquet-1.26.2.tgz", + "integrity": "sha512-6qjyK7R2tZ4vnYP1J8NpcCeDPK1UIYk3xh5XgtQUnUM1iwkYbvI6Mz3xtmpMd6AfCCeoUoJVbEq29k5GReZRWA==", + "license": "MIT" + }, "node_modules/ws": { "version": "8.21.0", "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz", diff --git a/web/package.json b/web/package.json index 26e25de6bbdead906b019b60e022cbbf81fbac7b..2886687eb222d3aaa40bed4299df892bf415e8d3 100644 --- a/web/package.json +++ b/web/package.json @@ -6,9 +6,10 @@ "author": "Dean Byrne (Quazim0t0) / DaisyChainAI", "scripts": { "start": "node server.js", - "test": "node test_core.js" + "test": "node test_core.js && node test_verified.js && node test_ieee.js && node test_gates.js && node test_metamorphic.js && node test_corpus.js && node test_optimizer.js && node test_transformer.js && node test_unit_backward.js" }, "dependencies": { + "hyparquet": "^1.26.2", "ws": "^8.21.0" } } diff --git a/web/public/app.js b/web/public/app.js index 25e3fc7f5330a6182e7f3a1f6fa0104c63e020b9..a28d15f7648098e8d2edbb10439c4d6d0ca8dadf 100644 --- a/web/public/app.js +++ b/web/public/app.js @@ -38,7 +38,6 @@ const ui = { cfgB: document.getElementById("cfgB"), cfgSteps: document.getElementById("cfgSteps"), cfgLr: document.getElementById("cfgLr"), - cfgData: document.getElementById("cfgData"), dataset: document.getElementById("dataset"), tokenizer: document.getElementById("tokenizer"), kit: document.getElementById("kit"), @@ -395,16 +394,17 @@ const CFG_SENTINEL = -3; // wire: [int32 -3][int32 c,t,b,s function readCfgFromUI() { // lr crosses the wire as float32 — fround here so the leader trains with // the exact same value the followers decode (else Adam forks the weights) + // ds is FIXED: FineWeb-Edu is the only dataset (still broadcast, so mixed + // old/new clients keep interoperating — followers just ignore the choice) return { c: +ui.cfgC.value, t: +ui.cfgT.value, b: +ui.cfgB.value, steps: +ui.cfgSteps.value, lr: Math.fround(+ui.cfgLr.value / 1000), - ds: (ui.cfgData.value || "").trim() }; + ds: "HuggingFaceFW/fineweb-edu" }; } function showCfgInUI(cfg) { const set = (el, vid, val) => { el.value = val; document.getElementById(vid).textContent = val; }; set(ui.cfgC, "vcfgC", cfg.c); set(ui.cfgT, "vcfgT", cfg.t); set(ui.cfgB, "vcfgB", cfg.b); set(ui.cfgSteps, "vcfgSteps", cfg.steps); set(ui.cfgLr, "vcfgLr", Math.round(cfg.lr * 1000)); - if (cfg.ds) ui.cfgData.value = cfg.ds; } // switch the training text to a chosen dataset (whole group uses the same one); // on failure the built-in corpus stays and we log why @@ -606,6 +606,28 @@ function hashWeights() { // FNV-1a over the flat param by for (let i = 0; i < b.length; i++) { h ^= b[i]; h = Math.imul(h, 0x01000193); } return h >>> 0; } +// ---- roster-gradient repair -------------------------------------------------- +// Gradients travel peer-to-peer over a mesh, and the mesh can be asymmetric: a +// peer's gradient reaches the leader (so it enters the roster) but never +// reaches some follower — one flaky WebRTC link. The follower then can't +// average (partial average = forked weights) and used to hard-stop the whole +// run. But the LEADER holds every roster gradient by construction (the roster +// IS the set that reached it), so the follower asks the leader to forward the +// missing bytes instead of dying. Bit-exact: the same Float32Array bytes the +// origin peer broadcast. The leader retains a few recent steps to serve these +// (it steps at ~8s while a follower is stalled — its own cohort timeout — so a +// small window covers the repair round trip). +// wires: request [i32 -7][i32 step][i32 peerNum] +// response [i32 -8][i32 step][i32 peerNum][u32 whash][u32 phash][f32 loss][f32 grad...] +const REPAIRQ_SENTINEL = -7, REPAIRG_SENTINEL = -8; +const REPAIR_RETAIN = 8; // steps of grads kept for repair +function requestRepair(step, peerId2) { + const dc = chans.get(leaderId); + if (!dc || dc.readyState !== "open") return; + const rq = new ArrayBuffer(12); + new Int32Array(rq).set([REPAIRQ_SENTINEL, step, +peerId2.slice(1)]); + return dcSend(dc, rq); +} // roster wire: [int32 -4][int32 step][int32 n][int32 peerNums...] const ROSTER_SENTINEL = -4; function broadcastRoster(step, ids) { @@ -639,6 +661,33 @@ function onGrad(peerId, buf) { rosters.set(iv[1], [...iv.slice(3, 3 + iv[2])].map(n => "p" + n)); wake(); return; } + if (step === REPAIRQ_SENTINEL) { // a peer asks for a grad it never got + const [, rstep, num] = new Int32Array(buf, 0, 3); + const pid = "p" + num; + const g = (incoming.get(rstep) || new Map()).get(pid); + const meta = (peerHashes.get(rstep) || new Map()).get(pid); + if (!g || !meta) return; // already evicted — requester halts as before + const out = new ArrayBuffer(24 + g.byteLength); + new Int32Array(out, 0, 3).set([REPAIRG_SENTINEL, rstep, num]); + new Uint32Array(out, 12, 2).set([meta.hash, meta.phash]); + new Float32Array(out, 20, 1)[0] = meta.loss; + new Float32Array(out, 24).set(g); + const dc = chans.get(peerId); + if (dc && dc.readyState === "open") dcSend(dc, out); + return; + } + if (step === REPAIRG_SENTINEL) { // leader-forwarded copy of a missing grad + if (peerId !== leaderId) return; // only the leader may vouch for another peer's grad + const iv = new Int32Array(buf, 0, 3); + const rstep = iv[1], pid = "p" + iv[2]; + const [rwhash, rphash] = new Uint32Array(buf, 12, 2); + const rloss = new Float32Array(buf, 20, 1)[0]; + if (!incoming.has(rstep)) incoming.set(rstep, new Map()); + if (!incoming.get(rstep).has(pid)) incoming.get(rstep).set(pid, new Float32Array(buf.slice(24))); + if (!peerHashes.has(rstep)) peerHashes.set(rstep, new Map()); + if (!peerHashes.get(rstep).has(pid)) peerHashes.get(rstep).set(pid, { hash: rwhash, phash: rphash, loss: rloss }); + wake(); return; + } const [whash, phash] = new Uint32Array(buf, 4, 2); const loss = new Float32Array(buf, 12, 1)[0]; const grad = new Float32Array(buf.slice(16)); @@ -743,6 +792,12 @@ async function train(cfg, resume) { if (auditFailure) { halted = auditFailure; break; } lastLocal = { s, loss, grad }; await broadcastGrad(s, whash, probeHash, loss, grad); + // keep my own grad addressable too, so repair requests for MY grad + // (a peer that never received it) can be served like any other + if (!incoming.has(s)) incoming.set(s, new Map()); + incoming.get(s).set(myId, grad); + if (!peerHashes.has(s)) peerHashes.set(s, new Map()); + peerHashes.get(s).set(myId, { hash: whash, phash: probeHash, loss }); } } let all; @@ -794,7 +849,17 @@ async function train(cfg, resume) { } const roster = rosters.get(s); const need = roster.filter(id => id !== myId); - const ok = await waitFor(() => need.every(id => (incoming.get(s) || new Map()).has(id)), 10000); + const haveAll = () => need.every(id => (incoming.get(s) || new Map()).has(id)); + let ok = await waitFor(haveAll, 2500); + if (!ok && chans.has(leaderId)) { + // mesh asymmetry: the grad reached the leader (it's in the roster) but + // not me — ask the leader to forward it rather than stopping the run + const missing = need.filter(id => !(incoming.get(s) || new Map()).has(id)); + log(`step ${s + 1}: still missing ${missing.map(nmeOf).join(", ")} — requesting repair from ${nmeOf(leaderId)}`); + for (const id of missing) await requestRepair(s, id); + ok = await waitFor(haveAll, 12000); + if (ok) log(`step ${s + 1}: repaired — continuing`); + } if (!ok) { halted = `missing a roster gradient at step ${s + 1} — applying a partial average would fork the weights`; break; @@ -831,7 +896,8 @@ async function train(cfg, resume) { const avg = TrainCore.averageGrads(all); const upd = opt.step(avg); // DaisyAdam on the cluster-avg grad Transformer.applyUpdate(model, upd); // W -= upd (lr folded into upd) - incoming.delete(s); rosters.delete(s); peerHashes.delete(s); + // retain a few applied steps so stalled peers can repair (evict beyond that) + incoming.delete(s - REPAIR_RETAIN); rosters.delete(s - REPAIR_RETAIN); peerHashes.delete(s - REPAIR_RETAIN); trainedSteps++; if (s % 10 === 0 || s === steps - 1) { ui.loss.textContent = clusterLoss.toFixed(5); @@ -976,8 +1042,8 @@ function buildModel(cfg) { navigator.connection.addEventListener("change", () => { if (!ws || ws.readyState > 1) { log("network changed — reconnecting"); connectSignaling(); } }); - // dataset: stream the chosen dataset from HuggingFace; built-in corpus if offline - loadDataset((ui.cfgData.value || "").trim()); + // dataset: FineWeb-Edu (hardcoded), streamed from this Space's /data endpoint + loadDataset("HuggingFaceFW/fineweb-edu"); ui.start.onclick = async () => { if (training) return; const cfg = readCfgFromUI(); diff --git a/web/public/index.html b/web/public/index.html index 502b30ba46f4ec87b3e31c0b6c8afd02f55161d3..8d0bf6a57922b64648a74d24054406de501ee2e0 100644 --- a/web/public/index.html +++ b/web/public/index.html @@ -118,10 +118,8 @@ - - -

Any public dataset with a text column works — each device streams its own random slice. Whoever presses Start picks it for the whole group; if streaming fails, the built-in corpus is used.

+ +

FineWeb-Edu (HuggingFaceFW/fineweb-edu, 10BT sample) — fixed. Each device pulls its own random slice of the parquet shards, served by this Space; if streaming fails, the built-in corpus is used.

This is pretraining, not fine-tuning — every run trains a brand-new model from random weights. You are watching a language model learn from scratch.

A mini transformer language model — attention, MLP blocks, next-character prediction — every multiply through the verified INT8 units. Training text is streamed from FineWeb-Edu (HuggingFace); offline devices fall back to a built-in corpus. Whoever presses Start sets the settings for the whole group; total batch scales with every device that joins.

diff --git a/web/public/transformer.js b/web/public/transformer.js index 75e09221cf886a236d5268a74204a6b4f7bb066b..c8d5829b432ed28dd13994b59e14ccff1f533596 100644 --- a/web/public/transformer.js +++ b/web/public/transformer.js @@ -87,28 +87,24 @@ let IDS = encode(CORPUS); let DATASET = "built-in corpus"; - // Stream real training text from any public HuggingFace dataset with a - // `text` column, via the datasets-server rows API. Each device pulls its own - // random slice (that's data parallelism — batches were always per-device - // anyway). Offline or on API failure the built-in corpus stays in place. + // Training text: FineWeb-Edu (10BT sample), HARDCODED as the only dataset. + // The serving Space reads random slices of the parquet shards straight off + // the HF CDN with range requests (see server.js /data) — no dependency on + // the datasets-server rows API, which 503s routinely. Each device pulls its + // own random slice (that's data parallelism — batches were always + // per-device anyway). Offline or on failure the built-in corpus stays. const DEFAULT_DS = "HuggingFaceFW/fineweb-edu"; - async function streamDataset(ds) { - ds = (ds || DEFAULT_DS).trim(); - if (!/^[\w.-]+\/[\w.-]+$/.test(ds)) throw new Error(`invalid dataset id "${ds}"`); - const offset = Math.floor(Math.random() * 9900); - const url = `https://datasets-server.huggingface.co/rows?dataset=${encodeURIComponent(ds)}` + - `&config=default&split=train&offset=${offset}&length=100`; - const r = await fetch(url); - if (!r.ok) throw new Error(`datasets-server HTTP ${r.status}`); - const j = await r.json(); - const text = j.rows.map(x => String(x.row.text ?? "")).join("\n").replace(/[^\x20-\x7e\n]/g, " "); - if (text.length < 10000) throw new Error("too little text returned (does the dataset have a `text` column?)"); + async function streamDataset() { // dataset choice removed on purpose + const r = await fetch("data"); + if (!r.ok) throw new Error(`/data HTTP ${r.status}`); + const text = (await r.text()).replace(/[^\x20-\x7e\n]/g, " "); + if (text.length < 10000) throw new Error("too little text returned"); CORPUS = text.slice(0, 500000); IDS = encode(CORPUS); - DATASET = `${ds} (streamed)`; + DATASET = `${DEFAULT_DS} · 10BT sample (parquet via this Space)`; return { name: DATASET, chars: CORPUS.length }; } - const streamFineWebEdu = () => streamDataset(DEFAULT_DS); + const streamFineWebEdu = () => streamDataset(); function datasetName() { return DATASET; } // ---- verified matmul: block-scaled INT8 through the units ------------------ @@ -334,26 +330,82 @@ m.ctx.fgemm(cache.dlogits, cache.lnf.y, { m: vocab, k: BT, n: C, transA: true }), m.ctx.fgemm(cache.dlogits, m.emb, { m: BT, k: vocab, n: C }), // split-K shape ]); + } else if (units && m.ctx.bgemm) { + // units + GPU: the g.emb operand is dlogitsᵀ (vocab×BT, ~4M elements), and + // tr() + quantizeRows() is three full passes over it in JS. Quantizing the + // COLUMNS of dlogits directly into transposed int8 is one pass and + // bit-identical: same |max| scan, same rounds, in the same order — only + // the write pattern changes. The GEMM itself still goes through ctx.bgemm + // (exact-gated), and the live-shape audit still samples it. + const quantizeColsAsRows = (X, rows, cols) => { // == quantizeRows(tr(X), cols, rows) + const q = new Int8Array(cols * rows), s = new Float32Array(cols); + for (let c = 0; c < cols; c++) { + let mx = 0; + for (let r = 0; r < rows; r++) { const a = Math.abs(X[r * cols + c]); if (a > mx) mx = a; } + const sc = Math.max(mx / 127, 1e-8); s[c] = sc; + for (let r = 0; r < rows; r++) { + const v = Math.round(X[r * cols + c] / sc); + q[c * rows + r] = v < -128 ? -128 : v > 127 ? 127 : v; + } + } + return { q, s }; + }; + const dlq = quantizeColsAsRows(cache.dlogits, BT, vocab); // dlogitsᵀ quantized, one pass + const wq2 = V.quantizeCols(cache.lnf.y, BT, C); + const dEmb = { m: vocab, k: BT, n: C, batch: 1 }; + const [gEmb, dIn] = await Promise.all([ + m.ctx.bgemm(dlq.q, wq2.q, dlq.s, wq2.s, dEmb), + bmm(cache.dlogits, m.emb, BT, vocab, C), + ]); + if (m.ctx.audit && m.ctx.audit.due()) { + const bad = V.auditTile(dlq.q, wq2.q, dlq.s, wq2.s, dEmb, gEmb, m.ctx.L, m.ctx.audit.cells); + if (bad) m.ctx.audit.fail(bad); + } + g[gi.emb] = gEmb; dlnfIn = dIn; } else { - g[gi.emb] = await bmm(tr(cache.dlogits, BT, vocab), cache.lnf.y, vocab, BT, C); - dlnfIn = await bmm(cache.dlogits, m.emb, BT, vocab, C); + // independent GEMMs — overlap them (these are the two vocab-sized calls, + // the largest in the whole backward; each is its own round trip) + [g[gi.emb], dlnfIn] = await Promise.all([ + bmm(tr(cache.dlogits, BT, vocab), cache.lnf.y, vocab, BT, C), + bmm(cache.dlogits, m.emb, BT, vocab, C), + ]); } let dx = lnBwd(dlnfIn, cache.lnf.y, cache.lnf.sig, BT, C); const scale = 1 / Math.sqrt(hd); + // concat helper for fusing sibling GEMMs into one batched dispatch + const cat = (...arrs) => { + const out = new Float32Array(arrs.reduce((a, x) => a + x.length, 0)); + let o = 0; for (const x of arrs) { out.set(x, o); o += x.length; } + return out; + }; for (let l = layers - 1; l >= 0; l--) { const bl = m.blocks[l], cb = cache.blocks[l]; // mlp: x3 = x2 + relu(ln2 @ W1) @ W2 + // gW2 and dh1 are independent — overlap their dispatches. On GPU each bmm + // is a full upload/submit/readback round trip, so sequential awaits leave + // the GPU idle between every pair; this is pure latency, not arithmetic, + // and each GEMM's int32 accumulation is exact so overlap changes no bit. const dmlpOut = dx; // residual passthrough handled below - g[gi[`b${l}.W2`]] = await bmm(tr(cb.h1, BT, hidden), dmlpOut, hidden, BT, C); - const dh1 = await bmm(dmlpOut, tr(bl.W2, hidden, C), BT, C, hidden); + const [gW2, dh1] = await Promise.all([ + bmm(tr(cb.h1, BT, hidden), dmlpOut, hidden, BT, C), + bmm(dmlpOut, tr(bl.W2, hidden, C), BT, C, hidden), + ]); + g[gi[`b${l}.W2`]] = gW2; for (let i = 0; i < dh1.length; i++) if (!cb.mask[i]) dh1[i] = 0; - g[gi[`b${l}.W1`]] = await bmm(tr(cb.ln2.y, BT, C), dh1, C, BT, hidden); - const dln2in = lnBwd(await bmm(dh1, tr(bl.W1, C, hidden), BT, hidden, C), cb.ln2.y, cb.ln2.sig, BT, C); + const [gW1, dln2raw] = await Promise.all([ + bmm(tr(cb.ln2.y, BT, C), dh1, C, BT, hidden), + bmm(dh1, tr(bl.W1, C, hidden), BT, hidden, C), + ]); + g[gi[`b${l}.W1`]] = gW1; + const dln2in = lnBwd(dln2raw, cb.ln2.y, cb.ln2.sig, BT, C); const dx2 = new Float32Array(BT * C); for (let i = 0; i < dx2.length; i++) dx2[i] = dx[i] + dln2in[i]; // attention: x2 = xin + (ctxOut @ Wo) - g[gi[`b${l}.Wo`]] = await bmm(tr(cb.ctxOut, BT, C), dx2, C, BT, C); - const dctx = await bmm(dx2, tr(bl.Wo, C, C), BT, C, C); + const [gWo, dctx] = await Promise.all([ + bmm(tr(cb.ctxOut, BT, C), dx2, C, BT, C), + bmm(dx2, tr(bl.Wo, C, C), BT, C, C), + ]); + g[gi[`b${l}.Wo`]] = gWo; // gather every head once, then run each stage as ONE batched GEMM over all // B*heads problems: 4 dispatches per layer instead of 4 per head. const BH = B * heads; @@ -363,8 +415,10 @@ const dchb = gatherHeads(dctx, B, T, C, heads, hd); const aT = batchedTranspose(cb.aAll, BH, T, T); // BH×T×T const vT = batchedTranspose(vb, BH, T, hd); // BH×hd×T - const dvAll = await bmmB(aT, dchb, T, T, hd, BH); // aᵀ @ dctx - const daAll = await bmmB(dchb, vT, T, hd, T, BH); // dctx @ vᵀ + const [dvAll, daAll] = await Promise.all([ + bmmB(aT, dchb, T, T, hd, BH), // aᵀ @ dctx + bmmB(dchb, vT, T, hd, T, BH), // dctx @ vᵀ + ]); // softmax backward is elementwise + a causal row reduction: stays in float // (no matrix math here, so nothing for the units to do) const dsAll = new Float32Array(BH * T * T); @@ -378,18 +432,36 @@ } } const dsT = batchedTranspose(dsAll, BH, T, T); - const dqAll = await bmmB(dsAll, kb, T, T, hd, BH); // ds @ k - const dkAll = await bmmB(dsT, qb, T, T, hd, BH); // dsᵀ @ q + const [dqAll, dkAll] = await Promise.all([ + bmmB(dsAll, kb, T, T, hd, BH), // ds @ k + bmmB(dsT, qb, T, T, hd, BH), // dsᵀ @ q + ]); const dq = new Float32Array(BT * C), dk = new Float32Array(BT * C), dv = new Float32Array(BT * C); scatterHeadsAcc(dq, dqAll, B, T, C, heads, hd); scatterHeadsAcc(dk, dkAll, B, T, C, heads, hd); scatterHeadsAcc(dv, dvAll, B, T, C, heads, hd); - g[gi[`b${l}.Wq`]] = await bmm(tr(cb.ln1.y, BT, C), dq, C, BT, C); - g[gi[`b${l}.Wk`]] = await bmm(tr(cb.ln1.y, BT, C), dk, C, BT, C); - g[gi[`b${l}.Wv`]] = await bmm(tr(cb.ln1.y, BT, C), dv, C, BT, C); - const dln1in = new Float32Array(BT * C); - const addIn = async (dsrc, W) => { const t2 = await bmm(dsrc, tr(W, C, C), BT, C, C); for (let i = 0; i < dln1in.length; i++) dln1in[i] += t2[i]; }; - await addIn(dq, bl.Wq); await addIn(dk, bl.Wk); await addIn(dv, bl.Wv); + // The QKV weight grads share the same left operand (ln1ᵀ), and the three + // dln1in terms share one shape — each trio fuses into ONE batched GEMM + // (batch=3) instead of three dispatches. Bit-identical to separate calls: + // block scales are per-row of X and per-column of W PER BATCH ELEMENT, so + // concatenation changes no scale and no product. + const ln1T = tr(cb.ln1.y, BT, C); + const [gQKV, dIn3] = await Promise.all([ + bmmB(cat(ln1T, ln1T, ln1T), cat(dq, dk, dv), C, BT, C, 3), + bmmB(cat(dq, dk, dv), cat(tr(bl.Wq, C, C), tr(bl.Wk, C, C), tr(bl.Wv, C, C)), BT, C, C, 3), + ]); + const CC = C * C, BTC = BT * C; + g[gi[`b${l}.Wq`]] = gQKV.slice(0, CC); + g[gi[`b${l}.Wk`]] = gQKV.slice(CC, 2 * CC); + g[gi[`b${l}.Wv`]] = gQKV.slice(2 * CC, 3 * CC); + // sum the three dln1in terms in q,k,v order with an f32 round after EACH + // add — the old code accumulated into a Float32Array element three times, + // which rounds per step; a bare q+k+v here would run in f64 and round + // once, a last-ulp difference that forks replicas. (Exactly the epilogue + // mirror lesson: match the rounding schedule, not just the values.) + const dln1in = new Float32Array(BTC); + for (let i = 0; i < BTC; i++) + dln1in[i] = Math.fround(Math.fround(dIn3[i] + dIn3[BTC + i]) + dIn3[2 * BTC + i]); const dxin = lnBwd(dln1in, cb.ln1.y, cb.ln1.sig, BT, C); dx = new Float32Array(BT * C); for (let i = 0; i < dx.length; i++) dx[i] = dx2[i] + dxin[i]; diff --git a/web/server.js b/web/server.js index c25a855bf9e192e98516cfa0826da7bb3b4a76c2..b0fd6ce13cf70e1441f23987b09dc7a230f33d0b 100644 --- a/web/server.js +++ b/web/server.js @@ -30,6 +30,39 @@ const PUB = path.join(__dirname, "public"); const TYPES = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".json": "application/json" }; +// ---- training text: FineWeb-Edu, served by THIS server ---------------------- +// The browser used to call datasets-server.huggingface.co/rows directly, which +// 503s routinely under load. The parquet files themselves sit on the HF CDN +// behind plain resolve/ URLs — public, range-request friendly, and far more +// reliable than the rows API. So this endpoint reads a random slice of a +// random shard of the 10BT sample straight off the CDN with hyparquet (pure +// JS, no native deps) and hands the browser plain text. FineWeb-Edu is the +// ONLY dataset; there is no pass-through of client-chosen dataset ids. +const FWE_BASE = "https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/resolve/main/sample/10BT/"; +const FWE_SHARDS = Array.from({ length: 14 }, (_, i) => `${String(i).padStart(3, "0")}_00000.parquet`); +const FWE_ROWS_PER_SLICE = 130; // ~550k chars, what one training run consumes +const fweCache = new Map(); // shard -> { file, metadata } (metadata is ~kBs) +let hyparquet = null; // ESM — loaded on first use + +async function fweSlice() { + hyparquet = hyparquet || await import("hyparquet"); + const shard = FWE_SHARDS[Math.floor(Math.random() * FWE_SHARDS.length)]; + let entry = fweCache.get(shard); + if (!entry) { + const file = await hyparquet.asyncBufferFromUrl({ url: FWE_BASE + shard }); + const metadata = await hyparquet.parquetMetadataAsync(file); + entry = { file, metadata }; + fweCache.set(shard, entry); + } + const total = Number(entry.metadata.num_rows); + const start = Math.floor(Math.random() * Math.max(1, total - FWE_ROWS_PER_SLICE)); + const rows = await hyparquet.parquetReadObjects({ + file: entry.file, metadata: entry.metadata, columns: ["text"], + rowStart: start, rowEnd: start + FWE_ROWS_PER_SLICE, + }); + return { shard, start, text: rows.map(r => r.text).join("\n") }; +} + // keepalive: proxies (incl. HuggingFace's) close idle WebSockets — ping every // 30s keeps the pipe warm and detects dead clients within a minute setInterval(() => { @@ -53,6 +86,19 @@ const server = http.createServer((req, res) => { try { if (process.env.DAISY_RTC_CONFIG) rtc = JSON.parse(process.env.DAISY_RTC_CONFIG); } catch (e) {} return res.end(JSON.stringify({ forceRooms: !!process.env.DAISY_FORCE_ROOMS, rtc })); } + if (p === "/data") { // a fresh random slice of FineWeb-Edu + return fweSlice() + .then(({ shard, start, text }) => { + res.writeHead(200, { "Content-Type": "text/plain; charset=utf-8", + "X-FWE-Shard": shard, "X-FWE-Row": String(start) }); + res.end(text); + }) + .catch((e) => { + console.error("fineweb slice failed:", e.message); + res.writeHead(503, { "Content-Type": "text/plain" }); + res.end("fineweb-edu unavailable: " + e.message); + }); + } if (p === "/") p = "/index.html"; const file = path.join(PUB, path.normalize(p)); if (!file.startsWith(PUB)) { res.writeHead(403); return res.end(); } diff --git a/web/test_corpus.js b/web/test_corpus.js new file mode 100644 index 0000000000000000000000000000000000000000..baccabfe45a559bb4623c7edc53e79e178c14206 --- /dev/null +++ b/web/test_corpus.js @@ -0,0 +1,223 @@ +// Mutation-scoring the ORACLES, using a bug taxonomy someone else wrote. +// +// Bugs ported from dipankarsarkar/gpuemu-corpus (the 10 buggy variants). Their +// kernels are Triton/numpy and mine are WGSL/JS, so nothing runs across. What +// ports is the taxonomy: each bug is specified, so it can be re-injected into a +// different codebase. The point is that the bug list has a different author than +// the checks, which is the one thing I cannot manufacture myself. +// +// The question is NOT "do my kernels pass". It is "which of my checks catch a +// bug I did not think of". Every bug an oracle waves through is a hole with a +// name on it instead of a feeling about my own coverage. +const fs = require("fs"); +const path = require("path"); +const V = require("./public/verified_core.js"); +const L = { mul: new Int16Array(fs.readFileSync(path.join(__dirname, "public", "mul_lut.bin")).buffer.slice(0)) }; + +const f32 = Math.fround; +const randf = (n) => Float32Array.from({ length: n }, () => Math.random() * 2 - 1); + +// ---- MY kernel, with one corpus bug injected at a time ---------------------- +// Faithful to the real bgemmJS/WGSL structure, so the injected bug is the only +// difference. `bug` names map to corpus variants; see TAXONOMY below. +function kernel(bug) { + return (Xq, Wq, rs, cs, d) => { + const { m, k, n } = d, batch = d.batch || 1, relu = !!d.relu, mul = L.mul, raw = !!d.acc; + const out = raw ? new Int32Array(batch * m * n) : new Float32Array(batch * m * n); + const acc = new Int32Array(n); + for (let bz = 0; bz < batch; bz++) { + const xo = bz * m * k, wo = bz * k * n, oo = bz * m * n, co = bz * n; + for (let i = 0; i < m; i++) { + acc.fill(0); + const xrow = xo + i * k; + for (let p = 0; p < k; p++) { + const au = (Xq[xrow + p] & 0xFF) * 256, wrow = wo + p * n; + for (let j = 0; j < n; j++) { + const prod = mul[au + (Wq[wrow + j] & 0xFF)]; + // corpus: matmul_triton_buggy — acc= instead of acc+=, "correct only when K=1" + if (bug === "accOverwrite") acc[j] = prod; else acc[j] += prod; + } + } + const orow = oo + i * n; + if (raw) { + for (let j = 0; j < n; j++) out[orow + j] = acc[j]; + continue; + } + const rscale = rs[bz * m + i]; + for (let j = 0; j < n; j++) { + // corpus: softmax_llm_buggy — "correct only when last dim is a multiple + // of 128". My analogue: workgroup_size(8,8), so a missing bounds guard + // is correct only when n is a multiple of 8. + if (bug === "boundsMult8" && j >= (n - (n % 8))) { out[orow + j] = 0; continue; } + let v = V.epi(acc[j], rscale, cs[co + j]); + // corpus: gelu_triton_buggy — a dropped constant factor, uniformly ~2x off + if (bug === "droppedFactor") v = f32(v * 2); + // corpus: leaky_relu_triton_buggy — wrong alpha (0.1 vs 0.01) + if (relu && bug === "leakyAlpha") v = v < 0 ? f32(v * 0.1) : v; + else if (relu && v < 0) v = 0; + out[orow + j] = v; + } + } + } + return out; + }; +} + +// what each injected bug is, and where it lives +const TAXONOMY = { + accOverwrite: { from: "matmul_triton_buggy", where: "loop", desc: "acc= instead of acc+= (only correct at K=1)" }, + boundsMult8: { from: "softmax_llm_buggy", where: "loop", desc: "correct only when n is a multiple of the workgroup width" }, + droppedFactor: { from: "gelu_triton_buggy", where: "math", desc: "constant factor dropped, uniformly 2x" }, + leakyAlpha: { from: "leaky_relu_buggy", where: "math", desc: "wrong negative-slope alpha" }, +}; + +// ---- oracle 1: the property suite (no reference implementation) ------------- +const SHAPES = { m: 6, k: 32, n: 5, batch: 1 }; +function quantize(Xf, Wf, d) { + const batch = d.batch || 1; + const x = V.quantizeRows(Xf, batch * d.m, d.k); + let wq, ws; + if (batch === 1) { const w = V.quantizeCols(Wf, d.k, d.n); wq = w.q; ws = w.s; } + else { + wq = new Int8Array(batch * d.k * d.n); ws = new Float32Array(batch * d.n); + for (let bz = 0; bz < batch; bz++) { + const w = V.quantizeCols(Wf.subarray(bz * d.k * d.n, (bz + 1) * d.k * d.n), d.k, d.n); + wq.set(w.q, bz * d.k * d.n); ws.set(w.s, bz * d.n); + } + } + return { x, wq, ws }; +} +const call = (K, Xf, Wf, d) => { const q = quantize(Xf, Wf, d); return K(q.x.q, q.wq, q.x.s, q.ws, d); }; + +function propertySuite(K) { + // NON-TRIVIALITY. Added after the corpus scored my suite 0/4: a kernel that + // returns all zeros satisfies every relation below, because zero is + // permutation-equivariant, zero-row-preserving and batch-decomposable. Every + // algebraic property suite needs this or it can be passed by doing nothing. + { + const d = { ...SHAPES }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + const o = call(K, A, B, d); + let nz = 0; + for (let i = 0; i < o.length; i++) if (o[i] !== 0) nz++; + if (nz < o.length / 4) return "nonTriviality"; + } + // SENSITIVITY. Every element of A must be able to move its output row. This is + // what catches an accumulator that overwrites instead of accumulating: the + // structure is perfect, every relation holds, but only the last k contributes. + // + // Measured against the RAW accumulator, and perturbed by a sign flip that + // leaves |value| untouched. Both matter: block scaling coupled the row absmax + // to every output in the row, so a naive perturbation moves the output through + // the SCALE and the test passes while proving nothing about the accumulation. + // That is how the first version of this property fooled me. + { + const d = { ...SHAPES, acc: true }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + for (let p = 0; p < d.k; p++) A[1 * d.k + p] = 0.3; + A[1 * d.k + 0] = 1.0; // pin the row absmax at p=0 + const q0 = quantize(A, B, d); + const base = K(q0.x.q, q0.wq, q0.x.s, q0.ws, d); + for (let p = 1; p < d.k; p++) { + const A2 = Float32Array.from(A); + A2[1 * d.k + p] = -0.3; // sign flip: absmax unchanged + const q2 = quantize(A2, B, d); + if (q2.x.s[1] !== q0.x.s[1]) continue; // scale moved anyway: inconclusive + const o2 = K(q2.x.q, q2.wq, q2.x.s, q2.ws, d); + let moved = false; + for (let j = 0; j < d.n; j++) if (o2[1 * d.n + j] !== base[1 * d.n + j]) { moved = true; break; } + if (!moved) return `sensitivity(A[.,${p}] ignored)`; + } + } + // zero row of A -> zero row out + { + const d = { ...SHAPES }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + for (let p = 0; p < d.k; p++) A[2 * d.k + p] = 0; + const o = call(K, A, B, d); + for (let j = 0; j < d.n; j++) if (o[2 * d.n + j] !== 0) return "zeroRow"; + } + // permuting rows of A permutes output rows + { + const d = { ...SHAPES }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + const perm = [3, 1, 5, 0, 4, 2]; + const Ap = new Float32Array(A.length); + perm.forEach((src, dst) => Ap.set(A.subarray(src * d.k, (src + 1) * d.k), dst * d.k)); + const o = call(K, A, B, d), op = call(K, Ap, B, d); + for (let r = 0; r < d.m; r++) for (let j = 0; j < d.n; j++) + if (op[r * d.n + j] !== o[perm[r] * d.n + j]) return "rowPermutation"; + } + // permuting columns of B permutes output columns + { + const d = { ...SHAPES }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + const perm = [2, 0, 4, 1, 3]; + const Bp = new Float32Array(B.length); + for (let p = 0; p < d.k; p++) perm.forEach((src, dst) => { Bp[p * d.n + dst] = B[p * d.n + src]; }); + const o = call(K, A, B, d), op = call(K, A, Bp, d); + for (let r = 0; r < d.m; r++) for (let j = 0; j < d.n; j++) + if (op[r * d.n + j] !== o[r * d.n + perm[j]]) return "colPermutation"; + } + // batched == each element on its own + { + const d = { m: 4, k: 32, n: 5, batch: 3 }; + const A = randf(d.batch * d.m * d.k), B = randf(d.batch * d.k * d.n); + const together = call(K, A, B, d); + for (let bz = 0; bz < d.batch; bz++) { + const alone = call(K, A.subarray(bz * d.m * d.k, (bz + 1) * d.m * d.k), + B.subarray(bz * d.k * d.n, (bz + 1) * d.k * d.n), { ...d, batch: 1 }); + for (let i = 0; i < alone.length; i++) + if (alone[i] !== together[bz * d.m * d.n + i]) return "batchDecomposition"; + } + } + return null; +} + +// ---- oracle 2: the exact differential gate (kernel vs the JS mirror) -------- +function differentialGate(K) { + for (const d of [{ m: 5, k: 9, n: 6, batch: 3, relu: true }, + { m: 32, k: 64, n: 32, batch: 1, relu: false }, + { m: 7, k: 253, n: 5, batch: 2, relu: true }, + { m: 17, k: 33, n: 9, batch: 1, relu: true }]) { + const A = randf(d.batch * d.m * d.k), B = randf(d.batch * d.k * d.n); + const q = quantize(A, B, d); + for (const acc of [true, false]) { + const dd = { ...d, acc }; + const hw = K(q.x.q, q.wq, q.x.s, q.ws, dd); + const ref = V.bgemmJS(q.x.q, q.wq, q.x.s, q.ws, dd, L); + for (let i = 0; i < ref.length; i++) if (hw[i] !== ref[i]) return (acc ? "accumulator" : "epilogue"); + } + } + return null; +} + +// ---- run -------------------------------------------------------------------- +console.log("\nBugs from dipankarsarkar/gpuemu-corpus, re-injected into my kernel."); +console.log("Scoring the ORACLES, not the kernel.\n"); +console.log("bug lives in properties differential"); +console.log("---------------------------------------------------------------"); +const score = { prop: 0, diff: 0, total: 0 }; +for (const [bug, meta] of Object.entries(TAXONOMY)) { + const K = kernel(bug); + const p = propertySuite(K), dgate = differentialGate(K); + score.total++; + if (p) score.prop++; + if (dgate) score.diff++; + console.log(`${bug.padEnd(16)} ${meta.where.padEnd(10)} ${(p ? "CAUGHT (" + p + ")" : "MISSED").padEnd(17)} ${dgate ? "CAUGHT (" + dgate + ")" : "MISSED"}`); +} +// control: the real kernel must pass both +const cleanP = propertySuite(kernel(null)), cleanD = differentialGate(kernel(null)); +console.log(`${"(no bug)".padEnd(16)} ${"-".padEnd(10)} ${(cleanP ? "FALSE POSITIVE" : "clean").padEnd(17)} ${cleanD ? "FALSE POSITIVE" : "clean"}`); + +console.log(`\nmutation score, properties: ${score.prop}/${score.total}`); +console.log(`mutation score, differential: ${score.diff}/${score.total}`); +console.log("\nby where the bug lives:"); +for (const w of ["loop", "math"]) { + const bugs = Object.entries(TAXONOMY).filter(([, m]) => m.where === w); + const caught = bugs.filter(([b]) => propertySuite(kernel(b)) !== null).length; + console.log(` ${w.padEnd(5)} properties caught ${caught}/${bugs.length}`); +} +const ok = !cleanP && !cleanD; +console.log(ok ? "\nCONTROL OK (clean kernel passes both oracles)" : "\nCONTROL FAILED"); +process.exit(ok ? 0 : 1); diff --git a/web/test_ieee.js b/web/test_ieee.js new file mode 100644 index 0000000000000000000000000000000000000000..c59b1c89c2e8e7d071f2d5349b2c73b728b7d636 --- /dev/null +++ b/web/test_ieee.js @@ -0,0 +1,216 @@ +// An IEEE-754 binary32 oracle, written from the standard, in exact integer math. +// +// Why this exists. The kernel epilogue and the JS mirror agree, and I made them +// agree by changing the mirror. Asked what decides which one is wrong when they +// disagree, the honest answer was: me, reasoning about the WGSL spec, with +// nothing enforcing it. That reasoning lived in a paragraph. This file is that +// paragraph made executable. +// +// Nothing here is derived from Verified.epi or from any shader. It is built from +// the binary32 definition: values are sign * mantissa * 2^exp with a 24-bit +// significand, products are computed EXACTLY as BigInt (no float arithmetic +// anywhere in the oracle), and the result is rounded once, round-to-nearest-even, +// per IEEE 754-2019 section 4.3.1. If the mirror and this disagree, the mirror is +// wrong, and no judgement call is involved. + +const fs = require("fs"); +const path = require("path"); +const V = require("./public/verified_core.js"); +const L = { mul: new Int16Array(fs.readFileSync(path.join(__dirname, "public", "mul_lut.bin")).buffer.slice(0)) }; + +// ---- the oracle ------------------------------------------------------------- +function bitLength(n) { let b = 0; while (n > 0n) { n >>= 1n; b++; } return b; } + +// binary32 -> exact (sign, mantissa, exp2) with value = ±mant * 2^exp2 +function decomposeF32(x) { + const dv = new DataView(new ArrayBuffer(4)); + dv.setFloat32(0, x); + const bits = dv.getUint32(0); + const s = bits >>> 31, e = (bits >>> 23) & 0xFF, m = bits & 0x7FFFFF; + if (e === 0xFF) throw new Error("inf/nan out of scope"); + // subnormal: no implicit leading 1, fixed exponent 2^-149 + if (e === 0) return { neg: s === 1, mant: BigInt(m), exp2: -149 }; + // normal: implicit leading 1; 2^(e-127) * (1 + m/2^23) = (2^23 + m) * 2^(e-150) + return { neg: s === 1, mant: BigInt(m) | (1n << 23n), exp2: e - 150 }; +} + +// Round ±mant * 2^exp2 to binary32, round-to-nearest-ties-to-even. +// Two constraints decide how many low bits get discarded: +// normals keep 24 significant bits -> shift = bitLength - 24 +// subnormals snap to the 2^-149 grid -> shift = -exp2 - 149 +// whichever discards more wins, which is exactly the binary32 value set. +function roundToF32(neg, mant, exp2) { + if (mant === 0n) return neg ? -0 : 0; + const bl = bitLength(mant); + const shift = Math.max(bl - 24, -exp2 - 149); + let r, e; + if (shift <= 0) { + r = mant << BigInt(-shift); + e = exp2 + shift; + } else { + const sh = BigInt(shift); + const keep = mant >> sh; + const rem = mant - (keep << sh); // exact remainder, no float involved + const half = 1n << (sh - 1n); + r = keep; + if (rem > half || (rem === half && (keep & 1n) === 1n)) r += 1n; // ties to even + e = exp2 + shift; + } + if (bitLength(r) > 24) { r >>= 1n; e += 1; } // rounding carried into a 25th bit + if (e + bitLength(r) - 1 > 127) return neg ? -Infinity : Infinity; + const val = Number(r) * Math.pow(2, e); // r <= 2^24 and e in range: exact in f64 + return neg ? -val : val; +} + +const i32ToF32Spec = (s) => s === 0 ? 0 : roundToF32(s < 0, BigInt(Math.abs(s)), 0); +// IEEE 754-2019 §6.3: the sign of a product is the XOR of the operand signs, +// even when the product is zero. `a < 0` is false for -0, so the sign test must +// use the sign BIT. (Found by a random sweep here: a subnormal float rounded to +// f32 becomes ±0, and positive*(-0) came back +0 where the spec says -0. The +// epilogue never multiplies by -0 — block scales are clamped >= 1e-8 — but an +// oracle that is wrong about a sign bit is not an oracle.) +const signbit = (x) => x < 0 || Object.is(x, -0); +function mulF32Spec(a, b) { + if (a === 0 || b === 0) return (signbit(a) !== signbit(b)) ? -0 : 0; + const A = decomposeF32(a), B = decomposeF32(b); + return roundToF32(A.neg !== B.neg, A.mant * B.mant, A.exp2 + B.exp2); +} +// what WGSL `f32(s) * a * b` means: left-associative, rounded after every step +const epiSpec = (s, a, b) => mulF32Spec(mulF32Spec(i32ToF32Spec(s), a), b); + +// ---- checks ----------------------------------------------------------------- +let pass = true; +const ok = (c, msg) => { console.log(`${c ? " ok " : " FAIL"} ${msg}`); if (!c) pass = false; }; +const f32 = Math.fround; + +console.log("\nthe oracle reproduces known binary32 behaviour:"); +ok(i32ToF32Spec(1 << 24) === 16777216, "2^24 is exact"); +ok(i32ToF32Spec((1 << 24) + 1) === 16777216, "2^24+1 ties down to even (not 2^24+2)"); +ok(i32ToF32Spec((1 << 24) + 3) === 16777220, "2^24+3 rounds up to the even neighbour"); +ok(i32ToF32Spec(-((1 << 24) + 1)) === -16777216, "sign symmetry on the tie"); +ok(mulF32Spec(f32(0.1), f32(0.1)) === f32(f32(0.1) * f32(0.1)), "0.1*0.1 matches a correctly-rounded f32 multiply"); +ok(mulF32Spec(1.5, 2) === 3, "exact small product"); +ok(mulF32Spec(f32(1e-30), f32(1e-15)) === f32(f32(1e-30) * f32(1e-15)), "underflow into subnormals"); +ok(i32ToF32Spec(0) === 0 && mulF32Spec(0, 5) === 0, "zeros"); +ok(Object.is(mulF32Spec(5, -0), -0) && Object.is(mulF32Spec(-0, -0), 0), "signed zero: sign of a zero product is the XOR of the operand signs"); + +// broad agreement with Math.fround, which is an independently correctly-rounded +// f32 conversion (spec'd by ECMA-262, implemented by V8 — neither is this file) +console.log("\nthe oracle agrees with an independent correctly-rounded implementation:"); +const f32buf = new Float32Array(1); +const asF32 = (x) => { f32buf[0] = x; return f32buf[0]; }; +{ + let bad = 0, n = 0; + const cases = []; + for (let t = 0; t < 200000; t++) cases.push(((Math.random() * 2 - 1) * 2.1e9) | 0); + for (const s of cases) { n++; if (!Object.is(i32ToF32Spec(s), f32(s))) bad++; } + ok(bad === 0, `i32ToF32Spec matches Math.fround on ${n} random int32 (incl. |s| > 2^24)`); +} +{ + // magnitudes chosen to land products in every regime: exact, rounded normal, + // subnormal underflow, and overflow to infinity. The near-zero magnitudes also + // generate ±0 operands, which is what caught the signed-zero bug above. + let bad = 0, n = 0; + const mags = [1, 1e-3, 1e-8, 1e-20, 1e-38, 1e-40, 1e-44, 3e38, 1e30, 127, 1 / 127]; + for (let t = 0; t < 300000; t++) { + const a = asF32((Math.random() * 2 - 1) * mags[(Math.random() * mags.length) | 0]); + const b = asF32((Math.random() * 2 - 1) * mags[(Math.random() * mags.length) | 0]); + n++; + if (!Object.is(mulF32Spec(a, b), f32(a * b))) { // f64 product of f32s is exact, so fround(a*b) is the correctly-rounded f32 product + if (bad++ < 3) console.log(` ${a} * ${b} spec=${mulF32Spec(a, b)} fround=${f32(a * b)}`); + } + } + ok(bad === 0, `mulF32Spec matches the correctly-rounded product on ${n} draws (subnormal..overflow)`); +} + +// ---- the actual point: is Verified.epi the spec, or just agreeable? --------- +console.log("\nVerified.epi vs the oracle, over the range the kernels produce:"); +let mism = 0, n = 0; +for (let t = 0; t < 200000; t++) { + // s spans the int32 accumulator range the LUT path can reach (k up to ~16k, + // products up to 127*127), scales span plausible block-scale magnitudes + const s = Math.round((Math.random() * 2 - 1) * 2.6e8); + const a = f32(Math.random() * 2e-2 + 1e-8); + const b = f32(Math.random() * 2e-2 + 1e-8); + const mine = V.epi(s, a, b), spec = epiSpec(s, a, b); + n++; + if (mine !== spec && !(Number.isNaN(mine) && Number.isNaN(spec))) { + if (mism++ < 3) console.log(` s=${s} a=${a} b=${b} epi=${mine} spec=${spec}`); + } +} +ok(mism === 0, `${n} random triples, ${mism} disagreements`); + +// edge magnitudes, where rounding actually decides things +console.log("\nsame check pinned to the awkward values:"); +let em = 0; +for (const s of [0, 1, -1, (1 << 24) - 1, 1 << 24, (1 << 24) + 1, (1 << 24) + 3, -(1 << 24) - 1, 2 ** 30, -(2 ** 30)]) + for (const a of [f32(1), f32(0.5), f32(1 / 3), f32(1e-20), f32(3e-8)]) + for (const b of [f32(1), f32(2), f32(1 / 7), f32(1e-20)]) { + if (V.epi(s, a, b) !== epiSpec(s, a, b)) { + if (em++ < 3) console.log(` s=${s} a=${a} b=${b} epi=${V.epi(s, a, b)} spec=${epiSpec(s, a, b)}`); + } + } +ok(em === 0, `edge grid, ${em} disagreements`); + +// the tie ladder: s straddling 2^24 with power-of-two scales keeps every product +// dyadic, so f32(s)*a lands EXACTLY on rounding midpoints — where ties-to-even +// and ties-away disagree, and where a round-once mirror slips a whole ulp +{ + let bad = 0, n = 0; + for (let k = 0; k <= 20; k++) { + const a = f32(Math.pow(2, -k)), b = f32(1); + for (let s = (1 << 24) - 4; s <= (1 << 24) + 4; s++) { + n += 2; + if (V.epi(s, a, b) !== epiSpec(s, a, b)) bad++; + if (V.epi(-s, a, b) !== epiSpec(-s, a, b)) bad++; + } + } + ok(bad === 0, `tie-to-even ladder around 2^24, ${n} cases, ${bad} disagreements`); +} + +// end to end: rebuild a live GEMM's outputs from its raw int32 accumulator using +// ONLY the oracle, and compare against the mirror's finished floats. This closes +// the loop from exact integer accumulation to final f32 with nothing but IEEE-754. +{ + const d = { m: 7, k: 40, n: 6, batch: 2 }; + const Xf = Float32Array.from({ length: d.batch * d.m * d.k }, () => Math.random() * 2 - 1); + const Wf = Float32Array.from({ length: d.batch * d.k * d.n }, () => Math.random() * 2 - 1); + const x = V.quantizeRows(Xf, d.batch * d.m, d.k); + const wq = new Int8Array(d.batch * d.k * d.n), ws = new Float32Array(d.batch * d.n); + for (let bz = 0; bz < d.batch; bz++) { + const w = V.quantizeCols(Wf.subarray(bz * d.k * d.n, (bz + 1) * d.k * d.n), d.k, d.n); + wq.set(w.q, bz * d.k * d.n); ws.set(w.s, bz * d.n); + } + const raw = V.bgemmJS(x.q, wq, x.s, ws, { ...d, acc: true }, L); // exact int32 + const fin = V.bgemmJS(x.q, wq, x.s, ws, d, L); // mirror epilogue + let bad = 0; + for (let bz = 0; bz < d.batch; bz++) + for (let i = 0; i < d.m; i++) + for (let j = 0; j < d.n; j++) { + const idx = (bz * d.m + i) * d.n + j; + if (!Object.is(epiSpec(raw[idx], x.s[bz * d.m + i], ws[bz * d.n + j]), fin[idx])) bad++; + } + ok(bad === 0, `bgemmJS outputs rebuilt from the raw accumulator via the oracle (${d.batch}x${d.m}x${d.n}), ${bad} disagreements`); +} + +// ---- prove the oracle bites -------------------------------------------------- +// The mirror USED to do the whole chain in f64 and round once on the store. That +// is not what WGSL computes. The oracle must reject it, or it is not an oracle. +console.log("\nthe oracle rejects the mirror I actually shipped before:"); +const oldEpi = (s, a, b) => f32(s * a * b); // one rounding, not three +let oldBad = 0; +for (let t = 0; t < 200000; t++) { + const s = Math.round((Math.random() * 2 - 1) * 2.6e8); + const a = f32(Math.random() * 2e-2 + 1e-8); + const b = f32(Math.random() * 2e-2 + 1e-8); + if (oldEpi(s, a, b) !== epiSpec(s, a, b)) oldBad++; +} +ok(oldBad > 0, `old single-rounding mirror disagrees on ${oldBad} of 200000 (${(oldBad / 2000).toFixed(2)}% — the bug the 1e-6 tolerance hid)`); +// and a value bug of the kind the corpus uses +const factorBug = (s, a, b) => f32(V.epi(s, a, b) * 2); +ok(factorBug(12345, f32(0.01), f32(0.02)) !== epiSpec(12345, f32(0.01), f32(0.02)), + "oracle catches a dropped-constant-factor bug (corpus: gelu_triton_buggy)"); + +console.log(pass ? "\nIEEE ORACLE TEST PASSED — the mirror's epilogue IS the spec, not merely agreeable." + : "\nIEEE ORACLE TEST FAILED"); +process.exit(pass ? 0 : 1); diff --git a/web/test_metamorphic.js b/web/test_metamorphic.js index a32d90c599b60bbbc0d1335dd2f1d4bcc0ffc769..2efff04859677318500362ea40336ff48152a743 100644 --- a/web/test_metamorphic.js +++ b/web/test_metamorphic.js @@ -24,6 +24,16 @@ const randf = (n, f) => Float32Array.from({ length: n }, f || (() => Math.random function makeKernel(bug) { return async (Xf, Wf, d) => { const { m, k, n } = d, batch = d.batch || 1; + if (bug === "accOverwrite") { // corpus: matmul_triton_buggy + const q = V.quantizeRows(Xf, batch * m, k), w = V.quantizeCols(Wf, k, n); + const out = d.acc ? new Int32Array(batch * m * n) : new Float32Array(batch * m * n); + for (let i = 0; i < m; i++) for (let j = 0; j < n; j++) { + let a = 0; + for (let p = 0; p < k; p++) a = L.mul[(q.q[i * k + p] & 0xFF) * 256 + (w.q[p * n + j] & 0xFF)]; + out[i * n + j] = d.acc ? a : V.epi(a, q.s[i], w.s[j]); + } + return out; + } const x = V.quantizeRows(Xf, batch * m, k); let wq, ws; if (batch === 1) { const w = V.quantizeCols(Wf, k, n); wq = w.q; ws = w.s; } @@ -58,6 +68,42 @@ const eq = (a, b) => { for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) ret // ---- the properties --------------------------------------------------------- const PROPS = { + // NON-TRIVIALITY. Added after an external bug corpus scored the suite below + // 0/4: the zero function satisfies every relation here, because zero is + // permutation-equivariant, zero-row-preserving and batch-decomposable. Without + // this, a kernel can pass the whole suite by doing nothing at all. + async nonTriviality(K) { + const d = { m: 6, k: 32, n: 5, batch: 1 }; + const out = await K(randf(d.m * d.k), randf(d.k * d.n), d); + let nz = 0; + for (let i = 0; i < out.length; i++) if (out[i] !== 0) nz++; + if (nz < out.length / 4) return `output is ${out.length - nz}/${out.length} zeros`; + return null; + }, + // SENSITIVITY. Every element of A must be able to move its output row. Catches + // an accumulator that overwrites instead of accumulating (acc= for acc+=): + // every structural relation still holds, but only the last k contributes. + // Measured on the raw accumulator, perturbed by a sign flip that leaves the + // row absmax alone — otherwise the perturbation moves the output through the + // quantization SCALE and the property proves nothing. + async sensitivity(K) { + const d = { m: 6, k: 32, n: 5, batch: 1, acc: true }; + const A = randf(d.m * d.k), B = randf(d.k * d.n); + for (let p = 0; p < d.k; p++) A[1 * d.k + p] = 0.3; + A[1 * d.k + 0] = 1.0; // pin the absmax at p=0 + const s0 = V.quantizeRows(A, d.m, d.k).s[1]; + const base = await K(A, B, d); + for (let p = 1; p < d.k; p++) { + const A2 = Float32Array.from(A); + A2[1 * d.k + p] = -0.3; + if (V.quantizeRows(A2, d.m, d.k).s[1] !== s0) continue; // inconclusive + const o2 = await K(A2, B, d); + let moved = false; + for (let j = 0; j < d.n; j++) if (o2[1 * d.n + j] !== base[1 * d.n + j]) { moved = true; break; } + if (!moved) return `A[.,${p}] does not affect its output row`; + } + return null; + }, // a zero row of A must produce a zero row of output, whatever the scales are async zeroRow(K) { const d = { m: 6, k: 32, n: 5, batch: 1 }; @@ -130,6 +176,12 @@ const PROPS = { ok(swapBad !== null, `rowPermutation catches swapped output rows (${swapBad || "MISSED"})`); const swapZero = await PROPS.zeroRow(makeKernel("rowSwap")); console.log(` note zeroRow vs the same rowSwap bug: ${swapZero ? "caught" : "missed (properties are partial, not a proof)"}`); + // a bug from someone else's taxonomy, not mine: acc= instead of acc+=. Every + // structural relation survives it, which is why sensitivity had to exist. + const accSens = await PROPS.sensitivity(makeKernel("accOverwrite")); + const accPerm = await PROPS.rowPermutation(makeKernel("accOverwrite")); + ok(accSens !== null, `sensitivity catches acc= instead of acc+= (${accSens || "MISSED"})`); + console.log(` note rowPermutation vs that same acc= bug: ${accPerm ? "caught" : "missed — it is structure-preserving, which is the whole trap"}`); console.log(pass ? "\nMETAMORPHIC TEST PASSED" : "\nMETAMORPHIC TEST FAILED"); process.exit(pass ? 0 : 1);