{ "type": "module", "source": "doc/api/creating-generators.md", "modules": [ { "textRaw": "Creating Generators", "name": "creating_generators", "type": "module", "desc": "

This guide explains how to create new documentation generators for @doc-kit/core.

", "modules": [ { "textRaw": "Generator Concepts", "name": "generator_concepts", "type": "module", "desc": "

Generators in doc-kit transform API documentation through a pipeline. Each generator:

\n
    \n
  1. Takes input from a previous generator or raw files
    2. \n
  2. Processes the data into a different format
    3. \n
  3. Yields output for the next generator or final output
    4. \n
", "modules": [ { "textRaw": "Generator Pipeline", "name": "generator_pipeline", "type": "module", "desc": "
Raw Markdown Files\n    ↓\n  [ast] - Parse to MDAST\n    ↓\n  [metadata] - Extract structured metadata\n    ↓\n  [jsx-ast] - Convert to JSX AST\n    ↓\n  [html] - Generate HTML/CSS/JS bundles\n
\n

Each generator declares its dependency using the dependsOn field, allowing automatic pipeline construction. Each stage is documented alongside the rest of the generators.

", "displayName": "Generator Pipeline" } ], "displayName": "Generator Concepts" }, { "textRaw": "Generator Structure", "name": "generator_structure", "type": "module", "desc": "

A generator is defined as a module exporting an object conforming to the GeneratorMetadata interface.

", "displayName": "Generator Structure" }, { "textRaw": "Creating a Basic Generator", "name": "creating_a_basic_generator", "type": "module", "modules": [ { "textRaw": "Step 1: Create the Generator Files", "name": "step_1:_create_the_generator_files", "type": "module", "desc": "

Create a new directory in your project:

\n
/\n├── index.mjs         # Generator metadata (required)\n├── generate.mjs      # Generator implementation (required)\n├── constants.mjs     # Constants (optional)\n├── types.d.ts        # TypeScript types (required)\n└── utils/            # Utility functions (optional)\n    └── formatter.mjs\n
", "displayName": "Step 1: Create the Generator Files" }, { "textRaw": "Step 2: Define Types", "name": "step_2:_define_types", "type": "module", "desc": "

Create a types.d.ts file containing a Generator export. Use this when typing your generator.

\n
export type Generator = GeneratorMetadata<\n  {\n    // If your generator supports a custom configuration,\n    // define it here\n    myCustomOption: string;\n  },\n  Generate<InputToMyGenerator, Promise<OutputOfMyGenerator>>,\n  // If your generator supports parallel processing:\n  ProcessChunk<\n    InputToMyParallelProcessor,\n    OutputOfMyParallelProcessor,\n    DependenciesOfMyParallelProcessor\n  >\n>;\n
", "displayName": "Step 2: Define Types" }, { "textRaw": "Step 3: Define Generator Metadata", "name": "step_3:_define_generator_metadata", "type": "module", "desc": "

A generator module's default export is a plain object with its metadata and\nimplementation. Create it in index.mjs:

\n
import { generate } from './generate.mjs';\n\n/**\n * Generates output in MyFormat.\n *\n * @type {import('./types').Generator}\n */\nexport default {\n  name: 'my-format',\n\n  description: 'Generates documentation in MyFormat',\n\n  // This generator depends on the metadata generator. Dependencies are\n  // declared as import specifiers, so they can live in any package.\n  dependsOn: '@doc-kit/core/metadata',\n\n  defaultConfiguration: {\n    // If your generator supports a custom configuration, define the defaults here\n    myCustomOption: 'myDefaultValue',\n\n    // All generators support options in the GlobalConfiguration object\n    // To override the defaults, they can be specified here\n    ref: 'overriddenRef',\n  },\n\n  generate,\n};\n
", "displayName": "Step 3: Define Generator Metadata" }, { "textRaw": "Step 4: Implement the Generator Logic", "name": "step_4:_implement_the_generator_logic", "type": "module", "desc": "

Create the generator implementation in generate.mjs:

\n
import { writeFile } from 'node:fs/promises';\nimport { join } from 'node:path';\n\nimport getConfig from '../../utils/configuration/index.mjs';\n\n/**\n * Main generation function\n *\n * @type {import('./types').Generator['generate']}\n */\nexport async function generate(input, worker) {\n  const config = getConfig('my-format');\n\n  // Transform input to your format\n  const result = transformToMyFormat(input, config.version);\n\n  // Write to file if output directory specified\n  if (config.output) {\n    await writeFile(\n      join(config.output, 'documentation.myformat'),\n      result,\n      'utf-8'\n    );\n  }\n\n  return result;\n}\n\n/**\n * Transform metadata entries to MyFormat\n * @param {Array<MetadataEntry>} entries\n * @param {import('semver').SemVer} version\n * @returns {string}\n */\nfunction transformToMyFormat(entries, version) {\n  // Your transformation logic here\n  return entries\n    .map(entry => `${entry.api}: ${entry.heading.data.name}`)\n    .join('\\n');\n}\n
", "displayName": "Step 4: Implement the Generator Logic" }, { "textRaw": "Step 5: Make the Generator Loadable", "name": "step_5:_make_the_generator_loadable", "type": "module", "desc": "

Generators are loaded dynamically by import specifier. Anything that resolves\nto a module whose default export is a generator works as a --target:

\n
# A package (subpath) export\nnpx @doc-kit/cli generate -t @my-scope/my-package/my-format ...\n\n# A local file\nnpx @doc-kit/cli generate -t ./generators/my-format/index.mjs ...\n
\n

Built-in generators additionally get a shorthand alias in\npackages/core/src/generators/index.mjs, which maps the name users type to\nthe import specifier it resolves to:

\n
export const publicGenerators = {\n  'json-simple': '@doc-kit/core/json-simple',\n  'my-format': '@doc-kit/core/my-format', // Add this\n  // ... other generators\n};\n
\n

If the generator lives in this repository, also add a matching subpath to the\nexports map of its package's package.json.

", "displayName": "Step 5: Make the Generator Loadable" } ], "displayName": "Creating a Basic Generator" }, { "textRaw": "Parallel Processing with Workers", "name": "parallel_processing_with_workers", "type": "module", "desc": "

For generators processing large datasets, implement parallel processing using worker threads.

", "modules": [ { "textRaw": "Implementing Worker-Based Processing", "name": "implementing_worker-based_processing", "type": "module", "desc": "

First, define the generator metadata in index.mjs:

\n
import { generate, processChunk } from './generate.mjs';\n\n/**\n * @type {import('./types').Generator}\n */\nexport default {\n  name: 'parallel-generator',\n\n  description: 'Processes data in parallel',\n\n  dependsOn: '@doc-kit/core/metadata',\n\n  // Indicates this generator has a processChunk implementation\n  hasParallelProcessor: true,\n\n  generate,\n  processChunk,\n};\n
\n

Then, implement both processChunk and generate in generate.mjs:

\n
import getConfig from '../../utils/configuration/index.mjs';\n\n/**\n * Process a chunk of items in a worker thread.\n * This function runs in isolated worker threads.\n *\n * @type {import('./types').Generator['processChunk']}\n */\nexport async function processChunk(fullInput, itemIndices, deps) {\n  const results = [];\n\n  // Process only the items at specified indices\n  for (const idx of itemIndices) {\n    const item = fullInput[idx];\n    const result = await processItem(item, deps);\n    results.push(result);\n  }\n\n  return results;\n}\n\n/**\n * Main generation function that orchestrates worker threads\n *\n * @type {import('./types').Generator['generate']}\n */\nexport async function* generate(input, worker) {\n  // Configuration for this generator is based on its name\n  const config = getConfig('my-format');\n\n  // Prepare serializable dependencies\n  const deps = {\n    version: config.version,\n    // ...other config\n  };\n\n  // Stream chunks as they complete\n  for await (const chunkResult of worker.stream(input, deps)) {\n    // Process chunk result if needed\n    yield chunkResult;\n  }\n}\n
", "displayName": "Implementing Worker-Based Processing" }, { "textRaw": "Key Points for Worker Processing", "name": "key_points_for_worker_processing", "type": "module", "desc": "
    \n
  1. processChunk executes in worker threads - No access to main thread state
    2. \n
  2. Only serializable data can be passed to workers (no functions, classes, etc.)
    3. \n
  3. fullInput and itemIndices - Workers receive full input but only process specified indices
    4. \n
  4. deps must be serializable - Pass only JSON-compatible data
    5. \n
", "displayName": "Key Points for Worker Processing" }, { "textRaw": "When to Use Workers", "name": "when_to_use_workers", "type": "module", "desc": "

Use parallel processing when:

\n\n

Don't use workers when:

\n", "displayName": "When to Use Workers" } ], "displayName": "Parallel Processing with Workers" }, { "textRaw": "Streaming Results", "name": "streaming_results", "type": "module", "desc": "

Generators can yield results as they're produced using async generators.

\n

Define the generator metadata in index.mjs:

\n
import { generate, processChunk } from './generate.mjs';\n\n/**\n * @type {import('./types').Generator}\n */\nexport default {\n  name: 'streaming-generator',\n\n  description: 'Streams results as they are ready',\n\n  dependsOn: '@doc-kit/core/metadata',\n\n  hasParallelProcessor: true,\n\n  generate,\n  processChunk,\n};\n
\n

Implement the generator in generate.mjs:

\n
/**\n * Process a chunk of data\n *\n * @type {import('./types').Generator['processChunk']}\n */\nexport async function processChunk(fullInput, itemIndices, deps) {\n  // Process chunk\n  return results;\n}\n\n/**\n * Generator function that yields results incrementally\n *\n * @type {import('./types').Generator['generate']}\n */\nexport async function* generate(input, worker) {\n  // Stream results as workers complete chunks\n  for await (const chunkResult of worker.stream(input, {})) {\n    // Yield immediately - downstream can start processing\n    yield chunkResult;\n  }\n}\n
", "modules": [ { "textRaw": "Benefits of Streaming", "name": "benefits_of_streaming", "type": "module", "desc": "", "displayName": "Benefits of Streaming" }, { "textRaw": "Non-Streaming Generators", "name": "non-streaming_generators", "type": "module", "desc": "

Some generators must collect all input before processing.

\n

Generator metadata in index.mjs:

\n
import { generate } from './generate.mjs';\n\n/**\n * @type {import('./types').Generator}\n */\nexport default {\n  name: 'batch-generator',\n\n  description: 'Requires all input at once',\n\n  dependsOn: '@doc-kit/generator-react/jsx-ast',\n\n  generate,\n};\n
\n

Implementation in generate.mjs:

\n
/**\n * Non-streaming - returns Promise instead of AsyncGenerator\n *\n * @type {import('./types').Generator['generate']}\n */\nexport async function generate(input, worker) {\n  // Collect all input (if dependency is streaming, this waits for completion)\n  const allData = await collectAll(input);\n\n  // Process everything together\n  const result = processBatch(allData);\n\n  return result;\n}\n
\n

Use non-streaming when:

\n", "displayName": "Non-Streaming Generators" } ], "displayName": "Streaming Results" }, { "textRaw": "Generator Dependencies", "name": "generator_dependencies", "type": "module", "modules": [ { "textRaw": "Declaring Dependencies", "name": "declaring_dependencies", "type": "module", "desc": "

In index.mjs:

\n
import { generate } from './generate.mjs';\n\nexport default {\n  name: 'my-generator',\n\n  // This generator requires the metadata generator's output. The dependency\n  // is an import specifier, so it may point at any installed package.\n  dependsOn: '@doc-kit/core/metadata',\n\n  // ... other metadata\n\n  generate,\n};\n
\n

In generate.mjs:

\n
export async function generate(input, worker) {\n  // input contains the output from 'metadata' generator\n}\n
", "displayName": "Declaring Dependencies" } ], "displayName": "Generator Dependencies" }, { "textRaw": "File Output", "name": "file_output", "type": "module", "modules": [ { "textRaw": "Writing Output Files", "name": "writing_output_files", "type": "module", "desc": "

In generate.mjs:

\n
import { mkdir, writeFile } from 'node:fs/promises';\nimport { join } from 'node:path';\n\nimport getConfig from '../../utils/configuration/index.mjs';\n\nexport async function generate(input, worker) {\n  const config = getConfig('my-format');\n\n  if (!config.output) {\n    // Return data without writing\n    return result;\n  }\n\n  // Ensure directory exists\n  await mkdir(config.output, { recursive: true });\n\n  // Write single file\n  await writeFile(join(config.output, 'output.txt'), content, 'utf-8');\n\n  // Write multiple files\n  for (const item of items) {\n    await writeFile(\n      join(config.output, `${item.name}.txt`),\n      item.content,\n      'utf-8'\n    );\n  }\n\n  return result;\n}\n
", "displayName": "Writing Output Files" }, { "textRaw": "Copying Assets", "name": "copying_assets", "type": "module", "desc": "
import { cp } from 'node:fs/promises';\nimport { join } from 'node:path';\n\nimport getConfig from '../../utils/configuration/index.mjs';\n\nexport async function generate(input, worker) {\n  const config = getConfig('my-format');\n\n  if (config.output) {\n    // Copy asset directory\n    await cp(\n      new URL('./assets', import.meta.url),\n      join(config.output, 'assets'),\n      { recursive: true }\n    );\n  }\n\n  return result;\n}\n
", "displayName": "Copying Assets" } ], "displayName": "File Output" } ], "displayName": "Creating Generators" } ] }