Steps Overview¶

This section documents all available pipeline steps in paper-scanner.

What are Steps?¶

Steps are individual processing units that can be chained together to create pipelines. Each step: - Has a specific responsibility (e.g., import, analyze, export) - Takes configuration in YAML or Python - Validates configuration before running - Processes papers from the database - Returns standardized results

Available Steps¶

Click on each step to see detailed documentation:

Step	Purpose	Status
bibtex_import	Import papers from BibTeX files	✅ Stable
ris_import	Import papers from RIS files	✅ Stable
citations	Extract forward and backward citations	✅ Stable
deduplication	Find and remove duplicate papers	✅ Stable
export	Export papers to various formats	✅ Stable
metadata_screening	Filter papers by language, type, quality	✅ Stable
keyword_screening	Keyword-based paper filtering	✅ Stable
patch	Update paper metadata	✅ Stable
retrieve_metadata	Fetch missing metadata	✅ Stable
run_template	Run analysis templates	✅ Stable
semantic_screening	ML-based paper screening	✅ Stable
summarize	Summarize papers and generate reports	✅ Stable
upload_database	Upload papers to remote database	✅ Stable

Screeners Group¶

Four specialized screening steps work together to filter papers progressively:

deduplication - Removes duplicate papers using DOI, title+author, or title matching
metadata_screening - Filters by language, paper type, and quality indicators
keyword_screening - Pattern-based study type detection with 64+ regex patterns
semantic_screening - Embedding-based relevance filtering to research question

A 5th screener (LLM-based) is planned for advanced semantic understanding.

Typical Screening Pipeline¶

Import and Process¶

steps:
  - name: bibtex_import
    file: references.bib
  - name: retrieve_metadata
    methods: [crossref, openalex]
  - name: summarize
    summary: true

Build Citation Network¶

steps:
  - name: citations
    backward:
      citations: [crossref]
      details: [openalex]
    forward:
      citations: [openalex]

Find Duplicates¶

steps:
  - name: deduplication
    strategy: doi_title_year

Screen Papers¶

steps:
  - name: semantic_screening
    model: sentence-transformers/all-MiniLM-L6-v2
    thresholds:
      include: 0.7
      exclude: 0.3

Extract Findings¶

steps:
  - name: run_template
    template: extract_findings

Step Configuration¶

Each step has: 1. Required fields - Must be present 2. Optional fields - Improve functionality 3. Defaults - Used if not specified

All steps support: - dry_run - Preview without database changes - verbose - Show detailed output - debug - Show debug information

Validation¶

Steps validate their configuration:

# Validate without running
paper-processor definition.yml --validate-only

# Run with validation
paper-processor definition.yml --verbose

Building Custom Steps¶

See Architecture: Pipeline for how to create custom steps.

Chaining Steps¶

Steps execute sequentially:

steps:
  - name: bibtex_import          # Step 1
    file: references.bib
  - name: citations               # Step 2 (runs after Step 1)
    backward:
      citations: [crossref]
  - name: export                  # Step 3 (runs after Step 2)
    format: bibtex
    output: processed.bib

Each step sees the results from previous steps and can operate on updated paper records.

Error Handling¶

Steps provide detailed error information:

steps:
  - name: citations
    backward:
      citations: [crossref]
    continue_on_not_found: true   # Don't fail on unresolved citations
    output_errors: errors.jsonl   # Log errors to file

Performance Tips¶

Batch Processing - Most steps support batch_size
Caching - Results are cached; re-running is fast
Filtering - Process only relevant papers with filters
Parallelization - Some steps support parallel workers