Skip to content

perf(bedrock): add SKILLSPECTOR_COMPACT_PROMPTS option to reduce LLM token usage - #457

Open
leogasparini wants to merge 1 commit into
NVIDIA:mainfrom
leogasparini:optimize-llm-token-usage
Open

perf(bedrock): add SKILLSPECTOR_COMPACT_PROMPTS option to reduce LLM token usage#457
leogasparini wants to merge 1 commit into
NVIDIA:mainfrom
leogasparini:optimize-llm-token-usage

Conversation

@leogasparini

Copy link
Copy Markdown

Introduce an opt-in env var (SKILLSPECTOR_COMPACT_PROMPTS=true) that reduces token consumption across all LLM analyzer prompts while preserving the default behavior for backward compatibility.

When enabled, compact mode applies five optimizations:

  1. Line numbers drop zero-padding (L001 -> L1), saving ~3 chars per line
  2. Meta-analyzer finding context omitted from prompt (already in file content)
  3. Meta-analyzer response schema drops unused OverallAssessment field
  4. All analyzer prompts condensed (redundant phrasing, verbose formatting)
  5. BASE_ANALYSIS_PROMPT output guidelines shortened from 5 lines to 2

Estimated per-scan savings with compact mode on:

  • ~100 input tokens/call from prompt condensation (x4-8 calls)
  • ~3 chars/line from line numbering (500-line file = ~375 tokens/call)
  • ~30-50 output tokens/call from OverallAssessment removal
  • Variable savings from context field omission

Default behavior is fully preserved: all original prompts, schemas, and formatting remain active unless the env var is explicitly set.

All 2955 tests pass in both default and compact modes. Lint clean.

Contributes to #456.

Introduce an opt-in env var (SKILLSPECTOR_COMPACT_PROMPTS=true) that reduces
token consumption across all LLM analyzer prompts while preserving the
default behavior for backward compatibility.

When enabled, compact mode applies five optimizations:

1. Line numbers drop zero-padding (L001 -> L1), saving ~3 chars per line
2. Meta-analyzer finding context omitted from prompt (already in file content)
3. Meta-analyzer response schema drops unused OverallAssessment field
4. All analyzer prompts condensed (redundant phrasing, verbose formatting)
5. BASE_ANALYSIS_PROMPT output guidelines shortened from 5 lines to 2

Estimated per-scan savings with compact mode on:
- ~100 input tokens/call from prompt condensation (x4-8 calls)
- ~3 chars/line from line numbering (500-line file = ~375 tokens/call)
- ~30-50 output tokens/call from OverallAssessment removal
- Variable savings from context field omission

Default behavior is fully preserved: all original prompts, schemas, and
formatting remain active unless the env var is explicitly set.

All 2955 tests pass in both default and compact modes. Lint clean.

Signed-off-by: leonardogasparini <leonardo.gasparini@tui.com>
@leogasparini
leogasparini force-pushed the optimize-llm-token-usage branch from 274189b to f575edb Compare August 28, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant