Unanchored regex mapped BCP-47 script subtags to bogus regions
(lang:zh-hans -> location=HA) and prefix-matched 3+ letter codes
(lang:eng -> language=en). Require a subtag boundary, matching the
Perplexity provider's parsing.
The shared query pipeline parses a lang:/language: directive into StructuredQuery.lang, which sibling providers (DuckDuckGo, Perplexity, SearXNG) map onto their native locale params. The TinyFish provider dropped parsed.lang entirely, so every request fell back to the API's US/English default and non-US locales were silently lost.
Map parsed.lang onto TinyFish location (ISO 3166-1 alpha-2) and language (ISO 639-1): lang:it-it yields location=IT&language=it, lang:it yields language=it only. Behaviour is unchanged when no locale directive is present.
Fixes#8913
- Expanded web search and fetching providers with robust parsing, authentication storage integration, and response validation.
- Added support for new configurations including SearXNG safesearch, Cloudflare AI Gateway endpoints, and dynamic Firecrawl base URLs.
- Implemented comprehensive test suites covering error handling, content filtering, and provider-specific response behaviors.
- Implemented a structured web-search query parsing module supporting directives, tokenization, date parsing, and syntax serialization.
- Updated search providers to map query directives and date bounds to native provider parameters and filters.
- Added lenient result constraint post-filtering and configuration settings for enhanced engine routing.
- Added comprehensive unit and integration tests covering query parsing, constraint filtering, and provider-specific request mapping.
can1357