Elasticsearch#

Full-text search, relevance ordering, highlighted snippets and facet counts.

Core: app/Core/Integrations/Elasticsearch/. Document sources: app/UI/Api/{module}/Models/Search/*DocumentSource.php. Endpoint: app/Core/Traits/Api/Presenters/SearchTrait.php. Configuration: config/Shared/elasticsearch.neon.

The founding rule#

Elasticsearch is a source of identifiers and ordering, not of data

Document bodies are never read from ES (_source: false). It returns a list of ids, optionally their score order and highlighted snippets. The data is then loaded from the database. The index therefore need not carry every field and cannot return stale content.

The database remains the arbiter of visibility

The caller's filters (active, published, paid, category…) are applied always and on both paths. A stale document can therefore never reveal an entity the user must not see — at worst it shortens a page. The pre-narrowing described below is an optimisation, not a security measure.

1. Two paths, one endpoint#

Every indexed module has an Api action api/{module}/{entity}/search. Its shape is module-neutral — SearchTrait supplies it, the module fills in three hooks.

Situation	Who orders	Who paginates	`totalCount`	`fromEs`	Snippets
search without `orderBys`	ES (score)	ES (`from`/`size`)	ES (`track_total_hits`)	`true`	yes
search with `orderBys` (manual choice)	DB	DB	DB	`false`	no
ES off / outage / breaker / query shorter than 2 characters	DB	DB	DB	`false`	no

A manual ordering choice overrides relevance — by design

If ES paginated even for "Newest", the user would get the newest only within the page of the most relevant. On that branch only the id set is taken from ES (phase one) and the database orders, paginates and counts — exactly as before ES.

Shape of the response#

Identical to getGetAllData() plus two extra keys:

[
    'items'       => [...],   // already reordered by score
    'totalCount'  => 42,
    '_references' => [...],
    'fromEs'      => true,    // ordering and totalCount come from ES
    'highlights'  => [        // entity id => logical field => safe HTML
        17 => ['name' => 'Selling a <em>drill</em>', 'text' => '…'],
    ],
]

fromEs is the only reliable signal for the front end

Configuration says nothing about ES availability — the breaker may be open, the index may not exist. So the front end asks the payload, not the config. fromEs decides whether the listing is relevance-ordered and whether any sort-box option is highlighted.

The highlights key is ALWAYS present

Even when empty. Callers therefore need not guard it by fromEs. A template that forgets it (highlights: [] when including a nested listing) fails on undefined variables.

Why no query cache here#

The cache would serve stale ordering

SearchIdFilter is a NonCacheableFilter — the key would have to contain the whole id set. On top of that the ES response changes without any database write (reindex, score changes), so the cache would have nothing to invalidate on.

2. Configuration#

parameters:
    elasticsearch:
        enabled: false                    # master switch — false = every service is a no-op
        hosts: ['http://127.0.0.1:9200']
        username: null
        password: null
        indexPrefix: cms
        enabledSources: []                # allowlist of source keys

Hosts and credentials belong in the gitignored config/Shared/local.neon.

Switch layer	Effect
`enabled: false`	no connection is opened, everything behaves as before ES
`enabledSources` without a source key	that one source runs on MySQL, the others on ES
circuit breaker open	ES is not called at all for 60 seconds

Roll out module by module

That allowlist is the whole point — you can have "ES on for the blog, classifieds still on MySQL". A new module is switched on only after a verified reindex.

Two clients#

Service	Timeout	Used for
`esClient`	short	search, monitoring — an outage must fail fast
`esIndexingClient`	60 s	indexing and bulk reindex — hundreds of documents must not die on 2.5 s

Constructing a client opens no connection

That is why it is safe to have the services in the container even with enabled: false.

3. Indices, aliases and a reindex without downtime#

A physical index is named {prefix}_{base}_v{N}, for example cms_blog_article_cs_v5. The application never touches it directly — it speaks through two aliases:

Alias	Used by	Points to
`cms_{base}` (read)	search, monitoring	the stable version
`cms_{base}_write`	indexing from save hooks	the new version during a reindex

Course of a full reindex (EsIndexManager):

beginReindex()    creates _v(N+1), switches the WRITE alias to the new version
                  └─ concurrent save hooks then write straight into the new version
   ...bulk...     fills the new version in batches of 200
finishReindex()   refresh the new version → atomic READ alias swap → drop old ones
rollbackReindex() on failure: write alias back to the stable version, discard the new one

Without refresh() before the alias swap, search sees nothing

Bulk writes carry no refresh and the default interval is one second. Were the read alias switched immediately, search and monitoring would briefly see an incomplete new version. The explicit refresh is therefore part of finishReindex().

A concrete index named like an alias blocks the whole bootstrap

It typically appears through auto-create from a write that arrived before the indices were set up. ensureIndexes() raises a readable exception with the fix (DELETE /cms_{base}), because otherwise updateAliases would fail forever.

Czech analysis#

Every full-text field carries two analyzers:

Analyzer	Filters	Purpose
`text_czech`	lowercase, Czech stopwords, Czech stemmer, asciifolding	the base field — also finds inflected forms
`text_folded`	lowercase, asciifolding	the `.folded` subfield — catches queries typed without diacritics

Changing the analysis requires a new index

The analysis of an existing index cannot be changed at runtime. A new version is built and switched to — exactly what a full reindex does.

4. The document source#

One class per entity, living in the module's Api layer (so it can reach services and managers); Core depends on it only through an interface. It is wired by hand in config/Shared/elasticsearch.neon in two places — esSearcher and esMonitor.

Method	Returns	Watch out for
`getKey()`	key for the allowlist and the searcher	must match `enabledSources`
`getIndexBases()`	name bases, one per language for translated entities
`getIndexBaseForLanguage()`	base for a language alias, `null` = unknown language	`null` means falling back to the DB path
`getIndexDefinition()`	settings + mappings	the source of truth about analysis
`getSearchFields()`	fields with boosts for `multi_match`	always the `.folded` variants too
`fetchFresh()`	fresh entities after the commit	`useCache: false`; the entity from the hook is not authoritative
`buildDocuments()`	`base => (_id => document)`	an empty array means the entity is deleted from the index
`getAllDocumentIds()`	every possible `_id` across languages	targeted deletes, never `delete_by_query`
`fetchReindexBatch()`	a batch for the full reindex
`getEntityIdField()`	name of the field holding the entity id	ordering and `search_after` during the orphan scan
`fetchIdsChangedSince()`	ids changed since the watermark	input of the incremental sync
`fetchExistingIds()`	which ids really exist in the database	orphan cleanup
`expectedDocumentBounds()`	`{min, max}` of the expected count	must mirror the `buildDocuments()` guards exactly

expectedDocumentBounds() must agree with buildDocuments()

It is the monitoring input. Once they diverge, a false drift is reported on every check — and then nobody looks at the real one.

A source that can only write leaves deleted records in the index

Ids that no longer exist in the database then show up in results. That is why deletion is part of the contract (getAllDocumentIds plus an empty buildDocuments).

The document `_id`#

One shape across all sources: "{entityId}_{languageId}". The searcher reads the entity id from it via strtok('_'), so a different shape breaks reading. For a single-language entity the insertion language is used.

Two optional extensions#

interface HighlightableDocumentSourceInterface extends DocumentSourceInterface
{
    /** logical key => ES fields in order of preference */
    public function getHighlightFields(): array;
}

interface VisibilityConstrainedDocumentSourceInterface extends DocumentSourceInterface
{
    /** clauses for bool.filter — they only narrow, they do not affect the score */
    public function getVisibilityFilters(): array;
}

Both are opt-in and backwards-neutral

A source without the interface sends a byte-identical query to the one it sent before. That is why rolling out to another module cannot show up in the others.

5. Highlighted snippets#

The security core#

EsHighlighter is the single place where an ES response becomes HTML

A raw fragment never leaves the searcher. Whoever wants to print a snippet gets a finished, safe string.

ES does not wrap matches in  but in two Private Use Area characters (U+E000 / U+E001). The conversion then follows a fixed order:

1. htmlspecialchars(THE WHOLE fragment)  → no executable markup is left in the string
2. only then sentinels → <em> / </em>

Reversing those two steps is stored XSS

The content comes from an editor, so <script> or "><img onerror=… can be in it. Inserting the tags first and escaping afterwards would escape them too — and narrowing the escaping to "the rest" is exactly the task on which XSS is made.

The resulting allowlist is therefore structural, not textual: the only tags that can appear in the output are  and  without a single attribute. The output is always balanced as well — unbalanced or nested sentinels are dropped.

Printing it in a template#

The renderer pulls the snippet into a variable up front ($nameHl, $textHl); the template then only decides whether there is something to replace the original with:

{* element body: the snippet through |noescape, otherwise the original value *}
<a class="list-item__title" href="{$link}" title="{$name}">
    {if $nameHl}{$nameHl|noescape}{else}{$name}{/if}
</a>

{* description: NO |truncate — EsHighlightSnippet already shortened it *}
<p class="list-item__desc">
    {if $textHl}{$textHl|noescape}{else}{$perex|stripHtml|truncate:150}{/if}
</p>

A snippet never belongs in an HTML attribute

The escaping is computed for an element body, not for an attribute value. title= and alt= take the plain text from the database — note that title="{$name}" above does not use the snippet.

Never run |truncate over a snippet

The shortening already happened in the right place (around the match). Truncating again in the template would cut off the very word that is highlighted.

Shortening to the listing's length#

EsHighlighter asks for number_of_fragments: 0, that is the whole field. That suits a title and a lead paragraph, but not the description of a classified ad, which easily runs to 1,200 characters. EsHighlightSnippet::around() shortens it.

A naive |truncate over finished HTML is not enough

It would cut in the middle of an  or of an entity (& → &am), it would count markup towards the limit — and above all it would cut from the start, so a match at character 400 would not be visible at all.

around() therefore picks a window around the first match and cuts by tokens: no new tag can appear,  is always closed and an entity is never split.

Narrowing to the items of the listing#

ES highlights documents the database has thrown away too

A stale document of a deactivated entity drops out of items, but without the filter its title and text would stay in highlights — a leak of text the caller must not see. SearchTrait::highlightsForItems() therefore narrows the map to the ids that really are in the listing. The condition is "the item is in the listing", not "the id is in ES".

6. Pre-narrowing by visibility#

An optional extension of a source. It was born for classified ads: by design the index also keeps expired and inactive ones, so sixty-nine documents stood against two visible ones. The relevance path paginates in ES, so pages came back shorter than the limit and totalCount was badly overstated.

public function getVisibilityFilters(): array
{
    return [
        ['term'  => ['active' => true]],
        ['range' => ['expirationDate' => ['gte' => 'now/d']]],
    ];
}

Property	Why
clauses go into `bool.filter`, not `must`	a filter does not affect the score — relevance ordering stays identical
`active` is a denormalised flag	a generated column in the database; the value is read from the DB, never recomputed in PHP
expiry is a query-time `range`	`NOW()` cannot go into a generated column, and in the index it would go stale every midnight

The definition of active DIFFERS between modules

For classifieds it includes the "sold" flag, for the company catalogue it does not. That is why the value is always read from the database through the generated column's getter and never reassembled in PHP.

Only conditions the source can propagate into the index belong here

The opposite direction (invisible in the index, already visible in the database) is the price of pre-narrowing: such an entity drops out of search until a hook or a reindex catches up. Conditions that depend on settings do not belong here at all — the query would behave one way in the CLI and another on the web.

7. The path to the front end#

The front end never talks to ES or to the database directly — it goes through the Api bridge:

Presenter trait ──► Manager::search() ──► Service ──► Mapper ──► Repository
                                                                     │
                                                        ApiBridge::internalCall
                                                                     ▼
                                                       Api presenter · SearchTrait
                                                           ├─ EsSearcher (ES)
                                                           └─ Manager::findAll (DB)

The manager returns a quadruple [entities, totalCount, fromEs, highlights].

Hydration keeps the payload order — it must not reverse it

The Api side reorders via orderItemsByIds(); the front end then hydrates in the order the items arrived in. Any interference with the order in the mapper silently discards relevance — and on a two-item result it need not even be visible.

The presenter builds three things out of it:

Template variable	Meaning
`items`	the listing in score order
`searchHighlights`	`id => field => HTML`; empty = a listing without highlighting
`sortActiveOrder`	`0` while relevance is active (no sort-box option is highlighted), otherwise the chosen ordering

A deliberate deviation: promoted items are not promoted on the relevance path

Without a search, and with a manual ordering choice, promotion works unchanged. When ordering by score, ES decides the order and the database does not interfere.

8. Permissions and routes#

Without the privilege, search is SILENTLY empty

Api RBAC runs in enforcing mode and a missing privilege is not an error — it is an empty result. Every module therefore needs a :Api:{Module}:{Entity}:search record; the scripts live in docs/sql/ (*-search-privilege.sql).

The listing route must be able to express "ordered by relevance". A sentinel in the mask does that:

o-<order=0>      the default value 0 means relevance

Nette drops the default value from the URL

A segment equal to the default is removed from the address and an incoming /o-0 is redirected (301) to the form without it. Relevance therefore lives on the bare URL; /o-0 exists only as canonicalisation. The scripts live in docs/sql/ (*-route-order-default.sql).

9. Operations#

Job	URL	Frequency	What it does
Full reindex	`/cron/{module}/elastic/reindex`	nightly	builds a new version and swaps the alias — the primary consistency guarantee
Incremental sync	`/cron/{module}/elastic/sync`	every 10 min	indexes what changed since the watermark (5 min overlap) plus orphan cleanup
Status	`/cron/system/elastic/status`	on demand	cluster health and document counts
Watch	`/cron/system/elastic/watch`	every 15 min	checks plus a summary email on findings

Writing to the index goes through the completedSave and completedDelete hooks, that is after the commit.

Bulk operations outside the hooks are invisible to the index

Scheduled jobs, payments, updateColumn and raw SQL write outside the hooks. The incremental sync will not catch them — only the nightly full reindex will. Where freshness matters, targeted reindex triggers sit in the code.

What the monitoring watches#

Check	When it is a problem
cluster availability and health	any status other than `green`
drift of the document count against the database	outside the tolerance (5 %, at least 5 documents)
age of the last successful reindex	over 26 hours

Findings are deduplicated before sending (Redis, 6 hours), so the same problem is not mailed out every quarter of an hour.

Outage#

During an outage, search returns database results, not an error

The circuit breaker (Redis, a key with a 60-second lifetime) stops calling ES after repeated failures, so nothing waits on timeouts. The caller falls through to MySQL full-text. It does mean, though, that "fewer results" can mean "the index is down" — hence the monitoring.

A 4xx response does NOT open the breaker

A missing index before bootstrap or a malformed query is not a cluster outage and must not turn search off for everyone. The breaker opens only on genuine outage errors.

Without Redis the breaker does not work

Degradation then only happens through the client's short timeouts — more slowly, and again for every request.

Limits#

Limit	Value	What happens when exceeded
the `from + size` window	10,000	a deeper page → fallback to the DB path (which reaches there)
phase-one id set	10,000	the excess is logged as an overflow
facet buckets	10,000	incomplete counts → fallback to the exact SQL aggregation
minimum query length	2 characters	ES is not called at all

Facet counts refuse imprecision, not just unavailability

When the number of matched documents disagrees with the number of visible ids, unavailability is returned and SQL takes over. Incomplete counts would lie silently, and that is worse than a slower query.

10. Adding a new source#

A document source in app/UI/Api/{Module}/Models/Search/ — take the closest existing one as the model (ArticleDocumentSource for an entity with columns, AdvertDocumentSource for one whose texts live in parameters).
Wire it in elasticsearch.neon in both places — esSearcher and esMonitor.
Api presenter: use SearchTrait and fill in the three hooks — getEsSourceKey(), getEsSearcher(), getFulltextFallbackFields().
The SQL privilege :Api:{Module}:{Entity}:search and the route mask with the sentinel.
The front-end stack: search() in the repository, mapper, service and manager; a search branch in the presenter.
Templates: print the snippet through |noescape, plain text into attributes.
Reindex, and only then add the key to enabledSources.

The fallback fields must match the indexed ones

getFulltextFallbackFields() is the MySQL variant of the same search. Once it diverges from getSearchFields(), users get different results during an ES outage — and nobody notices, because both paths "work".

For an entity whose texts live in parameters, the RENDERED value is indexed

The values are raw in the database and the replacement of links and blacklisted words happens only at render time. The source must therefore push them through the same path as the template — otherwise words the renderer hides leak into the index, and from there into the snippet.

11. Tests and verification#

Layer	Where
unit tests	`tests/Unit/Integrations/Elasticsearch/`
manual harnesses against a live ES	`tests/Manual/{Module}/f6_4_*.php`
browser click-throughs	`var/ds-baseline/tools/pages-es.json`

An ordering claim over a single item proves nothing

Over a one-item result the reversed order passes too. When the data is not enough for a proof, the harness needs a tripwire — a claim that fails as soon as more data arrives — not a silent omission.

Nor over two items with identical scores

With tied scores the order is decided by an internal tie-break, not by relevance. Print the scores before writing an ordering test and pick a query that splits the tie. For verifying the front end the right query is the opposite one: the one whose order differs from the default database ordering — only then can the ES path be told from the database path.

Every fix deserves a counter-test

Reintroducing the defect must make the harness fail. Without that you do not know whether the test tests anything.