Skip to content
V
For developers
Architecture, conventions, the core and security
Core / Robots, robots.txt and sitemaps

Robots, robots.txt and sitemaps#

What a robot may read and what not, how it is told, and how a search engine gets the list of addresses. Three related things that used not to work: robots were triggering action links, robots.txt was a static file with a foreign sitemap, and the pages sitemap was always empty.

What a robot may do#

A robot may read every content page. It may not click (trigger a signal ?do=…): favourites, cart, article rating, reporting a post. Robots would call such addresses in bulk and fill the database and the sessions.

Layer What it does Where
#[Requires(ajax: true)] a writing handler refuses a bare GET (403); AJAX from JavaScript passes Rating…HandleTrait, Discussion VoteTrait (vote, report)
CrawlerDetector a signal from a client that identifies as a robot ends in 403 before anything is loaded Front\BasePresenter::startup()
rel="nofollow" 50 action links in templates carry rel="nofollow" theme templates
robots.txt forbids the action addresses Seo\Presenters\RobotsPresenter

Favourites and cart stay GET

They write only to the session, so CrawlerDetector and robots.txt guard them. It works for people with and without JavaScript. Converting them to POST/AJAX-only is a further step.

CrawlerDetector#

App\Core\Utils\Helpers\CrawlerDetector::isCrawler(?string $userAgent): bool. It recognises bot, crawl, spider, slurp, scrap, fetch, headless, preview, monitor, archiver, the names of crawlers that do not carry these markers (Facebook, Bing preview, Google-InspectionTool, ChatGPT-User, OAI-SearchBot, Perplexity, Bytespider, PetalBot, ia_archiver) and generic clients (curl, wget, python-requests, go-http-client, okhttp, libwww). An empty User-Agent is always a robot.

It is a courtesy filter, not a security boundary

It stops clients that identify themselves. Anything that writes to the database needs its own protection (permissions, #[Requires(ajax: true)], isSameSite()).

When testing signals through curl send a browser User-Agent

Otherwise you get a 403 and look for a bug in the code. curl/… and an empty header are a robot to the detector.

View counters#

A crawler does not increase the view counter of a listing or detail (adverts and company directory, discussion messages, articles): seven places in the Front presenters call CrawlerDetector::isCrawlerRequest() and skip the counter write. The click counter is unchanged. When you add a new view counter, wrap the write in the same condition.

Procedure: adding a writing signal on the front#

  1. Mark the handler #[\Nette\Application\Attributes\Requires(ajax: true)] — a bare GET is then 403 and neither robots nor prefetching trigger the link.
  2. Give the link to the signal rel="nofollow".
  3. If the action should also work without JavaScript, do not use ajax: true; rely on CrawlerDetector and robots.txt, and write only to the session.
  4. Verify the values you write to the database in the manager; an ID from the address is only compared with what belongs to the component (pattern: reporting a post, handleReport()).
  5. Try the signal from AJAX, without AJAX and with curl/… — you get ✓, 403 and 403.

robots.txt#

It is generated by App\UI\Seo\Presenters\RobotsPresenter (the route robots.txt in RouterDispatcher). It is an ordinary Nette presenter — no theme, localisation or database. The static www/robots.txt was removed: Sitemap: needs an absolute address and one code base serves several domains (the old file pointed every site to a foreign host).

The fixed rules:

User-agent: *
Allow: /
Disallow: /*?do=
Disallow: /*&do=
Disallow: /*?_fid=
Disallow: /*&_fid=
Disallow: /admin/
Disallow: /api/
Disallow: /cron/

The line Sitemap: <host>/sitemaps/sitemap.xml is added only when the index file really exists (www/sitemaps/sitemap.xml, the directory from the parameter sitemapDir). The host is taken from the current request.

Without the sitemap index robots.txt does not point to a sitemap

Until /cron/system/sitemap/index?save=1 has run on the server, there is no Sitemap: line in the file. Nothing fails, the search engine just has no link.

Sitemaps#

They are generated by cron presenters (app/UI/Cron/<Module>/Presenters/SitemapPresenter.php) through SitemapBuilderTrait. Without ?save=1 they return a JSON preview (counts), with ?save=1 they write the file to www/sitemaps/.

The URLs in the sitemap are built by SitemapBuilderTrait through the SiteBaseUrl service: when the site.baseUrl setting is filled in, it always applies, not the Host under which cron called the site. An empty setting means the address of the request (see Sending email).

Job What it does
/cron/system/sitemap/generate pages of all language variants (hreflang)
/cron/blog/sitemap/{article,news,event} and …-category the blog
/cron/bazaar/…, /cron/comcat/…, /cron/discussion/… modules
/cron/eshop/sitemap/category all e-shop categories
/cron/eshop/sitemap/product products the listing shows (ProductSetEligibilityFilters), split into files by ID ranges (size see below)
/cron/system/sitemap/index the index of all sitemap-*.xml; run it after the others

The jobs are scheduled by the server's crontab and do not run by themselves, see Scheduled jobs. Frequency: daily at night.

The directory www/sitemaps/ belongs in .gitignore and in deploy/rsync-exclude.txt

The files are created at runtime on the server. If rsync (--delete) saw them as foreign on deployment it would delete them and robots.txt would stay without a sitemap until the next night. Both are already written; verify it on a new installation.

An index from old files

The index builds links from what is currently in the directory. If it runs before the other jobs it is built from the result of the last run.

Splitting into files#

Large sets (adverts, companies, products, articles…) are written in batches by id straight into the file (KeysetBatches, SitemapBucketWriter, SitemapUrlsetWriter), not through an array in memory. One file holds 45000 / number of languages items (SitemapBucketWriter::bucketSizeFor()), because each item yields one URL per language and a sitemap file may have at most 50,000 URLs. A file is named <prefix>-<bucket>.xml, bucket = intdiv(id - 1, size); the id boundaries are fixed, so deleting an item does not move the others between files. If the next item would exceed a safeguard (50,000 URLs or 45 MiB per file), writing continues in <prefix>-<bucket>-1.xml, -2, … A change in the number of languages changes the bucket size and with it the split of ids into files. Old numbered files of the set that the new generation did not overwrite are deleted (pattern <prefix>-[0-9]*.xml, so a foreign file with the same beginning, such as the categories file, stays).

Writing is atomic through StagedFileSet:

  1. every file is written to a random .tmp next to the target (reserve(), stage()) — the same filesystem, so rename is atomic;
  2. commit() renames the temporary files to the final names only after all files were written, sets 0664 (the web server serves them statically despite the cron umask) and deletes old files by the pattern;
  3. abort() (and a failed write) deletes only its own temporary files; the old set stays.

A crawler therefore sees, for every name, either the whole old or the whole new version.

Do not delete the old set first and do not write straight to the final name

A failure of the second file would leave the site without a sitemap, and a crawler reading a file during the write would get truncated XML. Publish your own generator of files for crawlers through StagedFileSet.

Items must arrive in ascending id order

SitemapBucketWriter::add() throws on an id smaller than or equal to the previous one. The loader for KeysetBatches therefore orders by ORDER BY id and returns only id > last.

Procedure: add a sitemap for a new module#

  1. In the module's cron presenter use SitemapBuilderTrait (buildSitemapXml(), respondSitemap(), for large sets respondBucketedSitemap()).
  2. Build the link target through routing (presenter->link(…) / buildPageUrl()), not by hand. The pages sitemap used to be empty because the link target was built as Front:Front:System:Page, which did not exist; after the fix it holds about a hundred addresses.
  3. Add the job to crontab before index.
  4. Verify: without save a JSON with the count, with save=1 the file; curl …/robots.txt shows Sitemap:; the file is served as application/xml.

One canonical address#

www/.htaccess unifies addresses into one variant (301, path and query are kept):

Input Result
www.<host>/… <host>/…
/www/… (the literal folder prefix) /…; existing files are not redirected

The redirect target is built from the rule's $1, not from %{REQUEST_URI}

The site runs through the root .htaccess (^(.*)$ → /www/$1), so REQUEST_URI already carries the internal /www/. The first version therefore redirected to /www/inzerce. The duplicate path /www/… is distinguished by %{THE_REQUEST} — the original request line, which the internal rewrite does not change.

A site that should live on www deletes these two rules

The www → no www redirect is on in the default .htaccess. On a site whose canonical host is www it must be removed.

Where to look#

I want Where
robot detection app/Core/Utils/Helpers/CrawlerDetector.php
the gate in the presenter app/Core/Base/Front/BasePresenter.php → startup()
robots.txt app/UI/Seo/Presenters/RobotsPresenter.php, the route in RouterDispatcher
sitemap generation app/Core/Traits/Cron/Presenters/SitemapBuilderTrait.php
e-shop sitemaps app/UI/Cron/Eshop/Presenters/SitemapPresenter.php
splitting and atomic writing of files app/Core/Utils/Helpers/Sitemap/, app/Core/Utils/Helpers/StagedFileSet.php
the canonical host www/.htaccess

Related: CORS, rate limit and audit · Scheduled jobs · Routing

Cookie settings