RobotsPresenter
extends Presenter
in package
`/robots.txt` generated per host.
A static file cannot name the sitemap: Sitemap: needs an absolute URL and one code
base serves many domains (the old file pointed every site at a foreign host). The
rules are fixed; the Sitemap: line is added only while the index
(www/sitemaps/sitemap.xml, built by /cron/system/sitemap/index) really exists.
Plain Nette presenter on purpose — no theme, localization or database work for a file crawlers fetch all the time.
Table of Contents
Constants
- RULES : array<string|int, mixed> = ['User-agent: *', 'Allow: /', 'Disallow: /*?do=...
- Content pages are open. Action links (signals `?do=…`: favourites, cart, rating, report), one-off flash ids, the administration and the Api are not content.
- SITEMAP_INDEX : string = 'sitemap.xml'
Properties
Methods
- __construct() : mixed
- actionDefault() : void
- sitemapUrl() : string|null
- Absolute URL of the sitemap index on the current host, null while the file does not exist.
Constants
RULES
Content pages are open. Action links (signals `?do=…`: favourites, cart, rating, report), one-off flash ids, the administration and the Api are not content.
private
array<string|int, mixed>
RULES
= ['User-agent: *', 'Allow: /', 'Disallow: /*?do=', 'Disallow: /*&do=', 'Disallow: /*?_fid=', 'Disallow: /*&_fid=', 'Disallow: /*?_theme=', 'Disallow: /*&_theme=', 'Disallow: /*?_skin=', 'Disallow: /*&_skin=', 'Disallow: /*?view=', 'Disallow: /*&view=', 'Disallow: /*?order=', 'Disallow: /*&order=', 'Disallow: /*?spec=', 'Disallow: /*&spec=', 'Disallow: /*?lspec=', 'Disallow: /*&lspec=', 'Disallow: /*?tag=', 'Disallow: /*&tag=', 'Disallow: /admin/', 'Disallow: /api/', 'Disallow: /cron/']
Neither are variants of the same page: the theme switcher (_theme/_skin, 158 links
on every demo page), list view and sorting, and e-shop filter combinations. Crawled,
they multiplied every page into tens of thousands of duplicates in Search Console and
overloaded the server (5xx). Their canonical drops these parameters as well
(CanonicalUrlLatteExtension).
SITEMAP_INDEX
private
string
SITEMAP_INDEX
= 'sitemap.xml'
Properties
$config read-only
private
Config
$config
Methods
__construct()
public
__construct(Config $config) : mixed
Parameters
- $config : Config
actionDefault()
public
actionDefault() : void
sitemapUrl()
Absolute URL of the sitemap index on the current host, null while the file does not exist.
private
sitemapUrl() : string|null