Fix caching of robots metadata

Currently, the cached RobotsMetadata includes the isAllowed check per user agent. The cache key is the host. The isAllowed check per user agent has to be excluded from the RobotsMetadata because this information depends on the concrete URL.