Facets don't take into account rights filtering of results

To me, the most promising idea to fix this would be to filter results based on XWiki's authorization manager directly in Solr with a plugin that we deploy as part of our existing Solr plugins. I found a tutorial on custom security filtering in Solr that describes using a PostFilter to implement custom filtering based on access control lists. From my understanding, this should allow us to filter results before facet counts are computed, ensuring that facets but also pagination accurately reflects user rights. There are several challenges to solve, though:

Accessing XWiki:

• For embedded Solr, we should make sure that we access XWiki's authorization manager directly through Java to avoid any overhead
• For external Solr, we would need some HTTP (REST?) API to perform authorization checks. For this to work, we would need to
  • ensure that Solr knows the current user ID and this can't be manipulated by the user (by, e.g., passing the same value that we use for this as part of the query)
  • ensure that Solr can reach XWiki, taking clustering setups into account. The callback URL would need to be configurable in XWiki as Solr might not be able to access XWiki in the same way as the user as the Solr server shouldn't have internet access. A possibility could be to let XWiki pass the callback URL in the request to Solr, making it possible to use the same cluster node as the original request.

Performance:

A Solr query could match a million documents and to calculate accurate facet counts, we would need to check rights for all of them. From a performance point of view, this is impossible with our current authorization manager. While it can cache computed rights efficiently, the initial computation can be costly as it loads the whole document from the database. This is fine for the 10 documents that are displayed on a single page, but not for thousands or even a million total results. For this reason, I think after an initial PoC that filtering in Solr works, we need to work on making right checks more efficient. A main idea here would be to modify the security cache loader to maintain in memory a list of all rights that are defined in the wiki, loaded with efficient database queries to get all right objects on first access and then updated as documents are modified with event listeners. That way, we should be able to check the rights of a document without loading it from the database. I don't know if this would be enough, but it could be an important first step.

An option could also be to limit facets to the first x documents, where x would be configurable with a default of, e.g., 1000. This would still require the above-mentioned refactoring for good performance but it would limit the worst case performance in big wikis. It could lead to confusing results, though. I don't know if Solr provides any way to more intelligently select which documents to check, e.g., at least one document per facet and then display approximate counts for facets of which not all documents have been checked - Solr supports approximate total counts for high numbers of results but I have no idea how this works or how we could influence this.