XConnect, the number of interactions are increasing with 5-6 interactions per second but the server was offline | Check Submit Queue

A few days ago, I got a task where a live website was not showing the reports for the analytics data. The client was not looking into reports after go live with last agency and when they realize that it was not working they created a ticket under a new contract with our agency. The website solution was new and required an audit to understand the problem and decide the approach to choose, as this was a live environment.

We did an audit and find the root cause of the problem and it was XConnect server was not working because of the certification issue. A certification issue in XConnect is a common problem and if “AllowInvalidClientCertificates” setting is set to false (default value) it will not work properly. It is not recommended to set “AllowInvalidClientCertificates” value to true in a live environment. So we decided to resolve this certification issue on live site and in order to resolve this issue, we updated the new correct certificate on CM server first and take one of the CD server off from load balancer to apply the patch there. Everything was work fine so far and we were able to see the interactions and contact in XConnect. But we saw that the number of interactions was increasing with 5-6 interactions per second and our CD server was off from LB, it was weird and we were not able to understand the reason. I posted the same question on Sitecore slack channel (join here – https://sitecore.chat/) and got some suggestions related to monitoring, load balancer, Keep-Alive, and other stuff. We check all those things but it was not related to these.

In random, to find the cause of the issue, I wanted to try a few things on the website folder and before that, I decided to take a backup of the complete website folder. The website size was huge and the only app_data folder was about 60GB and after checking the app_data folder we were able to understand the root cause of interactions. This was not an issue but a good practice to implement in a live environment –  Submit queue

So what is Submit Queue?

It is a kind of back up for the tracker if it is unable to submit a session to XConnect or XConnect is not available for any reason. Sitecore provides configurations to enable/disable the submit queue and determines how often the submit queue service will attempt to resubmit queued data to XConnect.

How is it a good practice to configure the submit Queue?

Well in a live environment, there are lots of scenarios where some time XConnect is not available, for examples during patching of the XConnect server, during a server restart, some time configuration related changes, certification related issues (certification expire). And during this time If the tracker is unable to submit a session to XConnect, session data should be added to a submit queue.

How Submit Queue works?

• Enable/Disable Submit Queue – “Sitecore.Analytics.Tracking.config” file is used to enable/disable submit queue, there is a setting node called “Analytics.UseSubmitQueue” where we can define the true/false value to enable/disable submit queue. If the value set to true, the contact or session is queued until xConnect is available.
  file path –App_Config\Sitecore\Marketing.Tracking\Sitecore.Analytics.Tracking.config

  <setting name="Analytics.UseSubmitQueue" value="true" />

• Submit Queue Folder Path – in the same “Sitecore.Analytics.Tracking.config” file, we can determine the submit queue implementation and the submit queue folder path. By default, FileSubmitQueue implementation is used and the default submit queue folder path is under data folder (app_data).

  <submitQueue>
      <queue type="Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue, Sitecore.Analytics"
              singleInstance="true">
          <param desc="folderPath">$(dataFolder)/Submit Queue</param>
      </queue>
  </submitQueue>

• Background Service Configuration – “Sitecore.Analytics.Tracking.Database.config” file is used to define the interval, how often the submit queue service will try to re-submit data to XConnect.
  file path–App_Config\Sitecore\Marketing.Tracking\Sitecore.Analytics.Tracking.Database.config
  

  <submitQueue>
    <backgroundService type="Sitecore.Analytics.SubmitQueueService, Sitecore.Analytics">
      <!-- Service wakeup interval in seconds. -->
      <Interval>60</Interval>
    </backgroundService>
  </submitQueue>

reference – https://doc.sitecore.com/developers/90/sitecore-experience-platform/en/submit-queue.html