Last month, we released an open beta of HTTP Payload & Response Collection, an exciting feature that allows you to collect payload and response data for HTTP issues. Having Noibu store payloads and responses eliminates the need for engineers to set up “log traps” to smoke out bad data as the root cause of an issue, which may expedite the debugging process.

As collecting HTTP payloads and responses may expose personally identifiable information (PII), we’ve left it up to you to identify URLs on which payload and response data collection should be allowed. So far, this feature has only been accessible through our support team, but we’re happy to announce the addition of a front-end component that allows you to configure and manage a list of allowed URLs on your own.

[http-data-collection]

Payload and response data collection is enabled by URL through the Domains module. Select a domain and open the new Data Collection tab. Then, enter allowed URLs in the field provided. URLs must respect the guidelines below:

• The proper URL to add is the HTTP request URL–the URL made to the server/API–not the webpage URL where a user experiences an error. For example, if you want to collect data from your site’s cart page, the client-facing URL may not be where HTTP payloads are sent and received. Instead, follow these steps to identify to request URL:
  1. Navigate to the page in question–in this case, your eCommerce site’s cart.
  2. Open your browser’s network request tab.
  3. Check the key HTTP requests made from this page to your servers/APIs. This will show you the URLs from which the requests originate.
  4. Add the URL(s) to your Allowed URLs list.
     
     
• The custom URL regex must start with / or http.
  
  
• If you allow a base URL, any sub URLs will also be permitted for payload and response data collection.
  
  
• You can enter a partial/relative URL, like a path or page name, to allow data collection on any URL that includes the fragment. However, relative URLs are only allowed on-domain.
  • Example: /api/products/add-to-cart
    
    
• You can insert a period and asterisk as a wildcard for a dynamic URL.
  • Example: /api/products/.*/add-to-cart
    
    
• URLs can be on or off domain if they are absolute. Absolute paths must start with http.

Once added, your list of allowed URLs will populate in the space below, where you can modify or remove them at any time. You can allow a maximum of 20 URLs per domain. However, at any time you can click the trash icon to remove a URL from the list.

You can also identify URLs to exclude or exempt, which may be useful, for example, if you want to collect data on all product pages EXCEPT those with a specific path. To exempt a URL, enter a path in the format /(?!path).

[http-urls]

Allowing or excluding URLs for HTTP data collection requires technical insight, and our technical solutions team is happy to analyze the needs of your eCommerce site and help you assemble this list. Feel feel to contact us at support@noibu.com for guidance.

Learn more about How to Allow or Exclude URLs for HTTP Data Collection.

Notes

• URL collection takes 1-2 hours to take effect.
• Any data collected is stored for 7 days.
• Payloads will only be collected when an HTTP error occurs.