Skip to main content

HTTP Data Collection: Overview

  • Updated

Contents

Introduction

Noibu includes the powerful ability to capture and view the headers, request payload, and response payload of any HTTP error that occurs on your eCommerce site. This means that when a request against any remote URL–first-party or third-party–fails, you’ll be able the exact request and response.

Once this is successfully set up, you’ll be able to see detailed HTTP error data in the Technical Details tab.

Setting Up HTTP Data Capture

HTTP data capture must be configured on a per-domain basis, and configuration works differently for HTTP error headers and HTTP error payloads:

  • HTTP Headers: HTTP header data capture takes effect domain-wide once enabled. This means that for any HTTP error on your domain, the HTTP headers sent and received are automatically captured.
  • HTTP Payloads: HTTP payload data capture can only be enabled once HTTP header capture is enabled for a domain. To minimize Personally Identifiable Information (PII) concerns, we have enabled you to specify the HTTP request URLs from which Noibu may collect HTTP payload data

For example, if you only want to capture error data for your site’s checkout API endpoint, you could enter api.yoursite.com/checkout and HTTP error payloads will only be captured when the error URL includes that endpoint URL.

The URLs you specify will include any child URLs that include that URL substring, and you can use wildcards and regular expressions to include URL slugs and exclude specific URLs within a URL pattern. See the sections below for more details.

Capturing HTTP Headers

To enable HTTP header capturing, open the Domains module, select your domain, and check the Collect HTTP Data box on the General tab.

[domain-details-general]

Once enabled, this setting allows Noibu to collect HTTP header information for HTTP errors on the domain.

Capturing HTTP Requests & Response Payloads

Once HTTP header data capturing is enabled for a domain, you can enable request URLs. When configuring HTTP error payloads capturing, keep in mind that:

  • Any HTTP payload capture URLs will take 1-2 hours to take effect
  • HTTP payload data will only be captured if an HTTP error is encountered with one of the HTTP request URLs specified
  • Captured payload data from any user session will remain available for 7 days

Step 1: Selecting HTTP Request URLs

Please note that the proper URLs are those used for HTTP requests made by your site to a server or API. This is usually NOT the URL of site page that you see in the address bar.

For example, if your site’s checkout URL is example.com/checkout but the page makes an HTTP request to api.example.com/submit-checkout-info, you must add the api.example.com/submit-checkout-info URL to capture errors.

Selecting proper URLs for HTTP data collection requires technical insight, and our technical solutions and support team is happy to analyze the needs of your eCommerce site and help you assemble this list. Feel feel to contact us at support@noibu.com for guidance.

URL Guidelines

  • The URL(s) you add can be from any first-party or third-party domain
  • URLs are case-sensitive
  • There are 2 kinds of URLs that you may add:
    • Absolute URLs (start with http) - Can be used for any HTTP request URL
    • Relative URLs (start with /) - Can only be used for HTTP requests that are on your current site domain.
      • For example, if your site domain is example.com and you want to capture any errors for HTTP requests to example.com/api/checkout, you may use either of the following as valid URLs:
        • /api/add-to-cart
        • http://www.example.com/api/checkout/add-to-cart
      • However, if your site domain is example.com, you would NOT be able to do the same if you have an HTTP request to api.example.com as api.example.com and example.com are not the same:
        • http://api.example.com/add-to-cart - This would work
        • api.example.com/add-to-cart~~ This would not work
  • You can insert a period and asterisk as a wildcard for a dynamic URL. Make sure to include the period before the asterisk!
    • Example: /products/.*/add-to-cart
  • If you allow a base URL, any sub-URLs will also be permitted for payload and response data collection. Use with caution!
    • Example: /products will include ALL of the following:
      • /products/
      • /products/.*
      • /products/anything
  • You can also identify URLs to exclude or exempt from a given path with the regular expression format /(?!exemptpath1|exemptpath2)
    • For example, if you want to collect errors on your site’s /checkout domain but don’t want to capture /checkout/savecreditcard or /checkout/deletecreditcard , you can use the following pattern to collect any error HTTP payloads with the checkout URL except those two specific URLs:
      • /checkout/(?!savecreditcard|deletecreditcard)

Step 2: Configuring your Domain with HTTP Request URLs

Once you have selected URLs for HTTP payload data capturing, your next step is to configure your domain to capture those URLs. Noibu can capture payload data for up to 20 URLs per domain.

To add a URL, open the Domains module, select your domain, and click into the Data Collection tab.

[http-data-collection]

Enter the URL in the field provided and click Add URL. Ensure the URL meets the guidelines listed in Step 1.

Viewing HTTP Data

Once HTTP header and payload collection is enabled for your domain, data will begin to populate in two areas: the Technical Details tab and in the Session Timeline.

Technical Details

In an HTTP issue's Technical Details section, you’ll be able to see a sample instance of the HTTP error’s headers, as well as its request and response payload if enabled. This is real HTTP data randomly selected from a real user request that resulted in the error.

The sample issue is random and will change from time to time. It is intended to provide a general overview of the nature of the error and allow you to quickly zero-in on a root cause.

The Technical Details tab displays HTTP data on three subtabs:

  • Headers



  • Payload



  • Response

Session Timeline Data

When HTTP data collection is enabled, Noibu will capture this data on every session where an HTTP error occurs. To view HTTP header and payload data in a Session Timeline, check the Show HTTP Events box.

[session-timeline-settings]

HTTP events will show up in the timeline as collapsed entries. Click the arrow to expand the event and view the same Header, Payload, and Response tabs seen on the Technical Details tab.

HTTP Cautions

PII

Noibu has carefully designed and developed our HTTP header and payload capture feature to minimize the risks to potentially exposing users’ personally identifiable information (PII). However, this risk is still inherent due to the nature of the data contained in HTTP requests.

To counter this risk, Noibu has taken the following steps to ensure PII data risks are minimized.

  1. Noibu does NOT collect any HTTP payload data by default. You must choose any URLs for which you wish to capture HTTP request and response payloads. Noibu is not responsible if this data contains personally identifiable information.
  2. Noibu tries to remove certain categories of PII automatically from payloads. We employ regular expression matching and other techniques to automatically mask any such fields we detect. Given the wide difference in HTTP payloads layouts and formats, this is not infallible.
  3. HTTP payload data is only kept for 7 days. It is deleted forever afterwards and will not be accessible again. This means any PII will only stay within our systems for 7 days at most.

Articles in this section