HTTP request header names case sensitivity problem

Created by: b1ngz

Hi, I found a problem about the case sensitivity of request header name.

If i provide a request header, for example accept: application/json, the key is accept which is lowercase and it's different from Accept as a Hash key.

I found the default value of Accept header here which is text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

After merge method, the headers would be:

...
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'accept' => 'application/json'
...

This will cause the accept header become Accept: ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "application/json"] .

The raw request from debug log would be something like this:

GET /header HTTP/1.1
Host: 192.168.1.233:18003
Accept-Encoding: gzip, deflate
User-Agent: Arachni/v2.0dev
Accept: ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "application/json"]
Accept-Language: ["en-US,en;q=0.8,he;q=0.6", "zh-cn"]
X-Arachni-Scan-Seed: f7462ff05cc9012d93abd5163e180946
Content-Type: application/x-www-form-urlencoded

In some of my tests, the incorrect headers would cause server return HTTP/1.1 406 Not Acceptable. In that case , the scan will "fail".

Here is how to reproduce this.

Add some code to print request. There i modify the csrf check

class Arachni::Checks::CSRF < Arachni::Check::Base

    def run
        print_status 'Looking for CSRF candidates...'
        print_status 'Simulating logged-out user.'
        # request page without cookies, simulating a logged-out user
        http.get( page.url, cookies: {}, no_cookie_jar: true ) do |res|
            # print request
            print_status("#{res.request}")
            print_status("#{res}")
       ....

Set up a target for scan.

require 'sinatra'

set :bind, '0.0.0.0'
set :port, 18003

post '/header' do
  content_type :text
  "test"
end

Create a scan with following config:

{
  "audit": {
    "forms": true,
    "headers": false,
    "cookies": false,
    "jsons": true,
    "ui_forms": true,
    "links": true,
    "xmls": true,
    "ui_inputs": true
  },
  "http": {
    "request_headers": {
	  "accept": "application/json",
      "content-type": "application/x-www-form-urlencoded",
      "accept-language": "zh-cn"
    }
  },
  "url": "http://192.168.1.233:18003/header",
  "plugins": {
    "vector_feed": {
      "yaml_string": "method: POST\naction: 'http://192.168.1.233:18003/header'\ninputs:\n  k1: 'v1'\ntype: form\n"
    }
  },
  "input": {
    "without_defaults": true,
    "default_values": {}
  },
  "scope": {
    "page_limit": 0
  },
  "checks": [
    "csrf"
  ]
}

You have to replace the ip with yours.

You will see the incorrect request header in debug log like:

...
[Tue Nov  1 21:49:49 2016] [status]  [HTTP: 200] http://192.168.1.233:18003/header
[Tue Nov  1 21:49:49 2016] [info]  DOM depth: 0 (Limit: 5)
[Tue Nov  1 21:49:49 2016] [status]  Looking for CSRF candidates...
[Tue Nov  1 21:49:49 2016] [status]  Simulating logged-out user.
[Tue Nov  1 21:49:49 2016] [status]  Harvesting HTTP responses...
[Tue Nov  1 21:49:49 2016] [info]  Depending on server responsiveness and network conditions this may take a while.
[Tue Nov  1 21:49:49 2016] [status]  GET /header HTTP/1.1
Host: 192.168.1.233:18003
Accept-Encoding: gzip, deflate
User-Agent: Arachni/v2.0dev
Accept: ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "application/json"]
Accept-Language: ["en-US,en;q=0.8,he;q=0.6", "zh-cn"]
X-Arachni-Scan-Seed: f7462ff05cc9012d93abd5163e180946
Content-Type: application/x-www-form-urlencoded


[Tue Nov  1 21:49:49 2016] [status]  HTTP/1.1 404 Not Found
Content-Type: text/html;charset=utf-8
X-Cascade: pass
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Length: 473
....