Debugging Tricky HTTP Problems with Fiddler and Charles

Almost every application these days communicates over HTTP: websites, RESTful services, and even SOAP APIs all make use of Hypertext Transfer Protocol. For the most part, we don’t worry too much about what is happening at the network level when we’re building these applications.

However, from time to time a problem will show up that needs us to drop to the level of the network to figure out what is going on. This is the story of one such situation and how using Fiddler or Charles could make your life much easier.

I was working with a relatively simple RESTful service which was actually hosted on the local machine. It was firewalled and I was interacting with it by pulling messages from Azure Service Bus, transforming them, and passing them to the local API. Everything was going well from Postman, the fantastic API test tool, but from the C# code I kept getting errors about being unauthorized. This API made use of Basic HTTP authentication, which passes a token in the HTTP headers.

Postman showing an HTTP request against localhost.

I had already logged the HTTP headers I was sending in my C# code and they all looked correct. Why on earth was this not working? To debug this issue I needed to get into the raw packets being sent back and forth between the service and the API.

In general proxies can be used for all sorts of things:

• caching data coming from external websites to reduce the amount of traffic going over the external interface.
• caching data coming from your web server onto cheaper machines that don’t need to hit a database or render a complex page.
• streamline the requests you make on your cell phone to reduce the bandwidth.

In this case, we want to simply log out all the information transmitted between the service and the API.

Fiddler vs Charles

There are a number of good options that can fill the role of a logging proxy. Today we’ll look at Fiddler and Charles. Both these tools have been around for many years and are highly mature. Fiddler was recently (well if 2012 is still recent) bought by Telerik who now maintain it, but it was originally released in 2003. Even before Fiddler, the Charles proxy existed having been released in 2002. In what is almost a microcosm of the development environment back in the early 2000s, Fiddler was written in .NET and Charles in Java. After all these years both tools have picked up a pretty impressive suite of capabilities.

Let’s see how they can be used to solve my confusing HTTP API authentication problem, starting with Fiddler.

Fiddler

The installation package can be obtained from http://www.telerik.com/fiddler, and it’s relatively small and easy to install. For many years Fiddler was a standalone product, but fairly recently it was purchased by Telerik. They left the application free but have started development on a spin-off tool called FiddlerCore that will be the foundation of Fiddler going forward. You are also able to purchase FiddlerCore for use in your own applications should they need to capture traffic (You can read more about the future of Fiddler at https://www.telerik.com/blogs/fiddlercore-for-net-standard-and-fiddler-orchestra-the-future-of-fiddler). I think there could be some very interesting applications built on top of FiddlerCore, but we’ll leave them for another article.

After Fiddler was installed I ran some requests through it and was immediately reminded that the API was using HTTPS. Putting a debugging proxy between a client and an HTTPS endpoint won’t work because HTTPS cannot be decrypted by the proxy. Fortunately, Fiddler offers a workaround where it will insert a new certificate in the SSL keychain and use that to fake a certificate for the endpoint.

The HTTPS intercept options dialog. Selecting Decrypt HTTPS traffic will insert a new certificate in the certificate chain

With the SSL certificate in place, network traffic can now be intercepted. Here we can see the basic authentication token being passed to the server correctly. You can also see that the server responded with a 301 reply. That’s quite interesting because 301 indicates that the resource we’re looking for has moved and that the client should try the new location.

Fiddler showing the token being passed across the network connection

The URL in the original request is /api/v1/companies/PTPTest/inventory/items, and the one to which the 301 redirects is /api/v1/companies/PTPTest/inventory/items/. The difference is subtle,  but the original URL is missing a trailing slash.

This highlights one of the differences between how Postman handles 301 redirects, and how they are handled by the HTTP client in .NET and specifically regarding the Authorize header. Postman preserves the header and sends it on the subsequent request. The HTTP client’s behavior is to strip the token on the next request. It isn’t clear from the HTTP spec what the correct behavior is here, but it seems that it is common to strip the Authorize header, possibly for security reasons or server compatibility. This explains why Postman worked fine, whereas the HTTP client failed with a permission error. Altering the URL on the .NET side to have the terminating slash (/) fixes everything.

Locating the error in Fiddler was fairly easy once the SSL Proxy was setup. The list of requests on the left-hand side and details on the right-hand side is pretty easy to understand. The interface does feel quite busy with multiple levels of tabs and then a great deal of wasted space. It is a user interface which was clearly created by a programmer.

Let’s try doing the same analysis using Charles.

Charles

Charles can be downloaded from their website. The Charles installation is painless and you can be up and running quickly. The one drawback to Charles is that it is not free. Licenses cost about $50 with some discounts available for purchasing multiple licenses. In my mind, $50 is a pretty reasonable amount to pay for a tool such as this.

Setting up Charles to handle HTTPS traffic is also quite easy. The setting can be found under the tools menu and is called SSL Proxying Settings (Proxying, incidentally, is a fantastic Scrabble word).

The options dialog for SSL interception in Charles

Once again I was able to identify the token was being correctly passed to the 301 and then being stripped when the request was redirected. The layout of the screen was such that it is immediately obvious what the redirect did.

Charles showing the authorization header being passed as part of the request. It is clear what the redirect did by examining the next request below it in the top panel.

The user interface in Charles is dramatically cleaner than Fiddler. The tab layout is nice and the vertical layout actually flows better than the left-to-right layout of Fiddler. It was easy to find the authorization token and the multiple requests to the server.

One of the useful features that Charles provides is the ability to limit bandwidth. This allows you to simulate how your site would perform on a variety of slower connection speeds.

Limiting the bandwidth to a 56.6k modem. Unfortunately, Charles does not include making modem connection sounds.

This can be done in Fiddler too, but it must be configured from the script editor which is far less friendly. Those with old eyes, like me, will notice the lack of high DPI support in the right pane of the editor too.

Limiting the bandwidth to modem speeds

At an initial glance, it might seem that to have a cleaner interface we might be losing out on a great deal of functionality in Charles. That may be the case but I could not find any common action that I couldn’t do in Charles but could in Fiddler. Any advanced options in Fiddler were not missed, but my scenario was also quite simple.

[adinserter block=”33″]

Conclusion

Fiddler provides some very advanced capabilities in its script editor. One can go about creating plugins to exercise quite complex and esoteric scenarios. There is even a site that lists a number of really nifty looking plugins for Fiddler. My favorite is the one which will rotate all the images that pass through the proxy – perfect for an April Fools joke.  Some of the plugins perform really advanced analysis of a site doing things like input fuzzing and generating content security policies. These extensibility points are simply missing from Charles.

For many years I’ve been a user of Fiddler – it has been my default go to. Both tools support HTTP2 so there is no fear that your slick new site won’t work in either tool. There is no question that Fiddler supports some much more advanced scenarios than Charles, but they are scenarios that I’ve infrequently found a need for in my career. With this in mind and knowing how much nicer the user experience is in Charles, I think I’ll probably switch my default tool to Charles. I’m certainly not going to uninstall Fiddler because it still has its place, but I just think that its place might be in a dusty corner into which I only reach when faced with the most unusual of problems.

Looking to improve your code while writing it? Download Stackify Prefix, the only dynamic code profiler for .NET for free.

Author

Simon

More articles by Simon