Understanding Proxy Browsers: Architecture
I did a bunch of research on proxy-browsers for a few projects I worked on. Rather than sitting on it all, I figured I’d write a series of posts sharing what I learned in case it’s helpful to anyone else. This first post looks at the general architecture of proxy browsers with a performance focus.
In the original story of the Wizard of Oz, the Emerald City isn’t actually green nor made entirely of emeralds. All of that came later. In the original story, before entering the city each person had to put on a pair of glasses. These glasses, they were told, would protect them from the bright glow of all the emeralds that would surely damage their sight. These glasses were attached and never removed. You wore them while eating, while going to the bathroom, while walking outside—you wore them everywhere and all the time.
This was all a ruse. The glow of the city wouldn’t damage anybody’s sight because there was no glow. That all came from the glasses which just happened to be tinted green. Through the lens of those glasses, everything glowed. The lens through which those people viewed their world shaped their perception of it.
I’d venture to say that most developers and designers are not big fans of proxy browsers—assuming they pay attention to them at all. They don’t behave in ways a typical browser does, which leads to frustration as we see our carefully created sites fall apart for seemingly no reason at all. And frankly, most of us don’t really need to use them on a day-to-day basis. Through the lens we view the web, proxy browsers are merely troublesome relics of a time before the idea of a “smartphone” was anything other than a pipedream.
But our view of the web is not the only view of the web. People all over the world face challenges getting online—everything from the cost of data and poor connectivity to religious and political obstacles. In these environments proxy browsers are far from troublesome; they are essential.
So while most of us building for the web have never used a proxy browser (outside of the quick spot check in Opera Mini, perhaps), they remain incredibly popular globally. Opera Mini, the most popular of all proxy browsers, boasts more than 250 million users. UC, another popular proxy browser, boasts 100 million daily active users and is the most popular mobile browser in India, China and Indonesia.
These browsers perform optimizations and transcoding that can provide significant improvements. Several proxy browsers claim up to 90% data savings when compared to a typical browser. That’s the difference between a 2MB site and a 200kb site—nothing to sneeze at.
To understand how they accomplish this—and why they behave the way they do—we first need to revisit what we know about how browsers work.
Typical Browser Architecture
A typical modern browser goes through a series of steps to go from the URL you enter in your address bar to the page you ultimately see on your screen. It must:
- Resolve the DNS
- Establish TCP connection(s) to the server(s)
- Request all the resources on a page
- Construct a DOM and CSSOM
- Build a render tree
- Perform layout
- Decode images
- Paint to the screen
That’s a very simplified list and some of them can happen in parallel, but it’s a good enough representation for the purpose of highlighting how proxy browser architecture differs.
We can break these steps out into two general buckets. Steps 1-3 are all network constrained. How quickly they happen, and the cost, depends mostly on the characteristics of the network: the bandwidth, latency, cost of data, etc.
Steps 4-8 are device constrained. How quickly these steps happen depends primarily on the characteristics of the device and browser: the processor, memory, etc.
Proxy browsers intercede on behalf of the user in an attempt to reduce the impact of one, or both, of these buckets. You can broadly classify them into two categories: browsers with proxy services, and remote browsers.
Browsers with proxy services
The first category of proxy browsers are really just your plain-old, everyday browser that happens to offer a proxy service. These browsers alter the typical browser behavior only slightly, and as a result they provide the least benefit for end users as well as—usually—the least noticeable impact on the display and behavior of a web site. (While not really tied to a browser, look at Google’s search transcoding service for an example of how substantially a proxy service could alter the display of a page.)
Instead of requests being routed directly from the client to the web server, they are first routed through some intermediary layer of servers (Google’s servers, UC’s servers, Opera’s servers, etc). This intermediary layer provides the proxy service. It routes the request to the web server on behalf of the client. Upon receipt of the request, it sees if there are any optimizations it can provide (such as minification, image compression, etc) before passing back the potentially altered response to the client.
The browser-specific behavior (steps 4-8) remains the same as the typical browsers you’re used to testing on. All of the optimizations that take place focus primarily on the reducing the impact on the network (1-3).
There are many examples but at the moment of writing some of the more popular options in this category are Google’s Data Compression tool (Flywheel), UC Web’s Cloud Boost, and Opera Turbo.
Remote browsers
Remote browsers push the limits a bit more. They aggressively optimize as much as possible providing a much larger benefit for the end user, but also a lot more trouble for developers. (If that bothers you try to remember that the proxy browsers exist because users need them, not because developers do.) These are the browsers you more typically think of when hearing the term “proxy browser”. With the increase in browsers offering proxy services, I think referring to these as remote browsers can be a helpful way of distinguishing them.
Unlike their more conservative brethren, remote browsers are not content to merely make a few optimizations on the network side of things. They’ve got more ambitious goals.
When a website is requested through a remote browser, the request is routed through an intermediary server first before being forwarded on to the web server. Sounds familiar right? But here’s where remote browsers start to break away from the traditional browser model.
As that request returns to the server, instead of the intermediary server routing it back to the client, it proceeds to request all the subsequent resources needed to display the page as well. It then performs all parsing, rendering, layout and paint on the intermediary server. Finally, when all of that is taken care of, it sends back some sort of snapshot of that page to the client. This snapshot does not consist of HTML, CSS and JavaScript—it’s a proprietary format determined by whatever the browser happens to be.
That’s why calling them “remote browsers” makes so much sense. The browser as we know it is really contained on the server. The application on the phone or tablet is nothing more than a thin-client that is capable of serving up some proprietary format. It just so happens that when it serves that format up, it looks like a web page.
The most important thing to remember for remote browsers is that because all they are doing is displaying a snapshot of a page, anything that might change the display of that page requires a trip back to the server so an updated snapshot can be generated. We’ll discuss that in more detail in a later post as the implications are huge and the source of most proxy browser induced headaches.
There are many options, but Opera Mini, UC Mini and Puffin are some of the more popular.
What’s up next?
Understanding the basic architecture of proxy browsers makes testing on them so much easier and far more predictable. It’s the key to understanding all of the atypical behavior that causes so many developers to cringe whenever they have to fire up a proxy browser for testing.
With the foundation laid, we can spend the next several posts digging deeper into the specific optimizations the two categories of proxy browsers make as well as consider the implications for developers.