Trevor B Johnson Millinery.

Flex Hat

Approaches to SaaS – ASPs' Server and Client Technology Overview

1.         Introduction

With the growing popularity of the hosted service models, more and more application providers and data center executives start considering entering the SaaS market. However, many different aspects need to be analyzed before such a key decision can be made. The business aspects for application providers and application service providers (ASPs) have been described in the whitepaper "Business as

a Service – SaaS Billing and Business Models" (http://www.veraxsystems.com/en/download/pubfiles/

verax-saas-billing-and-business-models.pdf/get). However, in order to set up a successful ASP or SaaS business, the top-level commercial decisions and growth strategy must be coupled with architecture considerations.

1.1.      Aim and scope

The aim of this document is to outline the most common technical approaches to SaaS applications and their provisioning platforms. By presenting the various solutions with their major advantages and drawbacks, the products offered by the market and side-by-side comparisons for some sample real-life applications and scenarios, its intent is to trigger analysis of the possible way to be taken in order to build a suitable on-demand application provisioning platform and enter the ASP / SaaS market.

1.2.    Intended audience

CIOs, CTOs and high level technical engineers willing to launch basic ASP / SaaS and grow it over time or become customers for these services.

1.3.      Definitions

Cloud computing – A sales model based on SaaS (and other similar models) where users pay only for usage of requested functionality. This model bases on the Internet and users are isolated from any technical details or licensing trouble.

Clustering/Failover Clustering – A hypervisor's ability to treat many physical machines as one logical virtualization server. A critical aspect of High Availability.

Dynamic Resource Management – Hypervisor's ability to automatically adjust the amount of allocated CPU/memory resources in response to dynamically changing workload of virtual machines.

Guest OS –An operating system running within a virtual machine.

High Availability –A solution ensuring certain level of operation continuity. For hypervisors, a functional ability essential to provide high level of operation continuity for virtual machines by using hardware redundancy and clustering. In virtualization practice, this term is equivalent to Failover Clustering. All legal aspects are regulated by SLA agreements between parties.

Hypervisor – A Virtual Machine Monitor, hardware or software solution enabling platform virtualization – it allows many operating systems (virtual machines) to run on the same physical server at the same time and concurrently use the same hardware.

Live Memory Management – An ability of a hypervisor to dynamically adjust the amount of memory allocated to a particular virtual machine without stopping it.

Live Migration – A hypervisor's ability to move a running virtual machine from one physical host machine to another without stopping it nor breaking any network connections between this virtual machine and remote clients.

Memory Over-Commit – A hypervisor's ability to allocate more total memory to all virtual machines than is available on a physical server.

Multi-tenancy – An architectural concept of hosted applications, where the customers share a single instance of the application with built-in data and configuration partitioning.

SaaS – Software as a Service, an on-demand software application sales and provisioning model. The applications are hosted by the provider (Application Service Provider) and users pay for remote usage.

Server consolidation – A business tendency to reduce the number of physical servers and use the platform virtualization and virtual servers instead.

SLA – Service Level Agreement – A contract detailing the service availability and reaction times in case of a failure.

Virtual machine – A virtual, simulated computer (platform virtualization) appearing to its users as a fully equipped machine with its own operating system (guest OS).

Virtualization – A set of technology solutions allowing simulation of computers and other physical devices.

2.         Provider-side architecture

The choice of the architecture to be used for providing application services is critical for the business model and as such, it must be made in-line with the commercial decisions. The tight coupling between the two aspects is not just related to the obvious capital expenditure (CAPEX) required to get going.

The initial choice of the application and platform architectures must reflect the strategic future plans of the ASP, as switching from one model to another can be very expensive and technically challenging.

The main question to be answered by each and every ASP is how to draw the separation line between the customers, their service and data in particular. The application users must be ensured security, availability and performance. They may be using various services and many kinds of licenses. Some of them may negotiate strict SLAs, while others will be satisfied by the basic levels. Efficient handling of update subscriptions, resource demands, backup policies and other unique customer requirements depends on the application and platform architecture used, so the link between the commercial offering and the solution deployed is self explanatory.

The separation line can be drawn on multiple levels – from hardware, operating system or connectivity through the application business logic to the database or file system storage level. The provider-side architectures presented below are the two most common, but contrary models. Both have their significant advantages, but also major drawbacks, which can make them unacceptable in certain circumstances or solutions. It must be therefore mentioned, that they can be used in mixed deployments, with the separation applied in a way that makes the overall application service providing the most effective technically and beneficial business-wise. The combinations will vary with each ASP, the multitude services they offer and their customer profiles.

2.1.      Multi-tenancy architecture

The simplest way to provide basic application services seems to be implementation of the separation logic within the application itself. In this approach, called multi-tenancy, a single application and database instance is shared by many customers, and their data and configuration are carefully partitioned by the application logic.

The most important feature of a multi-tenant application is the security and robustness of the data isolation mechanism. Another important requirement is to provide a high degree of customization to support the customers' needs – be it workflow configuration, user roles or "look & feel" flexibility to support branding.

The main benefit of a multi-tenancy architecture is that there is no need of setting up an expensive multi server environment platform on the ASP side and easier configuration management. However, as the customers share the hardware, storage mechanisms and the application instance, all of them may equally suffer from failures or increased load caused by a single organization.

Because of the difficulty in providing the necessary performance and organizational issues, this approach alone is usually limited only to ASPs with a low amount of non time-critical services.

2.2.      Multi-instance architecture

A multi-instance architecture represents a philosophy opposite to multi-tenancy. In this approach, also referred to as single-tenancy, every customer is assigned a dedicated instance of the application with the underlying hardware, connectivity and storage resources. In other words, every customer uses their own application instance and resources fully isolated from other customers.

As a natural consequence, this model makes it easier to ensure the required security and performance parameters. Scalability can be easily achieved and targeted by changing the amount of resources available to a particular customer instance.

On the other hand, providing applications in this model is only possible with a server farm and a suitable platform for the provisioning of new customer accounts, both of which are a significant cost to ASPs.

The key word which appears in this approach is virtualization - the only way to keep the multi server environment scalability costs at a reasonable level. It is described in the following section.

Virtualization

Virtualization has been one of IT's hot topics over the recent years. Large enterprises and technology-oriented companies consolidate their servers and re-design their approach to infrastructure management. It is a concept of dividing the resources of a computer into a number of isolated environments by applying various forms of separation, e.g. memory or storage partitioning, time sharing, emulation, etc

The early-age issues faced by virtualization, such as low hardware performance, lack of OS support or unreliable management software lacking important functionality are history now. With the mature existing technology, the benefits of virtualization are clear to the market and include:

•          Server consolidation and maximized utilization – several underutilized servers can be migrated into one machine

•          Better reliability and business continuity – easy migrations, failovers, disaster recovery, reduced maintenance downtimes

•          Flexibility – decoupling business from hardware allows better reactions to on-demand resource needs.

•          Portability – environments can be easily relocated or moved to other hardware

•          Reduced administration and hardware costs – replicating environments, lower desktop hardware needs

•          Security – sandboxes for untrusted applications, easy recovery, centralized access control, etc.

A wide range of needs, increasing numbers of platforms and applications, as well as various implementation approaches result in existence of many types of virtualization, which can be used by ASPs depending on their business model. The following are the most popular types of virtualization:

•          Platform virtualization – Virtualization of computers. In this model one big machine is used as

a host for many virtual computers used as servers for the applications. Every simulated computer is appears to users as fully equipped machine. Examples: VMware ESX Server, Citrix XenServer.

•          Operating system virtualization – In this model the virtualized part is the operating system. One OS kernel is cloned and used by many OS instances. From the user perspective such structure looks like separated yet identical machines. Examples: Linux jails, Sun Solaris containers, Microsoft Virtualization Server, FreeVPS.

•          Desktop virtualization (Virtual Desktop Infrastructure, VDI) – A virtualization of a desktop session on a remote computer. It offers users a possibility to run the applications on a remote computer and interact with them in the same manner as with locally installed versions. Multiple and independent users are supported. Examples: Microsoft Terminal Services, VMware Virtual Desktop, Citrix XenDesktop.

3.         Client architecture

While it is obvious that the provider-side architecture is critical in terms of the business strategy and vastly affects expenditure on infrastructure, application development and support and maintenance services, there is a risk that the choice of the client-side approach can be easily made without much thought. However a poorly designed server side architecture causing enormous costs to the providers may not necessarily be a huge problem for the customers, but an unsatisfactory user experience on the client side can make all the sales and customer loyalty difference, so essential for the business in the competitive market. The choice of the appropriate solution may also vastly depend on the strategy and future plans, e.g. when multiple integrated services are to be offered following just a single application in the start-up phase.

This means a right balance between a particular application's UI friendliness and the entire service providing means must be found. The key factors can be usability, responsiveness, infrastructure requirements (such as desktop computing power or network connectivity), overall performance, security policies or administrative effort required from the customer to handle the service, to name a few.

There are many ways the user interface for SaaS applications can be provided. The following subsections present a few of the most popular solutions.

3.1.      Web-based clients

One of the most obvious choices for hosted applications is providing the user interface through a web browser. This popular solution seems to be a good balance between user-friendly front-ends and using light, thin clients without much need for special installations or additional requirements.

Due to a rapid development of web-based technologies and depending on the required level of user interaction, the types of user interfaces can range from the most static "good old" simple synchronous request-response applications, through more dynamic ones with asynchronous communication features (e.g. AJAX-based search phrase suggestions, partial refreshes, etc.) to extended visual interfaces based on third party frameworks, such as Adobe Flex or Microsoft Silverlight, which guarantee a level of interaction comparable to desktop applications.

Obviously, the differences are not just limited to the user experience. The use of the various solutions may be restricted due to the available network bandwidth, browser incompatibility, platform constraints (e.g. Microsoft Silverlight or Active Server Pages) or the need to install additional frameworks or plugins in order to be able to use the application.

3.2.      Virtual desktops

A very convenient way to provide access to serviced applications is use of Virtual Desktop Infrastructure (VDI). A virtual desktop is a remote graphical access to a user session on another machine. In other words, the users are able to see and use a remote computer's desktop as a window on their desktop.

It can be configured to allow access to an entire desktop or just particular applications.

A major problem with graphic terminal access for ASPs is a requirement of setting up a server dedicated for hosting user sessions (terminal server). Such server applications require allocation of additional resources and they are usually not free. Moreover, in some cases they require purchase of Terminal Client Access Licenses (e.g. TCALs for Microsoft Terminal Services®) which can constitute an essential proportion of the general expenditure.

On the client side, the use of virtual desktops usually requires higher network capacities, as the load generated by terminal access connections is usually higher than for web based clients. On the other hand, the virtualization of the client side may be beneficial for the customers by reducing the hardware requirements of the desktop machines. This is especially vital for businesses with a large number of terminals, e.g. call centers.

When using the virtual desktop approach, no client programs are installed on the user machines. The only requirement is installation of client software providing the terminal access. An additional benefit is that the virtualization can be made seamless, as the terminal sessions can be configured in such way, that the users use the applications in dedicated windows, so they may even not be aware of working remotely.

Examples of virtual desktops include:

•          Citrix Xen Desktop.

•          Microsoft Terminal Services.

•          VNC.

3.3.      Text terminals

Some hosted applications offer access to their features through text terminals. Text terminal access usually requires minimal installation effort on the client side and causes minimum network load. This solution is usually client-platform independent and guarantees maximized productivity, but only for experienced users performing certain types of transactional activities

3.4.      Desktop clients

Some applications can be accessed by locally installed desktop client programs. They offer unique possibilities for the user interface, but the cost is a relatively high installation and configuration effort.

The main drawback is the need to repeat the installation procedure for every end-user when a client software update is required. This can be reduced with automated updates (although these may require additional development effort) or virtualization of the desktop OS (updates are installed on virtual machine images, which are then distributed to users).

3.5.      Client interfaces comparison for typical applications

The table below presents a side-by-side comparison of the client application concepts with regard to aspects such as infrastructure and administration requirements, as well as resource usage and support for local devices. Some of the results have been obtained by running tests using a sample business application use case scenario. The main advantages and drawbacks are provided as well.

Note: The test data presented below is approximate and used for comparison purposes assuming an average activity of one user displaying a typical grid report. Exact data will always depend on the type of the application, user activity and many other details.

4.         Products Overview

4.1.      Server-side. Hypervisors.

There are many different technologies which might be used to build foundation for service providers. This results in a wide selection of choices of how such infrastructure can be built for either small or enterprise IT businesses.

The virtualization infrastructure is a working horse for most of the companies. Below, a few enterprise level environments are described. It is by no means a full list of the available options, but a sample presenting the most popular solutions for business purposes. For a more comprehensive comparison of virtualization software see Wikipedia article:

http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines

VMware

VMware is one of the leading companies in virtualization business. It offers a wide range of products from home users to enterprise level companies. VMware's key product for middle and enterprise level service providers is the VMware vSphere. It is a set of solutions covering all the needs the IT provider may have. Starting from VMware ESX, a hypervisor software which runs directly on the hardware without an OS in the middle (a "bare-metal" solution), through Load Balancing and High Availability solutions (VMware Dynamic Resource Scheduler, VMware High Availability), to management consoles.

An interesting offer from VMware for IT providers entering the ASP world is a free version of VMware's ESX hypervisor engine called VMware ESXi.

VMware vSphere: http://www.vmware.com/products/vsphere/

VMware ESXi: http://www.vmware.com/products/vi/esx/esx3i.html

Citrix XENServer

XEN is a hypervisor software originally developed at the University of Cambridge as an open source project, maintained by XenSource, Inc. Since 2007 XenSource has been owned by Citrix Systems and the Citrix brand of Xen is called Citrix XenServer. Recently, Citrix has announced that all versions of XenServer will be available to public as free and open source applications. Besides the free XenServer, Citrix offers a package of high level management tools named "Citrix Essentials".

XenServer runs directly on the hardware layer without need of a host OS, but it requires a supporting OS partition (so called "Xen dom0" domain). XenServer supports mainly open guest operating systems such as Linux, NetBSD or Solaris, mostly because the virtual systems need to be modified (ported) to cooperate with Xen hypervisor. Some newer versions of XenServer running on dedicated hardware (Hardware Assisted Virtualization CPUs from Intel and AMD) can support also unported open OS and closed proprietary systems such MS Windows.

Citrix XenServer does not yet offer the High Availability and Dynamic Resource Management. However, these extensions are not initially a "must have" for most of the medium level companies.

Citrix Essentials management package also supports the Microsoft Hyper-V hypervisor.

Citrix XenServer: http://www.citrix.com/English/ps2/products/feature.asp?contentID=1686939

Citrix Essentials: http://www.citrix.com/English/ps2/products/feature.asp?contentID=1687093

Cambridge Xen: http://www.cl.cam.ac.uk/research/srg/netos/xen

Free Xen Software: http://www.xen-tools.org/software/

 

Microsoft Hyper-V

Microsoft's Hyper-V version 2.0 starts to become a serious competitor for Citrix and VMware hypervisors. The main virtualization server is, like Citrix XenServer, offered for free in one of the distributions.

It is shipped in two ways: as a part (role) of non-free Microsoft Windows Server 2008, or as a free distribution named Microsoft Hyper-V Server 2008, which is a limited distribution of Windows Server 2008 with all other roles disabled.

Hyper-V mostly supports Microsoft Windows as guest OS, but paravirtualized Red Hat Enterprise and Suse Linux Enterprise Server are also supported.

Since version 2.0, it supports Live Migration and provides High Availability solution.

 

Hyper-V™ Server 2008 R2: http://www.microsoft.com/hyper-v-server/en/us/default.aspx

Summary.

Currently there are three key players offering mature hypervisors: VMware, Citrix and Microsoft. The VMware solution offers most features, but its edge over the competition will most likely decrease in the coming years. An interesting aspect is that all these hypervisors are available for free in their basic versions. The competition zone is the administration consoles and value-added features (live migration, visual drag & drop candies, etc.).

 

It is quite hard to obtain an independent performance data comparing these products, although some lab comparison attempts has been made:

•          Lanamark: http://blog.lanamark.com/2009/03/performance-comparison-citrix-xenserver-vs-microsoft-hyper-v-vs-vmware-esx.html

•          Performance comparison: http://virtualizationreview.com/articles/2009/03/02/lab-experiment-hypervisors.aspx

 

4.2.      Client-side

 

The following section shows popular solutions for modern SaaS application clients and client access technologies. The advantages and drawbacks mentioned below are described from the end user or end admin perspective. The service provider aspects were not taken into account.

 

Adobe Flex

Abode Flex is a software development kit designed to create cross-platform applications. With Flex, the developer can easily produce rich web application interfaces based on Adobe Flash, which can be executed inside almost every web browser.

 

In order to run a client application created with Flex, the Adobe Flash plug-in for web browser must be installed on the user's machine.

 

Adobe AIR

Adobe AIR is a cross platform development environment designed to create client applications which do not use web clients, but behave more like standard desktop applications. In general, Adobe AIR wraps around the rich web contents application (e.g. Flash) and runs it as a local application.

 

An AIR application doesn't need a web browser to run, but a runtime framework must be installed locally. Such common frameworks are dedicated to specific platforms, while the business application is cross platform and covers all platforms for which the framework is available.

 

RDP – Remote Desktop Protocol

This protocol offers the possibility to run the application client on a remote computer and interact with it in the same manner as with locally installed applications. Example products build on this protocol are: Citrix XenApp, Microsoft Terminal Services.

 

In this case, no business application is installed on the user side. The user runs the client application a on remote computer inside its own private session. The access is provided by locally executed terminal client software which shows the remote computer graphic desktop inside its window. Terminal access provided by RDP uses this specially crafted protocol to control graphical remote session which is much more network effective than those protocols based on screen capture (e.g. VNC/RFB).

There are many similar products offering remote desktop access based on RDP. A quick summary is available on Wikipedia: http://en.wikipedia.org/wiki/Comparison_of_remote_desktop_software

 

RFB – Remote Framebuffer protocol

RFB is a protocol used in all VNC-type applications. Like RDP, this protocol also offers a possibility to run the application client on a remote computer and interact with it locally, but this protocol uses different approach, screen capture, which usually requires higher network and CPU capabilities than RDP.

 

Like in RDP, in this case also no business application is installed on the user's side. The users run the client application on a remote computer inside their own private session. The access is provided by locally executed VNC client software which shows the remote computer graphic desktop inside its window.

 

There are many similar products offering remote desktop access based on RFB. A quick summary is available on Wikipedia: http://en.wikipedia.org/wiki/Comparison_of_remote_desktop_software

 

5.         Performance tests

The following tables present the results of small scale tests performed by Verax Systems. A few application access methods were compared. The aim of the tests was to estimate the CPU and network loads depending on the technology used. Some of the tests covered both the client and server sides, while others were limited to the client side only. The results only cover the load caused by the client-server communication. The impact of the application itself was eliminated from the final scores.

 

Note: All performance test results included in this document are approximate. They are more like observations than tests. Their purpose is to illustrate the general performance of specific groups of software and not to promote any specific products. Such results cannot be used as

a base for business decisions.

 

Test conditions used:

•          Client side machine: MS Windows XP Pro, 2GB RAM, Intel Core2 CPU 1.66 GHz, Ethernet 100 Mbit/s card.

•          Server side machine: MS Windows XP Pro, 2GB RAM, Intel Core2 CPU 1.86 GHz, Ethernet 100 Mbit/s card.

•          Tested connection between one user and dedicated server.

 

6.         Summary

This document is an attempt to shed some light on the subject of SaaS-related technical infrastructure. The world of SaaS and virtualization is a dynamically changing one and adds more complexity to management of IT infrastructure. All numbers and product options listed in the document may not be accurate for the moment of reading. The SaaS, ASP, virtualization and all surroundings are now

a constantly changing environment.

 

Introduction of SaaS and virtualization requires new types of service monitoring tools – an additional layer for monitoring virtualization software has to be introduced. Verax Systems' Network Management System (NMS) has rich capabilities to monitor data center multi-vendor virtualization. For more information on the Verax NMS please go to http://www.veraxsystems.com/en/products/nms.

About the Author

Flex-Fit Hat $14.95 Throttle Threads Flex-Fit Hat Flex-fit embroidered Throttle Thread hats

Snow Flex Mesh Hat $24.99 Klim Snow Flex Mesh Hat Show off your snow lifestyle pride with a Snow Flex Mesh Hat from Klim Curved brim, Flexfit® hat

Enduro Flex Mesh Hat $24.99 Klim Enduro Flex Mesh Hat Show off your outdoor lifestyle pride with an Enduro Flex Mesh Hat from Klim Curved brim, adjustable fit

Athletic A-Flex Hat $28 Alpinestars Athletic A-Flex Hat Poly pro mesh 3-D front logo embroidery Silicone suede back patch

DVLP Cursive Brown Flex Hat $22.99 Put a little class in your act with this Cursive flex hat by DVLP!

Texas Longhorns Black Flex Hat $13.99 Texas Longhorns Black Flex Hat Size Small Black

UTEP Miners 1FIT Flex Hat $25.99 UTEP Miners 1FIT Flex Hat Size L/XL Blue

Maine Blackbears 1FIT Flex Hat $25.99 Maine Blackbears 1FIT Flex Hat Size S/M Navy Blue

Texas Longhorns White Flex Hat $13.99 Texas Longhorns White Flex Hat Size M/L White

Flex 45 Flexfit Hat $20.99 Fox Racing Flex 45 Flexfit Hat Simple Fox hat for simple style Embroidered 3-D logo on front and across back Medium profile with natural curve bill 98% cotton, 2% spandex

Black Label Society Logo Flex Fit Hat $26.99 BLACK LABEL SOCIETY LOGO FLEX FIT HAT

Alabama Crimson Tide Hounds Tooth Flex Hat $25.99 Alabama Crimson Tide Hounds Tooth Flex Hat

Strike A-Flex Hat $24 Alpinestars Strike A-Flex Hat Acrylic wool blend with spandex 6 panel flat bill A-Flex Pinstripe paneling with solid panels and white piping Front 3D white logo embroidery Suede silicon back crop logo patch **Closeouts are limited to stock on hand**

Pipe A-Flex Hat $28 Alpinestars Pipe A-Flex Hat "Pipe A-flex" - acrylic spandex with piping Front 3-D logo embroidery with ultra suede fill Ultra suede fabric under bill Silicone suede back patch **Closeouts are limited to stock on hand**

Monster Flex Hat $26.99 Pro Circuit Monster Flex Hat This Flex Cap Features the Logo of Team Monster Energy/Pro Circuit/Kawasaki, so you can sneak into the pits looking like part of the team. Just keep the hat pulled down and maybe carry a cool looking wrench. Fashion that rules Pro Circuit for your head Embroidered Logos Sport Brim

O'Neill Risen Black Flex Hat $26 Resurrect your look with this sleek Risen flex hat by O'Neill with tonal logo embroidery covering the front panels!

O'Neill Beyer Brown Flex Hat $29.5 Roll out in high style with this Beyer flex hat by O'Neill featuring a large offset applique logo!

Volcom Modulator White Flex Hat $26 Rock out in stand out style with the Modulator flex hat by Volcom featuring a broken up pinstripe print!

San Diego Chargers Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your San Diego Chargers team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this San Diego Chargers Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Chargers apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

Indianapolis Colts Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your Indianapolis Colts team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this Indianapolis Colts Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Colts apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

Dallas Cowboys Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your Dallas Cowboys team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this Dallas Cowboys Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Cowboys apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

Miami Dolphins Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your Miami Dolphins team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this Miami Dolphins Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Dolphins apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

Washington Redskins Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your Washington Redskins team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this Washington Redskins Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Redskins apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

Minnesota Vikings Flex Hat: Alternate Flat Brim Flex Hat $25.99 Hey fans--if you are looking for a simple yet timeless way to display your Minnesota Vikings team spirit- then try this Reebok hat on for size. Features bold team colors and flex design for desired comfort- this Minnesota Vikings Flex Hat: Alternate Flat Brim Flex Hat is the perfect addition to your growing collection of Vikings apparel. Structured flex Six panel construction with eyelets Quality construction Officially licensed

San Diego Chargers Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This San Diego Chargers Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Chargers in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

New York Jets Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This New York Jets Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Jets in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Minnesota Vikings Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Minnesota Vikings Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Vikings in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

St. Louis Rams Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This St. Louis Rams Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Rams in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Washington Redskins Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Washington Redskins Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Redskins in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Pittsburgh Steelers Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Pittsburgh Steelers Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Steelers in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Indianapolis Colts Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Indianapolis Colts Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Colts in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Philadelphia Eagles Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Philadelphia Eagles Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Eagles in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Atlanta Falcons Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Atlanta Falcons Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Falcons in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Jacksonville Jaguars Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Jacksonville Jaguars Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Jaguars in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL

Seattle Seahawks Flex Hat: 2011 Sideline Structured Flex Hat $17.59 This Seattle Seahawks Flex Hat: 2011 Sideline Structured Flex Hat gives you the cool sideline style you're looking for with the easy comfort of a Flex hat. Features raised embroidered team logo on the front so you will be representing your Seahawks in a very stylish way. Machine washable and easy to wear Flex design makes it easy and comfortable to wear Officially licensed by the NFL