Building VoiceXML Applications

Now that you have a solid understanding of the VoiceXML architecture, it is time to look more closely at the VoiceXML language itself. If you have experience with other XML-based markup languages, such as WML or XHTML, or even other tag-based languages, such as HTML or HDML, creating VoiceXML applications will not pose much difficulty for you. The main adjustment will be to the development of the user interface. Rather than sending content to a screen, it is spoken over a telephone. For this reason, it is very important to create clean, intuitive applications so users do not become frustrated and hang up. VoiceXML includes elements to help with this, as we will discuss later in this section.

Language Concepts
Before examining some VoiceXML code, let's take a quick look at the basic concepts behind a VoiceXML application.

Session
Once a user connects to the VoiceXML gateway, a session is started. This session is maintained as new VoiceXML documents are loaded and unloaded. The session ends only when requested by the user, the VoiceXML document, or the voice gateway. Each platform will have default session characteristics, many of which can be controlled by VoiceXML logic.

Dialogs
VoiceXML applications are constructed of one or many dialogs. Each dialog represents some form of conversational state with the user. After completing one dialog, you move on to another dialog, until the application is complete. There are two types of dialogs: forms and menus. A form collects user input, and a menu gives the user options to choose from. If at any point no dialog is specified, the VoiceXML application terminates automatically.

VoiceXML also has subdialogs. These are very similar to function calls, allowing the application to call out to a new dialog, then return to the original form. All of the variables, grammar, and state information is available upon returning to the calling document. Subdialogs can be used to create a set of components that may be used from several applications.

Applications
A VoiceXML application is a set of documents that share the same root document. This root document is automatically loaded any time a user interacts with any document in the application, and remains loaded until the user transitions to a document outside of the application. While it is loaded, the root document's variables are available to the other documents. It is also possible to specify grammars to be active for the duration of the application.

Grammars
Grammars make it possible to specify valid inputs from the user. Each dialog will contain at least one speech and/or DTMF grammar. In simple applications, only the dialog's grammars are active for that dialog. In the more complex, mixed-initiative applications, it is possible to have active grammars outside of the dialog being executed. Mixed-initiative refers to applications in which both the user and the gateway determine what will happen next.

Grammar creation is a very important aspect of designing intuitive, robust VoiceXML applications. It is essential to create grammars that accurately reflect the typical speech inputs from a user. If this is not achieved, either the prompts or the grammar will have to be changed. In some cases, the built-in grammars may be sufficient. VoiceXML 2.0 has the following built-in grammars: boolean, date, digits, currency, number, and time. In addition, many of the voice gateway vendors have proprietary grammars that you can use.

Events
When the normal execution of an application is interrupted, an event is thrown. In most cases, events are used when the user fails to respond to a prompt, or when the response is not suitable. They are also used when a user requests help or wants to exit the application. When an event is triggered, the element allows you to specify what the reaction should be. If there is no handler at the dialog level, the event can be caught at a higher level, since events follow an inheritance model.

Links
Links enable you to create mixed-initiative applications. They specify a grammar that is active when the user is within scope of the link. When the input matches the grammar, the user is redirected to the specified destination URI.

Scripting
If you require additional control over an application, which is not provided by standard VoiceXML elements, you can use scripting in the form of ECMAScript. This allows you to do such things as collect values of several fields in a single response.

VoiceXML

Voice eXtensible Markup Language (VoiceXML) is an XML-based language for creating voice-enabled applications. It provides a standard way for developers to extend enterprise data and Web content to a new medium. Just as HTML describes the visual interface for a Web browser, VoiceXML describes the voice interface for a voice browser, allowing for audio input and output. VoiceXML leverages the Internet for development and delivery, making it easy for developers to add voice integration to existing systems.

The major goal of VoiceXML is to bring the advantages of Web-based development and content delivery to IVR systems. To do this, VoiceXML brings together many technologies, including speech recognition, keypad input, synthesized speech, digitized audio, and audio recordings. The result is an efficient and robust way to create userfriendly voice applications.

History of VoiceXML
Even though VoiceXML is a relatively new technology, it already has quite a history. In 1995, a group of researchers at AT&T Research were working to discover ways to use the Internet for telephony applications. The goal was to devise a system that could deliver Web content and services to ordinary phones. Over the next several years, the research continued, although in separate projects, in separate companies. AT&T, Lucent, and Motorola were all working on essentially the same thing: voice Internet.

By 1999, AT&T and Lucent each had its own version of the Phone Markup Language (PML), and Motorola had developed a technology called VoxML. At the same time, IBM was working on a similar technology called SpeechML. It quickly became clear that a joint solution was required if the voice Web market was going to succeed; consequently, these companies created an organization called the VoiceXML Forum. Its members used the best features of each proprietary technology (with the majority of the syntax coming for Motorola's VoxML), along with some additions, to create the first version of VoiceXML, 0.9.

After VoiceXML 0.9 was published, the growing community of VoiceXML Forum members made huge improvements to the language, resulting in the release of VoiceXML 1.0 in March 2000. This release was well received, leading to several VoiceXML 1.0-compliant product offerings. With this initial success, the VoiceXML Forum submitted VoiceXML 1.0 to the W3C for consideration. With the future of the language in its hands, the W3C's Voice Browser Working Group put together version 2.0 of VoiceXML, which is now an official W3C recommendation. Even with all of the improvements to version 2.0, it is still very similar to version 1.0, making application upgrades trivial.

The VoiceXML Forum continues to flourish. Along with the founding members—AT&T, Lucent, Motorola, and IBM—there are close to 70 promoter members and almost 400 supporters of the technology! This is quite an accomplishment in just over three years. With such broad industry support, VoiceXML is destined to change the face of voice application development. It will not be long before we see a variety of consumer-oriented voice portals, as well as corporate applications, taking advantage of this flexible language.

Design Goals
The developers of VoiceXML had several goals, many centered on how VoiceXML relates to Internet architecture and development. Here are some of the top areas where VoiceXML benefits from the Internet:

VoiceXML is an XML-based language. This allows it to take advantage of the powerful Web development tools on the market. It also allows developers to use existing skill sets to build voice-based applications.

VoiceXML applications are easy to deploy. Unlike many of the proprietary Interactive Voice Response (IVR) systems, VoiceXML servers can be placed anywhere on the Internet, taking advantage of common Internet server-side technologies.

The server logic and presentation logic can be cleanly separated. This allows VoiceXML applications to take advantage of existing business logic and enterprise integration. Using a common back end allows the development of different forms of presentation logic based on the requesting device.

VoiceXML applications are platform-independent. Developers do not have to worry about making VoiceXML applications work on multiple browsers over multiple networks. When developing VoiceXML applications, the only concern is making sure it works with the VoiceXML browser being used. This leads to quicker development and less maintenance when compared to wireless Internet and desktop Web applications.

VoiceXML Architecture
VoiceXML uses an architecture similar to that of Internet applications. The main difference is the requirement for a VoiceXML gateway. Rather than having a Web browser on the mobile device, VoiceXML applications use a voice browser on the voice gateway. The voice browser interprets VoiceXML and then sends the output to the client on a telephone, eliminating the need for any software on the client device. Being based on Internet technology makes voice application development much more approachable than previous voice systems.

Figure shows the architecture of a VoiceXML system and an Internet application. Showing both on the same diagram clearly shows the similarity between the two solutions. As you can see, the VoiceXML application does have some additional complexity when compared to the Internet application. Instead of using the standard request/response mechanism used in Internet applications, the VoiceXML application goes through additional steps on the voice gateway. Let's go through the steps of a sample voice interaction.


Figure: VoiceXML architecture.

Just as Internet users enter a URL to access an application, VoiceXML users dial a telephone number. Once connected, the public switched telephone network (PSTN) or cellular network communicates with the voice gateway. The gateway then forwards the request over HTTP to a Web server that can service the request. On the server, standard server-side technologies such as JSP, ASP, or CGI can be used to generate the VoiceXML content, which is then returned to the voice gateway. On the gateway, a voice browser interprets the VoiceXML code using a voice browser. The content is then spoken to the user over the telephone using prerecorded audio files or digitized speech. If user input is required at any point during the application cycle, it can be entered via either speech or tone input using Dual-Tone Multifrequency (DTMF). This entire process will occur many times during the use of a typical application.

As just stated, the main difference between Internet applications and VoiceXML applications is the use of a voice gateway. It is at this location where the voice browser resides, incorporating many important voice technologies, including Automatic Speech Recognition (ASR), telephony dialog control, DTMF, text-to-speech (TTS) and prerecorded audio playback. According to the VoiceXML 2.0 specification, a VoiceXML platform must support the following functions in order to be complete:

Document acquisition. The voice gateway is responsible for acquiring VoiceXML documents for use within the voice browser. This can be accomplished within the context of another VoiceXML document or by external events, such as receiving a phone call. When issuing an HTTP request to a Web server, the gateway has to identify itself using the User-Agent variable in the HTTP header, providing both the browser name and the version number: "/".

Audio output. Two forms of audio output must be supported: text-to-speech (TTS) and prerecorded audio files. Text-to-speech has to be generated on the fly, based on the VoiceXML content. The resulting digitized speech often sounds robotic, making it difficult to comprehend. This is where prerecorded audio files come into play. Application developers can prerecord the application output to make the voice sound more natural. The audio files can reside on a Web server and be referred to by a universal resource identifier (URI).

Audio input. The voice gateway has to be able to recognize both character and spoken input. The most common form of character input is DTMF. This input type is best suited for entering passwords, such as PIN numbers, or responses to choice menus. Unfortunately, DTMF input is quite limited. It does not work well for entering data that is not numeric, leading to the requirement for speech recognition.

Transfer. The voice gateway has to be capable of making a connection to a third party. This most commonly happens through a communications network, such as the telephone network.

When it comes to speech recognition, the Automatic Speech Recognition (ASR) engine has to be capable of recognizing input dynamically. A user will often speak commands into the telephone, which have to be recognized and acted upon. The set of suitable inputs is called a grammar. This set of data can either be directly incorporated to the VoiceXML document or referenced to an external location by a URI. It is important to provide "tight" grammars so the speech recognition engine can provide accurate speech recognition in noisy environments, such as over a cell phone.

In addition to speech recognition, the gateway also has to be able to record audio input. This capability is useful for applications that require open dictation, such as notes associated with a completed work order.

Why Voice?

Before looking at the specifics of VoiceXML technology, it's a good idea to first consider the reasons why voice applications are being deployed. Some of the reasons are due to limitations of other application types, while others result from core benefits that voice solutions provide. Let's start with one of the most compelling reasons, accessibility.

There are more than 1 billion telephones in the world. This far exceeds the number of computers with Internet access and dwarfs the number of Internet-enabled mobile devices. Voice applications allow each of those telephones to act as an interface to the enterprise. In addition to universal accessibility, the medium of interaction is also desirable. Voice is the most efficient and preferred user interface for many applications, especially where a visual interface is impractical, such as when driving an automobile.

Cost savings is another factor. Since the users of the application already have access to telephones, there is no additional cost associated with purchasing and maintaining devices. Voice applications are device-independent as well, meaning you do not have to be concerned with developing multiple markup languages for multiple wireless network protocols for multiple devices. Finally, training costs are minimal; users will not have to be trained to use a new device with an unfamiliar user interface.

Voice applications benefit from the characteristics of the telephone, especially mobile phones. Mobile phones are small, inexpensive, easy to operate, and have a long battery life. This makes the phone an ideal platform for portable applications. As capabilities of mobile phones evolve, such as with the addition of location-based services, voice applications will quickly be able to take advantage of this new functionality.

The limitations of current user interfaces also provide reasons for developing voice applications. While WAP is a technology with lots of potential, many early users of WAP-based applications have been frustrated because the display screens are small, and inputting data using a keypad is slow and cumbersome. For devices with more advanced capabilities, such as wireless PDAs, the support for graphics and other rich content types are important features. The ideal solution will involve multimodal access to data. For example, to get weather information, you might speak the city name into the device to have the weather information for that city displayed on-screen of the device.

Voice does not have to be the only means of data access. In most cases, voice is used in conjunction with other application types. You might, for example, already have a smart client application for which you want to provide an easy way to access real-time data without requiring a wireless connection to synchronize the data. Offering a voice solution will provide an effective way to extend your enterprise applications to both existing and new users alike.

Finally, we consider the area where voice applications are used most often. Call centers have been using Interactive Voice Response (IVR) systems for some time. When handling large volumes of calls from people seeking similar sets of information, a voice system can be very effective. An IVR system allows a company to provide better service while reducing costs. Rather than providing limited hours of availability, with long hold times, voice systems can allow for immediate access to data 24 hours a day. In most cases, users can find the information they need without requiring human interaction. Unfortunately, user satisfaction with traditional IVR systems has been low. Rigid menus and slow response times often cause users to hang up or to press 0 to get a human attendant, rather than use the IVR system to its full benefit. This is just one of the many application areas that can be improved by a more flexible voice solution in VoiceXML.

Voice Applications with VoiceXML

Overview

When building mobile and wireless applications, most corporations do not give adequate consideration to the benefits that a voice interface provides. Voice applications make it possible to extend enterprise data to anyone with a telephone, wireless or otherwise. Unlike the proprietary, inflexible systems of the past, the new wave of voice applications provides an open, robust solution for a variety of business and consumer applications.

The following posts discusse building voice applications using the Voice eXtensible Markup Language, VoiceXML. Unlike other applications described in this blog, VoiceXML provides a voice, rather than visual, interface to enterprise systems. The VoiceXML architecture is very similar to that of Internet applications, with the Web browser replaced by a voice browser, and the handheld device replaced with a telephone. Voice interfaces give true universal access to your applications.

After reviewing the history of VoiceXML, we will examine the VoiceXML architecture and then provide information on building VoiceXML applications. The goal is to give you enough information so that you can make educated decisions as to whether a voice interface is suitable for your enterprise applications. An overview of the VoiceXML language is also given, to enable you to get started building custom solutions.

SyncML Overview

The way data is synchronized varies dramatically from one vendor to the next. Each implementation has its own proprietary way of exchanging data, sometimes making it difficult for device manufacturers, service providers, and application developers to integrate these solutions. Furthermore, each synchronization solution has support for different protocols, devices, and data types, making the task of data integration an even greater challenge. This is particularly the case when you need to synchronize applications from multiple vendors.

In February 2000, a group of companies, including Ericsson, IBM, Lotus, Motorola, Nokia, Palm, Psion, and Starfish Software, joined forces to create a new industry initiative to develop and promote a single, common data synchronization protocol that could be used throughout the industry. This protocol is called SyncML.

To date, the objectives of SyncML have not yet been realized for enterprise data synchronization. The specification is still relatively new and not many vendors have implemented SyncML-compliant products. This is largely because SyncML is not yet proven for corporate applications. Many of the existing synchronization solutions have proven market success, so there is no need for these vendors to rush to implement new SyncML solutions. The one exception is in the PIM space, the first market targeted by SyncML.

What Is SyncML?

SyncML is a specification for a common data synchronization framework and XML-based format, or representation protocol, for synchronizing data on networked devices. SyncML is designed for use between mobile devices that are intermittently connected to the network and network services that are continuously available on the network. SyncML can also be used for peer-to-peer data synchronization. SyncML is specifically designed to handle the case where the network services and the mobile device store the data they are synchronizing in different formats or use different software systems.

As the definition reveals, SyncML is an XML-based specification for synchronizing all forms of data between remote devices and enterprise systems. Its goal is to provide an industry-standard protocol for synchronizing data between all types of devices, such as PDAs, handheld PCs, smart phones, pagers, and cell phones, over any network protocols, to any data store. To achieve this lofty goal, the SyncML initiative decided to start with personal information management (PIM) synchronization, and then move to other forms of enterprise data.

Why Use SyncML?
The overall benefit of SyncML is that it is an industry initiative aimed at bringing together synchronization technology for many devices, networks, and data sources. Many of the current synchronization solutions use proprietary conduits that are often complex and limited in the data sources they can communicate with. As the number of mobile computing devices increases, and the expectations for access to corporate data grows, the SyncML Initiative hopes to provide enough benefits to make it the leading protocol for data synchronization.

SyncML Target Audiences
Four main groups are expected to benefit from the adoption of SyncML as an industry standard, each in its own specific way:

End users. SyncML promises to provide a single way to synchronize email, calendar, to-do lists, contact lists, as well as corporate application data. This would be a welcome change to the current landscape where different synchronization software is required for each task.

Device manufacturers. For devices that have built-in synchronization capabilities, SyncML offers a protocol that will allow the device to access a broader range of data sources, over any network and transmission technology. This will enable device manufacturers to provide a wider range of features with reduced overhead.

Service providers. Companies in the application-hosting field currently have to limit their offerings due to the numerous forms of synchronization technologies on the market. As it stands now, service providers already have a difficult time managing the few technologies they do support because they all have different requirements for devices, networks, and back-end data sources. SyncML should alleviate these issues by providing a single protocol that will be able to work with a broad range of applications, devices, and networks.

Application developers. Developers want to focus their efforts on technology that is in demand and will result in successful applications. SyncML should provide a single technology that will allow them to build a variety of data-driven applications without having to learn a number of synchronization technologies. This should result in developers being able to build applications that will work on a larger number of devices, over many network protocols.

SyncML Advantages
In addition to the advantages for the specific groups just listed, SyncML provides seven other advantages that should prove to be beneficial in a synchronization solution. Many of the existing synchronization solutions already include many of these advantages, but again, SyncML is attempting to provide them in an industry standardized way:

Designed for wireless and wireline networks. SyncML works over all networks used by mobile devices, including both wireline and wireless networks. Some of the challenges in this effort come from the characteristics of today's wireless networks: high latency, low bandwidth, low reliability, and high costs. To address these issues, SyncML has implemented a robust protocol that uses WBXML, to reduce the data being transferred; uses a single request-response model, to minimize network traffic; and has support for dropped connections during synchronization.

Support for multiple transport protocols. SyncML works over the following leading network protocols: TCP/IP, HTTP, Wireless Session Protocol (WSP), and OBEX (Bluetooth, IrDA). It can also be deployed over SMTP, POP3, IMAP, and proprietary wireless communication protocols. SyncML does not require any additional capabilities that are not addressed by these protocols.

Support for arbitrary data. The capability to support various forms of data is a requirement for any general-purpose synchronization solution. Since SyncML is targeted at providing synchronization for all forms of enterprise data, it has built-in support for the following data formats, and can add new formats if the need arises:

Personal data formats, such as calendars, to-do lists, and contact lists

Collaborative formats, such as email and network news

XML and HTML documents

Relational data

Binary data formats, binary large objects (BLOBS)

Data access for multiple applications. To work effectively with the largest number of possible applications, SyncML is not tied to any specific programming language. It does not make any assumptions about the programming language being used, and will work effectively when a different language is used on the client and the server parts of the application.

Optimized for constrained devices. Since many of today's mobile devices have limited memory and processor capacities, SyncML is designed to work in very constrained environments. By keeping the footprint and processor requirements to a minimum, SyncML-based application should be able to operate on a broad range of devices with varying processors, memory capacities, and operating systems. In addition, the data exchanged using SyncML is in a binary format, reducing memory requirements for storage.

Takes advantage of existing Internet technologies. Since many of today's corporate applications are Web-based, it is important for a synchronization protocol to be able to work with Web technologies. XML has emerged as the preferred way to represent structured data on the Web, so SyncML is XML-based, ensuring easy implementation and quick interoperability testing.

High level of interoperability. SyncML has been designed to work over any network, with any device and any data source. Even though this goal has yet to be realized, SyncML does work very well with highly constrained devices, allowing them to communicate with synchronization servers that are capable of executing complex synchronization logic.

How SyncML Works
As we take a look at how SyncML works, we are going to focus on the guidelines that have been laid out in the specifications, not individual vendor products. Once you have an understanding of the underlying SyncML protocols, you will be able to quickly evaluate vendor offerings to determine if they meet your specific requirements.

Specifically, we will examine the following components of the SyncML specification:

SyncML Representation Protocol

SyncML Synchronization Protocol

SyncML Transport Bindings

The SyncML Representation Protocol specifies a common synchronization frame-work and format that can be used to create different data synchronization models. This protocol is defined by a set of well-defined messages (either XML documents or MIME content type) that are transported between the devices participating in a synchronization procedure. It supports both the request/response and the blind push command structures as data synchronization models.

SyncML synchronization is an exchange of packages; each package contains one or more messages. The SyncML representation protocol specifies the expected result of various synchronization operations, based upon the SyncML synchronization frame-work and associated data synchronization models. In this way, the SyncML representation protocol defines the structure of a SyncML message. Each message within a package contains a single data synchronization operation.

A SyncML message is a well-formed XML document consisting of a header and a body. The header contains metadata, which conveys information about the SyncML message. This metadata may include versioning, session, routing, and security information. The body of the SyncML message contains a list of commands, which allow for inserts, updates, deletes, and other actions to be executed to make the two datasets the same.

Figure shows an example of the SyncML framework; notice the dotted line that defines its boundaries. It is in this framework that the SyncML Representation Protocol is applied. When you look at the overall interactions between the SyncML client and the SyncML server, the SyncML Sync Protocol is used. This protocol is essential for achieving interoperable data synchronization. It defines the protocol for various synchronization procedures, which occur between the SyncML client and the SyncML server in the form of message sequence charts. The SyncML Sync Protocol allows for both one-way and two-way data transfer in a variety of modes. One-way synchronization is an important feature because it allows clients to save time by sending only their changes to the server; clients do not have to receive the server changes at the same time. It also is appropriate where only one set of data has changed, in which case two-way synchronization is not required; by default, only the changed records are synchronized between the client and the server. If the full set of data has to be synchronized, there is a slow-sync mode that will allow this.


Figure: SyncML framework.

Within the SyncML Sync Protocol, two roles are defined: a client and a server. Most of the time, synchronization is initiated by the client application, although some of the always-on networks allow the server to initiate the transfer. Do not be fooled into thinking that the server role within the protocol has to be performed by a powerful server-class machine: Both the client and the server may be mobile devices.

When it comes to the transport layer, SyncML defines a set of transport bindings. These bindings are required to ensure that there will not be any problems using the underlying protocols. The three transport bindings currently supported are HTTP, WSP, and OBEX. Each of these is supported for a reason. HTTP is widely used, and usually is able to pass through corporate firewalls; WSP is a part of WAP, allowing WAP phones to support SyncML; finally, OBEX provides support for short-range connectivity, possibly using Bluetooth, IrDA, or USB.

Now let's see how this works. Figureshows how two applications communicate with each other using SyncML. Application A represents a server application that can synchronize with multiple client applications, such as Application B. The communication between these two applications can occur over any network transport, such as HTTP, WSP, or OBEX. The client application (Application B) uses the Sync Client Agent to access the network and send messages to the server. For this to happen, the client agent communicates with the SyncML Interface (SyncML I/F), which in turn uses the SyncML adapter to send the XML message object. The server (Application A), then receives the message through the Sync Server Agent and manages the entire process using the Sync Engine. All of the Sync Engine's network access and data synchronization operations to and from the client application go through the Sync Client Agent.

This example is at a conceptual level. As vendors bring commercial products to market, these products may not contain all of these concrete components. However, the functionality that is contained in these components will have to be incorporated into their solution in some fashion.

Future of SyncML
The first products to support the SyncML initiative have focused on the synchronization of PIM data such as email, to-do lists, contact lists, and calendar information. Companies like Starfish Software, Pumatech, Synchrologic, and FusionOne have added support for SyncML at this level. There has been some adoption by other SyncML sponsors, including Palm, Symbian, and Nokia; all of which have added SyncML support to their latest device offerings.

In contrast, SyncML has not been widely adopted for enterprise data synchronization. To date, only IBM (a founder of the SyncML initiative) has released SyncML-compliant products. Oracle, Sybase, and Microsoft have all decided to stay with their own synchronization solutions, which are not based on SyncML. These products have been on the market for some time, and it is unlikely that they will support SyncML anytime soon.

It is still too early to tell whether SyncML will become an industry standard. SyncML is currently in a prestandards phase, where it will remain until it gains sufficient momentum and market acceptance to justify its adoption by an international standards body. At this time, more than 700 companies are supporting the SyncML initiative, although many of them are taking a wait-and-see approach. If these companies start to release SyncML-compliant products, we could very well see plug-and-play integration between any device, on any network, and be able to synchronize with any data source.

Synchronization Architectures

Enterprise data synchronization is not a straightforward task. Being able to communicate with a variety of enterprise systems over wireless and wireline networks to an assortment of client applications requires some flexibility at the architectural level. Considering that there are half a dozen mobile operating systems, with at least that many networks, and even more back-end systems that have to be accessed, the synchronization layer is often one of the most complex components of a complete smart client application.

After you take a look at the most common synchronization architecture, along with common synchronization topologies and data access methodologies, you will have a good foundation from which you can make informed decisions on implementing data synchronization within your mobile solution.

Architecture Overview
Synchronization is most often implemented in a distributed computing architecture with a client layer, a middle-tier layer, and an enterprise data layer. Each layer can be implemented using varying techniques, all aimed at accomplishing the same goal: providing a way to extend enterprise data to a variety of mobile devices.

We are going to take a closer look at each synchronization component and how it contributes to the overall synchronization process.


Figure: Synchronization architecture.

Client
When synchronizing data between an enterprise server and a persistent data store on the client device, a synchronization layer must be present to manage the two-way data communication (see Figure). Ideally, this layer will have a minimal impact on your client application, while still providing a simple, easy-to-use client API for controlling the synchronization process. By implementing a modular, self-contained synchronization layer, you can control the entire synchronization process with little interaction from the client application. In some cases, all that is required from the client is the invocation of the synchronization process; the synchronization layer does everything else from there.

Because so many client devices are on the market, the synchronization client must have support for the leading mobile devices, including laptops, Windows CE devices, Palm OS devices, Symbian OS devices, as well as specialized devices with add-on features such as barcode scanners and other industrial components. Each of these devices can have a different mobile operating system with different network protocol support. The synchronization layer on the client takes care of the network communication from the device back to the synchronization middleware.

Middleware
The synchronization server is where most of the synchronization logic is contained. Figureillustrates the role of the synchronization server in relation to the other components of the synchronization architecture. This server is responsible for communicating with the client application to send and receive required data packets. In order to do this, it has to be able to communicate over a protocol that the client application understands. Most of the time, this protocol is IP-based, and often is HTTP. When the synchronization server receives the data from the client, it then has to execute the synchronization logic to determine how this data is transferred into the enterprise data source.

Many of the advanced synchronization features are implemented within the synchronization server. Some of these features include data subsetting, conflict detection and resolution, data transformation, data compression, and security. All of these features have to be implemented while still maintaining server performance and scalability.

Two common synchronization server implementations exist: as a standalone server application, or as Java servlets running in a servlet engine. Both of these methods have benefits and drawbacks. The standalone synchronization server is convenient because it does not require any additional software to execute. These servers are usually programmed using the C programming language, taking advantage of OS-level calls, leading to enhanced performance. This also means that the server has to be available for the operating system to which you are deploying or you are out of luck. In terms of scalability and availability, the server can either have its own built-in load-balancing and failover mechanisms or use third-party load-balancing solutions, such as the hardware-based systems provided by Cisco systems.

For Java servlet-based synchronization servers, you will need a servlet engine for deployment. Since J2EE application servers are now commonplace in most organizations, this requirement does not usually pose a problem. By using an outside servlet engine, the performance, scalability, and availability of the synchronization server now rely on the capabilities of the application server/servlet engine being used. The same goes for the server operating systems that are supported; that is, as long as the servlet engine works on a given platform, the synchronization servlet should work as well. That said, you should give extra consideration to the synchronization vendor's supported platforms and recommended application servers when deciding which application server and operating system to use.

Enterprise Integration
The final part of a complete synchronization solution is the enterprise integration layer. While this layer is often part of the synchronization server, we are discussing it separately because it provides different functionality. The enterprise integration layer enables you to communicate with various back-end data sources. If you are using a commercial mobile relational database on the client, you will most likely have integration to enterprise relational databases on the server using ODBC, JDBC, or native drivers.

In addition to providing integration to relational databases, you may also require access to other forms of enterprise data, such as ERP systems, CRM systems, or XML data. If this is the case, you will have to look for additional enterprise adapters for the solution you are implementing; or if you have the resources, you can create your own adapter.

Publish/Subscribe Model
One of the most common models of data access being used by synchronization solutions today is the publish/subscribe model, so it is worthwhile to explore how it works.

The publish/subscribe synchronization model is based on the concept of having a master copy of your data, the publisher, and one or more copies of this data, the subscriber(s). The data between the publisher and subscribers is updated periodically to keep the data consistent. The update process, or synchronization, is bidirectional, allowing the publisher to update the subscriber data, and vice versa.

The publisher is responsible for defining which tables (or subsets of those tables) are available for external access. The data sets that are defined as being available are called publications. The application clients that are interested in having access to that information are called subscribers. They define which data they want access to using subscriptions. It is possible to have a number of publications defining the available data. These publications can have parameters, which makes it possible to use different subsets of data for different users. In this way, one publication can be used to send a different subset of data to each subscriber.

Figure shows an example of a field service application's data set. In this diagram, you can see that the publisher has the complete set of master data. This data contains all of the information for each field service representative. In this example, the field service representatives are the subscribers. They require only the subset of the publisher's data that pertains to their territory, so rather than keeping a copy of the entire set of data, they have a subset of data that is based on their individual subscriptions. In this way, you can easily define which data is available for mobile users and which parts of the data needs to be synchronized down to the mobile devices.


Figure: Publish/subscribe data synchronization.

Common Synchronization Configurations
A variety of network and data management configurations are commonly used with enterprise synchronization. Synchronization configurations (or topologies) are arrangements of publisher and subscriber databases that transmit data to one another. The publish/subscribe model of data access supports both peer-to-peer and hierarchical configurations.

The most common of these two is the hierarchical configuration, wherein every database has a single parent database, except for the master enterprise database, which has no parent. A diagram of common hierarchical layouts can be seen in Figure. In this diagram are two hierarchical configurations: the configuration on the right is commonly referred to as a hub-and-spoke configuration. In both cases, hierarchical configurations are good when a publisher needs to publish data to a large number of subscribers, or, in other words, when there are a large number of remote workers who need to have access to the enterprise data. In this configuration, the master database contains all of the changes made by any of the remote databases. Those updates can then be propagated to the other remote databases during their next synchronization.


Figure: Hierarchical database configurations.

Unlike the hierarchical configuration, peer-to-peer configurations do not have a single common enterprise data store. In Figureyou can see that peer-to-peer configurations enable each database to contain the same information, with no single database acting as a central server. In this configuration, you can still use the publish/subscribe model, but the data updates are not going to be propagated to all databases in the configuration. For this reason, peer-to-peer configurations are best suited for situations in which only two systems need to be synchronized, or when it is not important for each remote user to have the updates from all other users. In general, a peer-to-peer configuration is not recommended, because there is no central server responsible for collecting updates from the remote users and propagating those changes to the other users in the system. Other difficulties of peer-to-peer configurations include maintaining data integrity, implementing conflict detection, and programming synchronization logic.

Synchronization Fundamentals

In the previous post, we learned how persistent data storage enables users to access enterprise data without being connected to a network. Without enterprise synchronization, the data on the devices would quickly become stale, hence unsuitable for many corporate applications. The synchronization process allows you to execute bidirectional updates on the required data. Any changes that have been made on the client device can be transmitted to the server database, and any changes on the server can be transmitted to the client device. In this way, you can keep the data on the client and the server synchronized.

Whether you are synchronizing over wireless networks or a wireline connection, synchronization offers many important benefits over an always-connected solution, namely:

Reduced data transfer over the network, often leading to reduced transmission costs.

Reduced loads on the enterprise server.

Faster data access, because the user does not have to constantly wait for data to download.

Increased control over data availability.

In the following pages, we will take a look at how synchronization works and introduce features you should consider as part of an enterprise synchronization solution.

TYPES OF SYNCHRONIZATION
The term synchronization applies to more than the movement of enterprise data. It is often used to describe the exchange of personal information management (PIM) data and the transfer of application files. This section briefly explains other types of synchronization before we spend the remainder of this post on the synchronization of enterprise data.

PIM Synchronization
One of the first applications that many corporations make available to mobile workers is email. This is often enabled at the same time as contact lists, to-do lists, and calendar entries. The combination of this data is commonly referred to as PIM data. Wireless access to email is often considered the killer application in the mobile world. It is a subject to which everyone can relate, and the benefits of which most people by now take for granted.

The leading PIM applications on the desktop are Microsoft Outlook and Lotus Notes. Both vendors provide mobile access to PIM data via laptops, so users are familiar with accessing this data remotely. When it comes to smaller mobile devices such as PDAs, a PIM synchronization solution is required to provide this functionality. PIM synchronization solutions allow users to have offline access to their personal information and to synchronize any changes made back into the enterprise system, often without requiring a companion PC.

File/Application Synchronization
Within an organization, a variety of applications and other files need to be distributed to mobile users. In the wired world, this information is often made available through use of portable media such as floppy disks or CD-ROMs or by a corporate server or intranet site. Unfortunately, these techniques do not work nearly as well for mobile workers. Many mobile devices do not have support for traditional portable storage media, and the storage that is available is somewhat expensive, making this an impractical means of distribution. Adding to the challenge is the fact that most users are at remote sites, without access to internal corporate systems.

To overcome these challenges, several file and application synchronization products have come to market. These products allow an administrator to deliver these files to remote workers in an automated way; they require no additional effort by the remote user. This means you can deploy and manage files and applications on remote devices, so you can quickly update remote workers with the applications they need. In most cases, this type of synchronization is one-way: Updates are sent from the server to the remote workers; only an acknowledgment receipt comes back.

Data Synchronization
Data synchronization refers to the bidirectional exchange and transformation of data between two separate data stores. For our purposes, these data stores are most often located on the client device and the enterprise server. In many cases, the synchronized data set is only a subset of the enterprise database. The reason for the limitation is twofold: one, because of the limitations of most mobile devices, and, two, because individual mobile users only require data that is relevant to them.

The data transfer itself occurs between the synchronization layer on the client and the synchronization middleware on the server. The server middleware then communicates with the enterprise data source. This is the most common way that smart client applications are given access to corporate data.

Database Development Options

Now that we have shown some of the benefits of including persistent data storage in your mobile applications, it is time to take a look at the options available for implementing this functionality.

Several factors will influence which method you decide to use. They include the following:

Required features. Read the features listed earlier. Which are important to your particular application? A few key ones are performance, data storage capacity, transactional capabilities, operating system support, and tools integration.

Application complexity. How complex is the application's data structure? Does it require relationships between data sets? What forms of data manipulations are required?

Available resources. Such resources include both financial and development resources. Often a trade-off occurs between the cost and functionality. In addition, consider the internal development and maintenance costs versus the costs of a commercial solution.

Size of deployment. The number of seats being deployed can have a big impact on your decision. The per-seat cost of developing an in-house solution is higher for smaller deployments. At the same time, the development effort can become much more complex when building an in-house solution that has to support a large number of users. Determining the point at which it becomes more cost-effective to build instead of buy has to be determined on an individual basis

Support for standards. Is a proprietary system suitable for your corporate needs, or do you require support for standards such as SQL, ODBC, and JDBC?

Enterprise integration. What kind of enterprise integration is required? Many of the commercial systems come with built-in database synchronization capabilities that will work over a variety of protocols. Does your application require this, or would it be overkill?

As you read the options available, you will want to weigh the pros and cons of each solution against the requirements of your particular application. This decision will have both near-term and long-term consequences, so be sure to dedicate the appropriate effort to it.

Proprietary Storage
All of the leading mobile operating systems include a form of local data storage. These data storage systems are typically flat-file databases with limited features and storage capacity. The interface to these databases is most commonly at the API level. Let us take a look at how data is stored in each of the leading operating systems.

Palm OS
Any kind of persistent data storage on the Palm OS is contained in a flat-file database. This type of data includes both applications as well as user data. Two kinds of database systems are available: record database and resource database. Each is composed of blocks of memory that can contain any application-defined data. The only difference is the size and contents of the header for each of the objects. In general, record databases are used to store application data, such as contact lists. Resource databases are used to store the application code and user interface objects. For our purposes, we are going to concentrate on the record database, since the resource database is mainly used for managing applications, not user-defined data.

As the name indicates, a record database stores records of data. These records are essentially segments of memory that are part of the storage heap of the Palm OS. Each database contains a database header that describes the database, AppInfo and SortInfo blocks, and the record list. Both AppInfo and SortInfo are optional blocks that describe the application information and sorting information, respectively. Figure shows the logical record database layout.


Figure: Logical layout of the record database.

Within the header, various fields describe the database name (which is a 32-byte string), its attributes, the last modified date, the creatorID, the AppInfo, the SortInfo, and many other items. The header ends with a record list, which has its own header, followed by the record entries. Each record entry references the location of the actual data for the record and contains the attribute and ID information for that record.

From a programmer's perspective, the Palm DB is a quick and easy way to add persistent data to an application. To read or write data, the Palm OS provides functions that can be used to convert data into or out of Palm database formats.

When using the Palm database for your applications, keep in mind that it does have some limitations. First, the application programmer is responsible for database creation and maintenance. This means that within your application code you will have to use the Palm OS API to open, close, and access the data in any way. In addition, you will have to manage the code to create new records, access or modify them, or remove records. If you require additional functionality, it will have to be created by the programmer.

Second, the Palm database is a flat-file database, with all the limitations discussed earlier in this post. The main issues are that the Palm database limits the complexity and size of the data storage capabilities and does not have support for transactions. This may not be a factor for simple applications, but it can be a major limitation when representing a portion of an enterprise system on a handheld device.

The final limitation relates to enterprise integration. Palm databases do not come with very robust synchronization capabilities. You can synchronize blocks of data using HotSync, but this capability is often quite limiting in the enterprise space, where a companion PC is not always available and the data integration is often directly to an enterprise RDBMS. In these cases, you are required to write your own data synchronization conduit. This effort will involve creating your own data synchronization layer, which is not a trivial task.

Windows CE
Windows CE contains a built-in database system in the form of its object store API. It is a flat-file database geared at application programmers, to enable them to include persistent data storage in their Windows CE applications. The database is physically located in what is referred to as the object store, a portion of a device's memory allocated by the operating system for persistent storage of files, the Windows CE Registry, and Windows CE databases.

One advantage of this flat-file database is its ease of use. By using the provided API functions, you can add and remove records, search for a particular record, and sort the records based on four different columns. Another helpful feature is the support for timestamps as a data type. This allows the programmer to see if the data has been altered by another application since its last use. The API also includes functions for finding records. These include the FirstEqual, NextEqual, ValueGreater, and ValueSmaller methods, providing conditional search functionality without much extra programming.

Similar to the Palm database, the object store API has some major limitations for more advanced applications. The first is that it can reside only within the device's RAM, meaning it cannot be placed on an auxiliary storage card. This limitation can pose problems for applications that are shipped on PCMCIA or Compact Flash memory cards. In these cases, before the application can run, the database will have to be written to the device's built-in RAM.

Although flat-file databases work well for simple applications or those with mainly static data, they can introduce performance problems for more complex applications. When searching for a specific set of data or attempting to sort data, extra commands and I/O are often required, thereby inhibiting performance. This will become especially noticeable with large data sets.

As with other flat-file solutions, database synchronization with Windows CE object store is also an issue. You are left to develop your own synchronization routines. As discussed in the Palm OS section, this undertaking can require a significant amount of effort to implement.

Symbian OS
The DBMS in the Symbian OS is significantly more advanced than that in either the Palm OS or Windows CE. Rather than using a flat-file database structure, Symbian OS has implemented a relational database. This means you can have multiple tables in the same database, allowing for more complex schemas. In addition to making the storage capabilities more advanced, Symbian has also added data access through a subset of SQL, as well as using C++ APIs. This too provides a more approachable way to interact with the Symbian DBMS.

For most of the common database commands that you will require, such as opening and closing the database, creating tables, creating indexes, and querying data, you have the option of using either SQL or the C++ API. In most cases, the SQL code is more efficient than using the C++ API, with the exception of inserting rows of data. In this case, using the C++ API provides better performance, especially on large amounts of data. Overall, incorporating the built-in DBMS into your applications will require a considerable amount of custom code.

One of the other nice features of the Symbian OS DBMS is the support for transactions. When database changes occur within a transaction, all changes can be undone if something goes wrong during any part of the data manipulation. This is an important feature when you are modifying data in more than one table, ensuring that internal references remain intact.

As with the other built-in databases, the Symbian OS DBMS also has some limitations. For one, the API is only available in C++, limiting access to the other supported programming languages such as Java. This can be a fairly significant limitation as it is expected that Java and other high-level languages will be frequently used for Symbian OS development. Second, even though it is a relational database, it does not offer functionality that is normally found in commercial products. For example, table joins are not supported. This means that you cannot select, insert, or update data from more than one table in a single statement. Finally, there is the synchronization issue: With this database, too, you are required to implement your own synchronization support, which as explained for the Palm OS and Windows CE databases, is a significant challenge.

J2ME
The Mobile Information Device Profile (MIDP) of J2ME provides a persistent storage mechanism called the Record Management System (RMS). The RMS is a set of Java classes for storing and retrieving simple sets of data. The storage takes place in what is called a RecordStore. Each RecordStore consists of a collection of records that are stored in nonvolatile memory, meaning they are persistent. Internally, the data is essentially stored as a byte array.

The MIDP applications that access the RecordStores are called MIDlets. These MIDlets can be grouped together for deployment into MIDlet suites. Each MIDlet suite has its own space for RecordStores; no MIDlet suite is permitted to access other MIDlet suites' RecordStores. That is, there is no mechanism to share RecordStore data between MIDlets in different MIDlet suites. So, if you want to enable more than one MIDlet to access the same set of data, the MIDlets in question have to be in the same suite.

Each RecordStore can be given a name, which is case-sensitive and has a maximum length of 32 Unicode characters. There are not many naming restrictions, other than that two RecordStores within the same suite cannot use the same name. Within the RecordStore, each record is assigned a unique ID. These IDs are managed by the RMS and are autoincremented; thus, the first record will have an ID of 1, the second an ID of 2, and so on.

The application programmer is responsible for managing the RecordStore. There is a set of APIs to open, close, insert, delete, and move through the records. More advanced classes, such as RecordComparator and RecordFilter, allow you to compare two records and filter records, respectively.

As you can see, the MIDP RMS is a very simple data storage mechanism, for limited amounts of data. As a developer, you are responsible for managing the RMS as well as for implementing any synchronization capabilities that may be required.

For other J2ME profiles, commercial database solutions are available. These are discussed later in this post.

Proprietary Storage Summary
The leading mobile operating systems all have a built-in form of persistent data storage. The capabilities of these databases vary, but they all are best suited for simple applications that do not require large amounts of data or have complex data structures. If this is the type of application you are building, then these databases offer you a quick, lowcost way to incorporate data storage into your application. You do not have to worry about licensing fees, because the database is part of the operating environment.

If you are building a more complex application, one that has more demanding data storage requirements, these databases are probably not the best choice. As mentioned before, when the data sets become large, these databases suffer from poor performance. They also do not offer many of the key characteristics that are required for many corporate applications. While they all have low resource requirements, they typically do not provide the security, data integrity, support for transactions, scalability, support for standards, or data synchronization capabilities that many solutions require.

One of the final points to keep in mind about all of these solutions is that they are very specific to the given operating system. If you are planning on deploying your application to multiple operating systems, you will incur additional development and maintenance costs using these solutions, as you are essentially working with a different database for each operating system. Everything about how you store and interact with the data is OS-specific, requiring a significant amount of additional work to port the application to another operating system.

Custom-Coded Databases
If the built-in databases do not meet your needs, you always have the option of creating your own database. In this scenario, you can create as simple or as complex of a database as your application requires. If you are looking for basic storage, but are not interested in the built-in databases due to their proprietary nature, you can create a simple flat-file storage mechanism that can work on many platforms. On the other hand, if you require complex data storage with relational database features, you can hand-code this solution. In most cases, this comes down to a "build versus buy" decision.

Companies face a predicament when they come to the conclusion that the available commercial solutions do not meet their needs for one reason or another. It could be because of cost, features, or because of anticipated performance issues. In any case, they have to make a decision whether building a database in-house will result in longterm benefits. This may not be an easy decision to make.

By building a custom database, you can ensure it has all of the features that you require. If you require a relational format with referential integrity, you can build it into the system. At the same time, to maintain a small footprint, you can omit any extra features that may not be required. When it comes to performance and scalability, the onus is again on you. This is sometimes preferable, as you can ensure that the applications run as expected; that said, development and testing often end up being very timeconsuming. Some of the other database features that you may have to custom-code include indexing capabilities to maintain performance, referential integrity to maintain internal data consistency, data-access APIs or SQL support, memory management, and file management, to name just a few. These features alone are often enough to prevent making an in-house solution feasible.

If you require enterprise data integration, you will have to consider building a synchronization layer into your solution as well. This involves building client-side as well as server-side code that can transfer data between your device and back-end server in such a way that you do not lose any data.

Before you start to build a custom solution, make sure that you are aware of the challenges that lie ahead so you are adequately prepared. It is no secret that building a custom database will require significant resources. This is true not only for the development stage but also for the maintenance stage. Remember, mobile operating systems change constantly. Possibly you will have to port your database to new operating systems and new versions of existing operating systems. Each port requires additional development and testing resources to ensure it operates as expected. These cautions are not meant to dissuade you from creating a custom database but rather to make you aware of the effort that will be required.

Of course, there are times when a custom database makes sense. If you are targeting an operating system that is not widely supported by commercial solutions, it may be the only way to get the functionality you require. If you have very specific data storage requirements, which are not addressed by commercial mobile databases, you may find building a custom solution to be to your advantage. These are just two scenarios for which building an in-house solution is beneficial. Just remember that building and maintaining mobile applications is challenge enough; coupled with adding the database, it becomes more so. So if you are contemplating building a solution in-house, research the solutions that already exist to make sure you use your resources to the maximum.

Commercial Relational Databases
Relational databases not only lead the enterprise database market, they are the clear leaders in the mobile market as well. This is largely due to the fact that many of the mobile offerings are from enterprise database vendors. These companies provide a variety of SQL-based solutions with a wide range of capabilities. Some offer key features such as referential integrity, support for indexes, and transaction support, while others do not. Some have open support for operating systems, synchronization integration, and programming languages, while others are more limited.

The point is, when it comes to mobile database implementations, all commercial databases are not created equal. In this section, we examine the leading vendors, summarizing their offerings to give you a place to start for evaluating which technology is best suited for your mobile applications.

Key Features:

Full-featured Adaptive Server Anywhere (ASA) database for desktop environments and Windows CE

UltraLite deployment for mobile operating systems, providing a relational database with a footprint as small as 50K

Built-in synchronization to any ODBC back-end data source

Strong encryption of both the data stores and communication stream

Summary:

To put more emphasis on its mobile and wireless computing solutions, Sybase launched a new company called iAnywhere Solutions. The flagship product offered by iAnywhere Solutions is SQLAnywhere Studio, which includes a variety of mobile database offerings. This product is the established leader in the mobile database market with 73 percent market share (Gartner Dataquest, December 2002). This lead over the competition is a result of iAnywhere's long-term focused commitment to mobile computing, which has enabled them to develop a set of products that squarely meet the requirements of enterprise customers. In the mobile database arena, three technologies are available: Adaptive Server Anywhere (ASA), UltraLite, and UltraLite Component Suite.

Adaptive Server Anywhere is a high-performance mobile database system that offers complete relational database functionality, including stored procedures, triggers, transaction management, record-level locking, and enterprise data replication. All of this is provided in an easy-to-use package that has a compact footprint (around 3 megabytes on Windows CE environments). This version of iAnywhere's technology is available for a variety of desktop and server platforms, as well as Windows CE.

If you are looking for a more compact solution, iAnywhere Solutions provides the innovative UltraLite deployment option. This technology analyzes the database features that your application requires and automatically generates a customized database containing only those features. You can then link this customized database with your application code to create a single executable for deployment. This reduces the complexity of implementing a mobile database solution. For development, you can use common C/C++ development tools such as eMbedded Visual C++ and Metrowerks CodeWarrior. If you are more interested in Java, iAnywhere also offers a pure Java version of the UltraLite database that allows you to develop with any Java development tool and deploy to any device with a Java VM.

In addition to the application-specific version of UltraLite, iAnywhere Solutions also has a more generic relational database for the same operating systems. This technology is called the UltraLite Component Suite; it provides interfaces for development using AppForge, Visual Basic, ActiveX controls, and Java, in addition to the C++ tools mentioned earlier.

On the synchronization side, iAnywhere Solutions offers its MobiLink synchronization server that provides direct, bidirectional synchronization to any ODBC data source, including Sybase, Oracle, Microsoft, and IBM. The synchronization rules are stored in the back-end database to provide a high level of customization capabilities. Both wireless and wireline protocols are supported for synchronization, including HTTP, TCP/IP, and common conduits such as HotSync and ActiveSync.

For security, iAnywhere provides 128-bit strong encryption of the communication stream as well as the data stores themselves.

With their wide platform support and open synchronization capabilities, the mobile database solutions from iAnywhere Solutions are well suited for companies that require a flexible solution that can integrate with heterogeneous back-end databases.

IBM

IBM incorporates three components in its DB2 Everyplace Enterprise product: a small relational database, a synchronization server, and a tool to develop smart client applications for the Palm Computing Platform and Java. Both native versions and Java versions (through IBM's acquisition of Informix) of the database are available. DB2 Everyplace is a relational database management system with a footprint as small as 150 KB. This RDBMS is composed of the default libraries and the database engine itself. The main goal of DB2 Everyplace it to provide a local data store on the mobile device while attempting to keep many of DB2's enterprise features. Some of the included features are SQL support for adding, deleting, and updating data. Missing is support for transactions, which is notable for many corporate developers. DB2 Everyplace implements a subset of the SQL 99 standard, DB2 Call Level Interface (CLI), Open Database Connectivity (ODBC), and Java Database Connectivity (JDBC).

On the development side, IBM offers its own development tool called Mobile Application Builder for Palm OS and Java applications. Mobile Application Builder offers a graphical rapid application development environment with wizards to aid development. DB2 Everyplace also integrates with Visual Basic, C/C++, and Java development tools.

For synchronization, the DB2 Everyplace Sync Server provides bidirectional synchronization from DB2 Everyplace to DB2 Universal Database. Once the data is in DB2 Universal Database, it can then be transmitted to other ODBC data sources, such as Oracle, Sybase, and Microsoft, by using adapters. The synchronization server is implemented as a Java servlet, so it can be hosted in any servlet engine, although it does work best in IBM's WebSphere application server. Being a Java servlet, the synchronization is HTTP-based, and works for both wireless and wired networks. SyncML is the synchronization protocol used by IBM for its data transfer.

With its broad platform support and small footprint, DB2 Everyplace is well suited for a variety of mobile applications. However, the enterprise integration is somewhat complex in that it requires DB2 Universal Database as an intermediate step before interacting with other data sources, so it may not be ideal for companies that do not currently use DB2 Universal Database.

Oracle

The Oracle9i Lite database is built upon the Oracle9i Database and Oracle9i Application server technology. It provides a lightweight relational database with a feature set aimed at being comparable to the Oracle9i database server. It incorporates other technologies (such as Web-to-Go and Mobile SQL) into a single product. This streamlines the installation and configuration procedures, breaking them into two groups: Oracle9i Lite Mobile Development Kit and Oracle9i Lite Mobile server.

The Mobile Development Kit incorporates a set of tools and APIs, along with sample code, to accelerate the development of Oracle9i Lite database applications on mobile devices. At the core of the product is the Oracle9i Lite database. It is a general-purpose database with support for ODBC and JDBC connectivity on all platforms using common development languages such as C/C++, Visual Basic, Satellite Forms, and Java. For Java development, you can take advantage of Oracle's lightweight JDBC driver from your Java clients, although the Oracle database itself is not written in Java.

As to the Oracle9i Lite footprint, it is considered more of a middleweight than a lightweight database. On Windows CE, for example, the combined memory required for the supporting files and engine is close to 5 MB, and this does not include the database itself. On other platforms, the footprint can be as low as 1 megabyte, which is still not small compared to other offerings on the market. In addition, the installation process and feature set vary from platform to platform, so there is no single solution for all mobile devices.

One of the unique features of Oracle9i Lite is how it manages mobile applications. The Oracle Mobile Server is an extension of the Oracle9i Application Server, making it possible to deploy, manage, and synchronize offline applications from a central console. Rather than having a separate engine for application management, Oracle has incorporated this capability directly into the database. This allows you to synchronize both data and applications from a single console. Unfortunately, the application deployment and management is available only for Win32 clients.

When it comes to synchronization, Oracle does not use standard interfaces such as ODBC or JDBC. This means that you can only synchronize Oracle9i Lite clients with Oracle databases. This is fine if your organization uses Oracle technology exclusively; if not, then you will find the synchronization limiting.

Overall, Oracle9i Lite is a capable mobile database platform best suited for companies with homogeneous Oracle server installations.

Microsoft

The latest offering in the Microsoft SQL Server family is SQL Server CE. This is a scaled-down version of SQL Server aimed at extending enterprise data to offline applications for Windows CE. It has maintained much of the functionality required for mobile solutions, including a wide range of data types and referential integrity. The database footprint varies, but is typically between 1 and 3 MB.

Microsoft SQL Server CE provides a familiar SQL grammar and a consistent development model and API set to the full-blown SQL Server. Development can be accomplished using Visual Studio .NET using the .NET Compact Framework. These tools and technologies allow developers to use their existing skills to build sophisticated Windows CE-based database applications for a wide range of mobile and embedded applications. In addition, Microsoft supports Remote Data Access (RDA) for accessing back-end Microsoft databases in real time without having to configure SQL Server synchronization.

Synchronization is provided over HTTP to Microsoft SQL Server back-end databases via Microsoft Internet Information Server (IIS). Replication can be performed over wired and wireless LANs and WANs. The communication protocol is well suited for wireless transports. Microsoft has also incorporated support for SSL, giving you the ability to encrypt the data stream to protect sensitive data.

Because the Microsoft SQL Server CE solution is targeted directly at Windows CE devices, and synchronizes only to Microsoft SQL Server, it is suitable only for a very specific market. If you are deploying Windows CE devices that only require integration with Microsoft SQL Server, then this solution should meet your requirements, while giving you a familiar development environment.

PointBase

PointBase is the most recent player to enter the mobile relational database market. Its database products are all 100 percent Java, to provide cross-platform development opportunities. There are two versions that will appeal to mobile developers: PointBase Embedded Edition and PointBase Micro Edition. Both are targeted at mobile applications development, but for different versions of the Java platform.

The PointBase Embedded Edition is aimed primarily at J2SE applications that require multiple database connections from within the same JVM. The Micro Edition works in both J2SE and J2ME environments that require only a single database connection. For this edition, the footprint can be as small as 45 KB; however, it is lacking many features that are required for enterprise mobile applications, most notably, synchronization support.

Both versions take advantage of common development tools, such as Metrowerks CodeWarrior for Java, for rapid application development. For deployment, you can use any JVM on the target platform. The Kada Systems' JVM is often used on the Palm OS, while Insignia Jeode is common on Windows CE. For smart phones and Symbian OS devices, you can use the JVM that comes with the device.

The Embedded Edition offers synchronization via PointBase UniSync. UniSync is a middleware server that allows the PointBase clients to synchronize to any JDBCcompliant back-end data source. Even though it is open to many back-end sources, the synchronization is better if you are communicating to the PointBase Server database. In this case, bidirectional, changes-based updates are supported. For other JDBC data sources, only snapshot updates are supported.

PointBase's main strength is its focus on Java. The concept of developing an application once and deploying it to any mobile device with a Java VM is very attractive to corporate developers. Unfortunately, Java on mobile devices is still relatively immature compared to server and desktop platforms, and users may suffer through some growing pains. However, on some of the more capable mobile devices, creating Java applications is a promising alternative.

Reasons for Using a Database

Database technology is the foundation of many enterprise applications. This is true in the world of e-business, and it is becoming more true in the world of m-business. As corporations continue to develop mobile business applications, the requirement for persistent data on the mobile device becomes increasingly prevalent. Just as databases made the client/server era of computing possible, databases are changing the way companies look at mobile computing. One of the main goals of mobile computing is to extend enterprise data beyond the confines of the enterprise. In doing so, many lessons have been learned, demonstrating how mobile data access for e-business (or m-business) differs from client/server data access. The following are some of the key differences between the two means of data access:

The mobile-device landscape is much more complex. Compared to the choices in the desktop world, there are many different devices, which use a broad range of hardware, operating systems, and user interfaces.

The number of devices and applications that need to be managed in the mobile landscape is increasing. As mobile computing grows, users may use more than one device, and will definitely use many mobile applications. Desktop computers are a known entity. There are not many new hardware systems or radical new applications coming to market.

Mobile users often need to use applications when not connected to a network. One reason is the unreliable nature of the wireless networks, but it can also be credited to the performance and depth that offline applications can provide. Desktop computers typically have reliable, continuous network connectivity, allowing data access to occur over a network without worrying about prolonged downtime.

All of these points lead the conclusion that mobile data storage is an important aspect of mobile applications.

Key Features
Now that we have looked at the types of databases available, as well as some of the key differences of mobile computing, we are in a better position to evaluate the key features that are important in a mobile environment.

Data Storage Properties
The database engine and the technique used for data storage will play a significant role in the database's storage capacity, performance, and scalability. You want to choose a solution that can store a large amount of data, being limited only by the memory available on the device. At the same time, the solution should have fast data access, to ensure that the database does not cause a performance bottleneck. There is often a trade-off between data capacity and performance; in some products, as the data set grows, the performance decreases.

Another important consideration is the support for enterprise class functionality, such as indexes, referential integrity, and, especially, transactions. Choosing a product that has these features will enable you to produce a robust, high-performance solution. The importance of transaction integrity cannot be overemphasized for mission-critical applications. Transaction integrity means that clients can perform database actions within transactions, so if something goes wrong in any part of the transaction, the entire transaction can be undone. This is an important feature when you are modifying data in more than one table, as you can ensure that internal references remain intact. Since your mobile application will not contain the entire schema of your enterprise data source, the database system should provide an easy way to define the required subset of data. Once defined, the synchronization layer should be able to automatically take care of the data mappings. This leaves you with a minimum amount of data on the device and a means of direct synchronization.

Tool Support
Having tools for the development, testing, administration, and deployment of your mobile application will make the development process more efficient. Support for leading development tools and programming languages enables developers to use products that they know. Ideally, for maximum flexibility, the database system will support development of both native and Java applications. Being able to leverage existing tools and programming language skills will dramatically reduce the amount of training required. When it comes to administration and deployment, the database vendor should provide tools that support all of your target operating systems.

Flexible Synchronization
The database solution should provide an integrated synchronization layer. The synchronization must be bidirectional, so that both the client and server data can be updated. At the same time, the amount of data being transferred has to be kept to a minimum. One way to accomplish this is by synchronizing only the net changes. Imagine the case where a data field changes several times between synchronizations. When synchronizing net changes, the only information sent to the enterprise server would be the final value, rather than all of the intermediate values, which could add considerable overhead to the data being transferred. For most applications, receiving only the final value from the synchronization should be sufficient, but for others a complete transaction history might be required. A solution that offers a choice between the net changes and complete transaction history would be ideal.

Mobile applications run on many devices, using many communication protocols. The synchronization capabilities should be able to run over a variety of both wireline and wireless protocols. The system should also support access to enterprise data being stored in the leading enterprise database products.

Administration Requirements
In most mobile environments, technical staff to troubleshoot and resolve problems is not readily available. At the same time, it is unrealistic to expect the end user to take care of troubleshooting tasks. This means that you need to choose a solution that requires zero client administration. To accommodate this requirement, the database must be easy to install, configure, and manage.

When support is required, having remote device support capabilities can be very helpful. Such support may include the ability to remotely control a mobile device and to remotely deploy application updates to the device.

Low Resource Requirements
Because mobile devices have constrained resources, the solution has to be able to run efficiently on handheld devices. The database should be capable of being run on slow processors and provide a small footprint reducing memory usage. Choosing a solution that is modular, giving you only the features required for your application will minimize resource usage. Also note that native applications are often more efficient than Java-based solutions, so native support for your target operating systems can be important.

Operating System/Device Support
At a minimum, the database you choose should be capable of running on the leading operating systems and devices, both during development and deployment. Moreover, because the mobile operating system and device market is changing constantly, you need a solution that can adapt quickly to those changes. At a minimum, the operating systems it supports should include Windows CE, Palm OS, and possibly Symbian OS. You are also advised to choose a database that can run in the emulation environments that you have selected. If it does not, the development process will be much more difficult, because physical devices will be required for all stages of the development cycle.

Standards Support
Probably the developers in your organization have done some database development, meaning they have SQL skills as well as knowledge of ODBC or JDBC. Because these standards are used widely and have a broad level of industry support, the database system you choose would ideally support these standards. Even if you do not have previous database experience, choosing a solution that is standards-based will help you to learn the system faster and gain skills that are transferable to other platforms. In addition, technical resources are more readily available for standard systems than for proprietary ones.

Having a SQL relational database on the client can also ease the integration process with enterprise databases, which are commonly SQL relational as well.

Security
The database system should support security throughout all parts of your application. This includes both authentication and data encryption. To prevent unauthorized users from accessing your confidential data, having built-in authentication is important. This authentication should also carry through to your back-end data source during synchronization.

Two levels of data encryption are required: encryption of the communication stream, and of the data stores themselves. The encryption of the data stream will prevent others from listening in when data is transmitted from the data store to the enterprise database. The encryption of the client data store will ensure that confidential data cannot be viewed on a device that is lost, misplaced, or stolen.

Persistent Storage versus Real-Time Access
An argument can be made that direct real-time access to enterprise data is better suited for enterprise applications than persistent client data storage. This solution would involve having a communication layer within the client application that would go over a wireless network to retrieve data from the enterprise server. In the same way, the business logic could reside on the server as well, removing the need for client-side logic. This would leave the presentation layer on the device, along with a communication layer, allowing for enterprise integration, making the result very similar to wireless Internet applications. In this case, the real-time data access solutions would suffer from some of the limitations that real-time browser-based applications do, namely:

Unavailable or unreliable wireless networks. In many cases, mobile tasks such as sales force and field service applications take place in remote locations, where wireless coverage is either unavailable or unreliable. This is also true of workers for whom physical barriers, such as office buildings, cause interference issues.

Insufficient bandwidth. A significant amount of data is being recorded with data capture applications. With the low bandwidth of today's networks, constantly transmitting data is impractical and costly.

Static data. Much of the data retrieved in mobile applications is relatively static, requiring only periodic updates. Examples include customer contact information or product catalogues. In these cases, it does not make sense to be downloading the information over a wireless network when it can be stored persistently on the device.

Battery life. Communicating over a wireless network requires significantly more battery power than accessing data locally. This is a concern for workers who do not have the ability or time to charge their devices throughout the workday.

For all of these cases, developing a smart client application with persistent data storage and enterprise synchronization is a more practical and cost-efficient solution. In many cases, the data can be synchronized via a wireline network, avoiding the wireless network limitations.

And when real-time data is required, wireless Internet applications appear to be a better choice than creating custom client applications with real-time access. Wireless Internet applications do not require a client to be deployed to the device, plus they can communicate over standard protocols such as WAP and HTTP. Another option is hybrid applications that use local data storage for most of the data access, and wireless Internet access for content that is the most critical or time-sensitive.