Enterprise Application Messaging

Mail Enabled Applications, or MEAs, have great potential in an Enterprise Messaging Environment. They have the potential to cause great harm to mail flow and mailbox availability, as well as the ability to enable better communication and performance of non-email based business processes. As such a comprehensive policy should be established in every Enterprise to provide a framework for managing them. By doing so, the Enterprise can reap the benefits of MEAs while minimizing or even eliminating the dangers and risks.

The Future of Application Generated Messages in the Enterprise

Author: Bill Anderson

Company: Baldguy Software

Last Update: 6. June 2006

 

 

 

 

Table of Contents

Policies

MEA Registration

Throughput and Capacity

Compliance Levels

Services

Custom Library Implementation

Email as Service: Going Beyond SMTP

Extending The Messaging Arena

Putting it Together, an Example Scenario

The Policy

The Server Side

Migrating Existing Alerting Systems

 

 

 

 

Policies

Policies surrounding an MEA Service Bus could include such items as definition of acceptable MEAs. This would imply or mandate that the service provider (SP) has vetted that the MEAs on the list have met the criteria for inclusion into the environment. Such a policy should also include a means for approving new MEAs for list inclusion. Additional policy ideas would include limits on acceptable rates, use periods, and penalties/ramifications of exceeding them. Included in such a policy should be acceptable uses of Email.

For example, it is common practice to use Email for monitoring other applications or processes. In an Enterprise Messaging Environment(EME) where these messages use the same destination and server chain as business critical or regular business email, the practice should be prohibited. Combined, this set of policies should be well communicated and easily available for application owners and developers. One could even call this an MEA Acceptable Use Policy, or AUP. A brief sample follows using arbitrary figures.

Example Policy Brief

  • Mail Enabled Applications (MEAs) will inject no more than 60 messages per minute to a given internal recipient without regard to size..

  • MEAs will inject no more than 60 messages per minute to a given external recipient without regard to size.

  • MEAs will inject no more than 600 messages per minute to a given external domain without regard to size.

  • MEAs will inject no more than 20MB per minute to a given recipient without regard to message counts.

  • Standard Email is not to be used for process or application monitoring.

  • Alerting and Monitoring mechanisms for application and processes will use the separate Alerting System provided by the Messaging Team

  • MEAs will utilize standard messaging libraries provided by the Messaging Team.

  • Application email is a lower priority and has a minimum SLA 40 minutes.

MEA Registration

An essential part of the process for any MEA should include some form of MEA registration. This provides an opportunity to provide standards and policy information to the application’s owner/developer and to do so in a verifiable manner. MEA registration also provides a means for the IT/Messaging Team to contact the MEA owner/manager should an incident with the application occur.

MEA registration should be integrated with the systems accepting email from applications. By limiting access to the MEA Bus to registered applications accountability is increased as is the ability to disable access under extreme conditions. When MEA incidents are traceable to specific individuals and organizations, they will police themselves more thoroughly. This can often be as effective as good communication and education regarding policies.

The registration process may include the ability to download any standard libraries, obtain configuration information for applications, as well as registration of contact information and agreement to SLAs. MEA Registration can provide the lynch pin for any well designed Messaging system.

Throughput and Capacity

Eventually even the best laid plans of education and policy policing will be circumvented, applications will encounter bugs, and alert triggers will spawn like bacteria. It is under these cases that a messaging system capable of throttling mail from MEAs will save the messaging team from countless hours of recovery and restoration, not to mention financial and reputation losses from an impacted mail system. The primary recommendation for providing this is what we call an Application Bus or MEA Bus. This is a set of one or more servers that handle all application mail and provide access and flow limits. This is particularly important if the Email back end consists of MS Exchange mailbox servers, or if much external mail may be generated. The use of an MEA Bus also isolates the potential complexity of application registration and access control, throttling mishaps, etc.. It also provides a buffer that the messaging team can utilize in the event of mail storms from misbehaving of broken applications.

Compliance Levels

Compliance with an MEA policy need not be a binary situation. A policy plan may include degrees or levels of compliance. This is especially useful in migrations to a policy and is strongly recommended for large enterprises. Tracking MEA compliance provides feedback to all necessary parties as well provides a plan for full compliance.

For example, consider the following level description:

  1. Zero Compliance
  2. Level One Compliance
    1. Application is properly registered
    2. Application uses the correct relay (system)
  3. Level Two Compliance
    1. All Level One requirements met
    2. Application meets RFC standards for SMTP integration
    3. Application is not used for non-critical (business or technical) monitoring processes
  4. Level Three Compliance
    1. Meets Level Two Compliance Requirements
    2. Is not used for monitoring at all
    3. Is not used for archival purposes
    4. Uses provided or approved libraries and applications for sending mail

Tracking of compliance levels, dates of achievement, and planned dates for increased compliance should be done per-application. This provides management and auditing with the ability to get a snapshot of the migration as well as a better understanding of the environment. Applications that cause email related incidents should have their Compliance Level (CL) lowered or modified by some measure. For example, an application that has caused two “serious” incidents or one major one may have a compliance modifier added or may have their CL lowered. One can think of this like movie ratings, specifically PG and PG-13 come to mind. An application that met the above Level Three compliance requirements would be labeled as “CL3”. If that MEA causes three serious incidents in the last 60 days it may be labeled as “CL3-C” with the C indicating the incident levels. Naturally the severity and frequency of incidents will be organization specific and the system should reflect this. Additionally periodic reviews or audits of the application for levels of compliance should be instituted to ensure an MEA remains compliant.

Services

When thinking of email and applications, most people are satisfied with thinking of the SMTP server as providing the service. Some see the application as providing the service. There is value to both of these thoughts. There is also value to be harvest by going beyond these concepts. Looking to the future of Enterprise Messaging, one thing is clear. Application generated messages will be a larger part of the infrastructure, or at least a larger portion of the message volume. These range from the aforementioned monitoring applications to marketing campaigns and distribution of company news and updates.

Custom Library Implementation

The Enterprise has several options here. The core of this concept is that the Enterprise’s Messaging Team provides a set of core libraries that behave as required. These libraries should be available for all popular languages such as C, C++, Python, PHP, Perl, Java, and a command line utility for system administration and unsupported language integration. The capabilities of this standard library could also implement EME policy requirements. For example, if a policy for MEA registration and tracking is in place, the library could implement the bits required for effecting this. If the MEA policies include rate limiting, the library could include the rate limiting as a reinforcement of the policy and to better prevent runaway apps.

Email as Service: Going Beyond SMTP

SMTP has shown itself to be a very scalable and surprisingly robust protocol. Unfortunately, there is an innumerable number of homegrown SMTP client libraries and toolkits out there. Many, perhaps even most, fail to implement enough of the standard to work properly or behave appropriately. As a result, EMEs are constantly dealing with the results: user support calls, broken applications sending hundreds to hundreds of thousands of the same message, and more. These situations can lead to much pain and unnecessary cost in the EME.

With SMTP’s robustness and scalability considered, it may be of value to an EME to offer Email sending as a service. This service could be provided via means of XML-RPC, HTTP “Web Services”, SOAP, HTTP application, or other network protocol. The purpose of this method is to centrally manage the code and thus the policies for MEA. This option may include the use of a standard library set to further assist application developers in participating in the EME and utilize proper behavior.

One of the goals of such a service would be to keep it simple for “small” uses. In so doing the non-developer can be provided a means for sending out targeted small email groupings. Additionally, it eases developer burden (whether a standard library is utilized or not) by providing a standard interface. An additional advantage of this option is that it can potentially reap rewards in the compliance field. The passage of Sarbanes-Oxley (SOX) has put a higher emphasis on business tools that have been “verified” to “do what they say they will”. The hodge-podge of EMAs and libraries provides a veritable nightmare for meeting process/software compliance/trust models. An Enterprise that can self-certify that the Applications introducing Email into the system can/have met a minimum criteria set can rest easier in both regulatory and system management matters.

Extending The Messaging Arena

In many existing EMEs, there is likely to be a heavy installed base of application generated email with the sole purpose of monitoring some process, result, condition, or other alert conditions such as environmental conditions monitoring. The installed base is likely to react unfavorably to a “new mandate” to stop doing this. The wise implementor will provide an alternative.

When the functionality of this use of email is analyzed, the reality is that they are a form of messaging. As such it is possible, even advantageous, to consider them as falling under the umbrella of Enterprise Messaging. By taking the standpoint that these messages fall under the Messaging team’s purview, the enterprise can leverage the existing knowledge and skill base. After all, an enterprise-level messaging implementation undoubtedly contains non-email based alerting and routing mechanisms.

Another advantage is that consistency across the Enterprise is achieved. Centralization of alerting system contact information will also enable rapid outage investigation to occur when needed. Additional benefits include the ability to compare and contrast levels of service, reliability, and scalability of alerting systems to minimize missed alerts. If we extend the concept of messaging to include alerts sent via pager, cell phone, email, instant messenger clients, and even delivering to a database for website viewing and trending, an Enterprise Messaging Team is positioned to extract and provide the maximum benefit to the client (in this case internal company clients) with the lowest development, integration, deployment and maintenance costs.

If the EME policy includes forbidding utilizing email for alerting mechanisms, the EM Team should provide alternatives. At a minimum they should be specific recommendations for various applications and service providers – to keep it simple for the end users – and ideally should include a standard tools offering. One option may include using email as a receipt mechanism. However, this would be a system separate from the primary email system of the organization. This would allow the clients (end uers and organizations) to transition off their existing system and/or provide a means for low-level email alerting without impact to the EME. If such a option is available, the emphasis should be on extremely robust and scalable architectures and agents.

Monitoring of non-IT managed assets has traditionally been left to the various users and organizations. It has been often the case that such a thought is abhorrent to both sides. However, the result has been a continual battle between the EME and the MEA owners and developers. Education may be the ideal solution, yet this is often difficult and ineffective. The proposed combination can eliminate the battle while improving overall service and availability.

Putting it Together, an Example Scenario

What follows is a hypothetical policy and implementation overview for a medium to large sized enterprise organization. Much of the technical detail is left out; the purpose is to illustrate the combination of the aforementioned options and policies into a coherent whole.

Company: ABWidgets Inc.
Employee Count: 25-35,000
Existing MEA Base 1200

Scenario:

The ABWidgets company has many manufacturing processes that utilize email for alerting. These include assembly line triggers, shipping and delivery triggers, inventory triggers, and other alerts. Daily non-MEA email accounts for some 1 million emails per day, while MEA messages account for approximately 2.5 million. The Messaging Team has traced the cause of mailbox unavailability as well as delay of critical email to the volume of MEA mail as well as the occasional faulty trigger and application tests. In order to correct this, the Messaging Team would like to isolate the MEA mail and limit it’s potential impact to the user level messaging systems. The first step the team takes is to assess the impact and potential impact of changing things. As a result of this an overall email AUP is created with an emphasis on MEA messaging. They come up with the following policy to be implemented over a nine month period.

The Policy

Purpose:

In order to minimize negative impact to the email systems used by ABWidgets’ employees, suppliers, and clients the following policy will be effective on 5 January 2007. A transitory period will be provided to migrate all application generated email to the new systems to minimize production impacts.

Policy

  1. All Mail Enabled Applications (MEA), defined as any automated email generation outside the human email client environment, must register with the Messaging Team. Minimum pieces of information will include but not be limited to:
    1. Application Title
    2. Application Purpose
    3. Application Owner
    4. Estimated Email Volume
    5. Owner Contact Information: contact information will be required for any hours the application will be in operation. If the application may run outside of normal business hours, contacts must be provided for these times.
    6. Originating machine information
      1. All hosts must be in DNS on Static IP addresses
      2. All hosts must have reverse DNS
    7. Owner’s organizational and operational manager information
  2. All MEA must send their mail using the mea.abwidgetco.com domain.
  3. All MEA must utilize DNS for configuration: no hard-coding of IP addresses or hostnames into application configuration
  4. All MEA messages will be considered low priority email. As such the following are best-case service levels to be expected:
    1. Message delays will not be considered problematic until they reach 45 minutes of delay.

    2. mea.abwidgetco.com is a 24×365 service with a minimum annual availability of 90%.

  5. All MEA mail will be subject to the following rate limits:

    1. No single recipient will be allowed to receive more than 60 messages per minute.

    2. No single receiving external domain will be allowed to receive more than 900 messages per minute.

  6. Applications exceeding the above rates will encounter intentional delays to keep mail injection rates to the specified limits.

  7. Applications exceeding 1000 messages to a single recipient in less than 10 minutes will be subject to quarantine. Exceeding a rate 3000 in 5 minutes will result in a blacklisting of the application by removing the MEA’s access to the servers.

    1. Applications in a Stage One violation (quarantined) will remain there for a minimum of 5 calendar days. See developer documentation for additional details.

    2. Applications in a Stage Two violation will remain blocked until management escalation is satisfied with the correction and prevention of future occurrences.

    3. Applications with 3 Stage One violations in a 30 day window will be escalated to a Stage Two level.

    4. Applications with 3 Stage Two level violations in a 30 day window will be banned for a minimum of 30 days. This is referred to as Stage Three.

    5. A second Stage Three occurrence in 120 days will result in senior level management getting involved.

  8. All currently deployed applications will be migrated to the provided Messaging Interface Library (MIL)

  9. All future applications must utilize the MIL

  10. Email is not to be used for Critical, Urgent, or regulatory alerting. Any applications or processes monitoring real-time systems, after-hours systems, and any urgent, critical, or regulatory alerting must use the ABWidget Alert Bus.

  11. Email is not to be used for archiving of campaign email. If archiving is required, the application must send mail to another application which stores the messages in a non-email environment.

     

The Messaging Team determines that this policy should alleviate their problem yet still provide application and process monitors with migration and long-term options.

The Server Side

In order to isolate the MEA mail, the Messaging Team chooses a bank of servers to handle the MEA mail system. The server breakdown is as follows:

  1. One server for the user interface.
  2. Two servers for mail relaying
  3. One server for Alert system capacity increase
  4. One server for the database and MEA Bus monitoring
  5. One server for MEA mailbox recipients (to be phased out)

The team utilizes a standard email server to validate application access, accept, store, and relay MEA email. The use of an additional server prevents the MEA Bus from overwhelming and complicating the internal IT monitoring systems. The mailbox server is a high performance server intended to serve as a migration path while removing this MEA mail from the production environment.

Migrating Existing Alerting Systems

In order to provide a migration path for the existing MEA base, a server is allocated to act as a proxy from email to the company pager network. This server is “protected” by the MEA Bus from flooding a pager and has a lower threshold applied to it given the smaller message queue of the chosen pager system. The MIL includes an interface to a series of internal alert and monitoring systems. This provides existing and new applications with a path to migrate way from using email.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s