Difference between revisions of "Internet Mail Archive System"

From PresenceWiki
Jump to: navigation, search
 
(86 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
{{Tutorials}}
 
{{Tutorials}}
  
<h2>Overview</h2>
+
<h1>Overview</h1>
  
 
This document describes the IMAS system and how it was created using Presence.
 
This document describes the IMAS system and how it was created using Presence.
  
The Imas System currently performs 3 functions:-
+
The IMAS System currently performs 3 functions:-
  
The Mail Service connects to a POP(email) account and downloads attachments, based on conditions defined under the rules menu.
+
The <b>Mail Service</b> connects to an email account and downloads the attachments, based on conditions defined under the rules menu.
 +
It will place each attachment in a separate folder in .\res\httpdocs\imas\scan\BasicDetach
  
The Rule Service will then copy each attachment to a Network or AS400 location or place it in a folder to be processed by a task service.
+
The <b>Rule Service</b> will then process each of these directories by moving each attachment to a Network or AS400 location,
 +
or by running a task that will access the file through the variable ${localfile}.
  
The Task Service will then process any attachments that need processing, custom tasks can be created for this purpose.
+
The <b>Task Service</b> can also be used to run extra user defined tasks if necessary.
  
<h2>Specific DAS Solution</h2>
+
<h1>Presence Tasks</h1>
  
   
+
Queued Processes
* The user can use their windows username and password.
+
  Queue Scan POP
* The httpsession variable is used so that the user only needs to login once.
+
Queue Rule Action
* The username is stored against the ip address, so that it will only need to be entered once for a machine.
+
Queue HouseKeeper - This is used to clean up the logs
* A session will automatically expire when no new requests have been made for a set time.
+
Queue Run Custom Tasks - This is used to run any tasks defined in imas.customtasks
 +
note We will also need to queue any tasks run by the rules
  
<h2>Importing</h2>
+
On-Demand Processes
 +
OnDemand MainPage (alias imas.mainPage)
  
You can download the tasks for this example from:-
+
<h1>General IMAS Solution</h1>
http://www.international-presence.com/wikidocs/samples/On_Demand_Windows_Login.xml
+
  
Once imported they should look like:-
+
Here we will take a look at the general IMAS solution and show how it can Scan attachments from emails and upload them to either an AS400 Folder a Network Folder or run a task.
  
http://www.international-presence.com/images/docs/userlogin/tasks.png
+
The first screen here is the IMAS config screen.
  
To embed the task, we just place the <b>Check Login Sub Task</b> before our normal OnDemand task page as you can see below in <b>Simple Task To Confirm Action</b>.
+
IMAS is located in the Presence .\res\httpdocs\imas directory.
 +
 
 +
So with the both the Presence HTTP Server and the IMAS tasks running, we can go to:-
 +
 
 +
http://localhost/imas
 +
 
 +
<hr/>
 +
 
 +
<b>The first screen we will see is the IMAS config screen.</b>
 +
 
 +
http://www.international-presence.com/images/imas/editconfig.png
 +
 
 +
Here we can define an administrator email, who will get notified of any of exceptions that occur.
 +
 
 +
As each attachment is processed, the name of the folder that gets created is incremented.
 +
 
 +
We can also see the next folder number as well as define the name of the folder that failed attachments end up in.
 +
 
 +
<hr/>
 +
 
 +
<b>Under the AS400 screen we can add and edit AS400 connections.</b>
 +
 
 +
http://www.international-presence.com/images/imas/editas400.png
 +
 
 +
<hr/>
 +
 
 +
<b>Similarly the Email screen allows us to configure which POP accounts we need to be read.</b>
 +
 
 +
http://www.international-presence.com/images/imas/editemail.png
 +
 
 +
<hr/>
 +
 
 +
<b>The rule screen lets us define what happens when we query a pop account.</b>
 +
 
 +
http://www.international-presence.com/images/imas/editrule.png
 +
 
 +
Each rule is associated with one email account as defined in the previous tabs.
 +
 
 +
The action to perform can be one of:-
 +
 
 +
* Upload To Network Folder
 +
 
 +
* Upload To An AS400 (as defined in the AS400 menu)
 +
 
 +
* Run A Task
 +
 
 +
Below the action type, we specify the Task or Network/AS400 folder.
 +
 
 +
The From Address, allows us to only process emails from a specific address (leave brank to process all the emails in this account).
 +
 
 +
In Attachment we can specify to only process emails that have a certain type of attachment.
 +
 
 +
In Filename we can specify to only process emails that have a certain filename match.
 +
 
 +
Subject allows us to only process emails that match this subject.
 +
 
 +
Convert XLS to CSV will convert any XLS attachments to CSV attachments.
 +
 
 +
<hr/>
 +
 
 +
<b>This Mail Scanner Task can be stopped and started using the Mail Service Screen.</b>
 +
 
 +
http://www.international-presence.com/images/imas/mailservice.png
 +
 
 +
When the attachments are taken from each email, a folder is created for each in the BasicDetach folder:-
 +
 
 +
http://www.international-presence.com/images/imas/basicdetach.png
 +
 
 +
Note that there are 2 files in each folder as one is the attachment and the other is the trigger folder (imasready).
 +
 
 +
<hr/>
 +
 
 +
<b>The Rule Service performs the required action on each of the folders created from the Mail Service.</b>
 +
 
 +
http://www.international-presence.com/images/imas/ruleservice.png
 +
 
 +
<hr/>
 +
 
 +
<b>We can find a list of any emails that failed to be processed in the Email Exceptions Page</b>
 +
 
 +
http://www.international-presence.com/images/imas/mailexceptions.png
 +
 
 +
From here we can email ourselves a copy of the report.
 +
 
 +
<hr/>
 +
 
 +
<b>We can find a list of Rules that failed to run in the Rule Exceptions Pages</b>
 +
 
 +
http://www.international-presence.com/images/imas/ruleexceptions.png
 +
 
 +
From here we can email ourselves a copy of the report.
 +
 
 +
<h1>Adding A Custom Task To The Solution</h1>
 +
 
 +
Here, we will study the solution delivered to DAS to give us an example of IMAS in action using Custom Tasks.
 +
 
 +
The requirement for DAS was to scan an email account for emails containting attachments that can be one of four types:-
 +
 
 +
* HomeFixed
 +
* HomeDelimitted
 +
* MotorFixed
 +
* MoterDelimitted
 +
 
 +
Attachments, can also come through as zipped, in which case we needed to get Presence to unzip them so that it can then treat them as standard attachments.
 +
 
 +
 
 +
As an example we will send 1 email that contains a zip file containing one of each file (HomeFixed,HomeDelimitted
 +
MotorFixed,MoterDelimitted) as well as an individual file MotorDelimitted.txt and another file Wrong.txt which contains invalid data.
 +
 
 +
When the Mailservice runs each of these files will be detached and placed in:-
 +
 
 +
res\imas\scan\basicdetach folder, each in a separate folder.
 +
 
 +
In this case we have configured the rule to tun the task <b>Move Folder Into Bucket</b>.
 +
 
 +
This task actually opens each file and decides whether it is:-
 +
 
 +
* HomeFixed
 +
* HomeDelimitted
 +
* MotorFixed
 +
* MoterDelimitted
 +
 
 +
And places it in the corresponding scan directory. If it can't decide it places it in the Unknown directory:-
 +
 
 +
http://www.international-presence.com/images/imas/buckets.png
 +
 
 +
We then need to process each of the 4 directories, to do this we added a Custom Task, <b>Process Bucket Directories</b>
 +
 
 +
The Custom Tasks need to be added to the table imas.customtasks, SQL helper nodes for this can be found in the Instructions task.
 +
 
 +
The SQL statement <b>Select * from imas.customtasks</b> should produce the following result:-
 +
 
 +
http://www.international-presence.com/images/imas/customtasksselect.png
 +
 
 +
For each type of directory, it will populate a table on the as400 (from the file), and then run a program call on the as400,
 +
after which the file will be deleted.
 +
 
 +
<hr/>
 +
 
 +
<b>To run this Custom Task, we need to go to the task service, select it from the combo, then hit
 +
start.</b>
 +
 
 +
http://www.international-presence.com/images/imas/taskservice.png
 +
 
 +
<hr/>
 +
<b>Once Again the exceptions for this task can be found in the Task Exceptions screen.</b>
 +
 
 +
This shows the as400 table (which should be empty) and any unprocessed folders.
 +
 
 +
http://www.international-presence.com/images/imas/taskexceptions.png
 +
 
 +
<h1>Installing/Moving IMAS to a new location</h1>
 +
 
 +
*Import the IMAS.xml into a new category on your Presence System.
 +
*Copy the imas folder from the Presence \res\httpdocs\imas directory to \res\httpdocs\imas on the new system
 +
*Run the task, Instructions from the Instructions Category.
 +
This will essentially, create the IMAS.CustomTasks table and insert the "Process Bucket Directories" item into it.
 +
*Drag the task "OnDemand MainPage" from the Create Panels category and then click on start.
 +
http://www.international-presence.com/images/imas/ondemandtasks.png
 +
*Drag the following tasks from the Runners category into the Queued tasks table and make sure they are running.
 +
http://www.international-presence.com/images/imas/queuedtasks.png
 +
*If you now go to http://localhost/imas, you should see the IMAS entry screen.
 +
If this does not work then you need to check that the HTTPServer is running along with the Presence Server by launching the Hearbeat client.

Latest revision as of 14:38, 24 May 2012

Tutorial Pages > Internet Mail Archive System

Overview

This document describes the IMAS system and how it was created using Presence.

The IMAS System currently performs 3 functions:-

The Mail Service connects to an email account and downloads the attachments, based on conditions defined under the rules menu.
It will place each attachment in a separate folder in .\res\httpdocs\imas\scan\BasicDetach
The Rule Service will then process each of these directories by moving each attachment to a Network or AS400 location, 
or by running a task that will access the file through the variable ${localfile}.
The Task Service can also be used to run extra user defined tasks if necessary.

Presence Tasks

Queued Processes

Queue Scan POP
Queue Rule Action
Queue HouseKeeper - This is used to clean up the logs
Queue Run Custom Tasks - This is used to run any tasks defined in imas.customtasks
note We will also need to queue any tasks run by the rules

On-Demand Processes

OnDemand MainPage (alias imas.mainPage)

General IMAS Solution

Here we will take a look at the general IMAS solution and show how it can Scan attachments from emails and upload them to either an AS400 Folder a Network Folder or run a task.

The first screen here is the IMAS config screen.

IMAS is located in the Presence .\res\httpdocs\imas directory.

So with the both the Presence HTTP Server and the IMAS tasks running, we can go to:-

http://localhost/imas


The first screen we will see is the IMAS config screen.

http://www.international-presence.com/images/imas/editconfig.png

Here we can define an administrator email, who will get notified of any of exceptions that occur.

As each attachment is processed, the name of the folder that gets created is incremented.

We can also see the next folder number as well as define the name of the folder that failed attachments end up in.


Under the AS400 screen we can add and edit AS400 connections.

http://www.international-presence.com/images/imas/editas400.png


Similarly the Email screen allows us to configure which POP accounts we need to be read.

http://www.international-presence.com/images/imas/editemail.png


The rule screen lets us define what happens when we query a pop account.

http://www.international-presence.com/images/imas/editrule.png

Each rule is associated with one email account as defined in the previous tabs.

The action to perform can be one of:-

  • Upload To Network Folder
  • Upload To An AS400 (as defined in the AS400 menu)
  • Run A Task

Below the action type, we specify the Task or Network/AS400 folder.

The From Address, allows us to only process emails from a specific address (leave brank to process all the emails in this account).

In Attachment we can specify to only process emails that have a certain type of attachment.

In Filename we can specify to only process emails that have a certain filename match.

Subject allows us to only process emails that match this subject.

Convert XLS to CSV will convert any XLS attachments to CSV attachments.


This Mail Scanner Task can be stopped and started using the Mail Service Screen.

http://www.international-presence.com/images/imas/mailservice.png

When the attachments are taken from each email, a folder is created for each in the BasicDetach folder:-

http://www.international-presence.com/images/imas/basicdetach.png

Note that there are 2 files in each folder as one is the attachment and the other is the trigger folder (imasready).


The Rule Service performs the required action on each of the folders created from the Mail Service.

http://www.international-presence.com/images/imas/ruleservice.png


We can find a list of any emails that failed to be processed in the Email Exceptions Page

http://www.international-presence.com/images/imas/mailexceptions.png

From here we can email ourselves a copy of the report.


We can find a list of Rules that failed to run in the Rule Exceptions Pages

http://www.international-presence.com/images/imas/ruleexceptions.png

From here we can email ourselves a copy of the report.

Adding A Custom Task To The Solution

Here, we will study the solution delivered to DAS to give us an example of IMAS in action using Custom Tasks.

The requirement for DAS was to scan an email account for emails containting attachments that can be one of four types:-

  • HomeFixed
  • HomeDelimitted
  • MotorFixed
  • MoterDelimitted

Attachments, can also come through as zipped, in which case we needed to get Presence to unzip them so that it can then treat them as standard attachments.


As an example we will send 1 email that contains a zip file containing one of each file (HomeFixed,HomeDelimitted MotorFixed,MoterDelimitted) as well as an individual file MotorDelimitted.txt and another file Wrong.txt which contains invalid data.

When the Mailservice runs each of these files will be detached and placed in:-

res\imas\scan\basicdetach folder, each in a separate folder.

In this case we have configured the rule to tun the task Move Folder Into Bucket.

This task actually opens each file and decides whether it is:-

  • HomeFixed
  • HomeDelimitted
  • MotorFixed
  • MoterDelimitted

And places it in the corresponding scan directory. If it can't decide it places it in the Unknown directory:-

http://www.international-presence.com/images/imas/buckets.png

We then need to process each of the 4 directories, to do this we added a Custom Task, Process Bucket Directories

The Custom Tasks need to be added to the table imas.customtasks, SQL helper nodes for this can be found in the Instructions task.

The SQL statement Select * from imas.customtasks should produce the following result:-

http://www.international-presence.com/images/imas/customtasksselect.png

For each type of directory, it will populate a table on the as400 (from the file), and then run a program call on the as400, after which the file will be deleted.


To run this Custom Task, we need to go to the task service, select it from the combo, then hit start.

http://www.international-presence.com/images/imas/taskservice.png


Once Again the exceptions for this task can be found in the Task Exceptions screen.

This shows the as400 table (which should be empty) and any unprocessed folders.

http://www.international-presence.com/images/imas/taskexceptions.png

Installing/Moving IMAS to a new location

  • Import the IMAS.xml into a new category on your Presence System.
  • Copy the imas folder from the Presence \res\httpdocs\imas directory to \res\httpdocs\imas on the new system
  • Run the task, Instructions from the Instructions Category.

This will essentially, create the IMAS.CustomTasks table and insert the "Process Bucket Directories" item into it.

  • Drag the task "OnDemand MainPage" from the Create Panels category and then click on start.

http://www.international-presence.com/images/imas/ondemandtasks.png

  • Drag the following tasks from the Runners category into the Queued tasks table and make sure they are running.

http://www.international-presence.com/images/imas/queuedtasks.png

If this does not work then you need to check that the HTTPServer is running along with the Presence Server by launching the Hearbeat client.