Overview
IS (Integration Server) is an application to administrate, monitor and create a journal of computable work by using the tool WISM (Web Integration Server Manager).
Concepts
Mappings
The foundation of IS is to perform mappings, i.e. transform data from some structure in some protocol or format to some other structure in some protocol or format and in order to do that IS contains a lot of other helper functionality.
Item
Apart from basic server configuration, every configurable unit is called an Item.
Work
Some items can perform some kind of executable work and that work is logged as a Transaction.
Flow
Items that can perform work can direct work to other items (called a Target) that pick up and continue with the work and such a chain of work is called a Flow.
Queues
Targeting is done by the system that puts messages on a first-in-first-out-queue.
Items
Apart from basic server configuration, every configurable unit is called an Item.
Executables
Adapters
There are two kinds of Adapter and both are considered as a processing Item and that is various kinds of Input Adapter (typically reading from something such as a filesystem) and Output Adapter (typically writing to something such as a filesystem) and that message (Backupfile) is persisted by the system. All adapters are single threaded, though adapters run in parallel with each other.
Both kind of adapters can target multiple other targetable items such as executes, routings and output adapters.
Scheduled Tasks
A Scheduled Task runs a program when triggered without the server knowing anything about a message.
Executes
An Execute can, for historical reasons, perform 0..2 computable units of work called a Process and a Mapping or it can just be used as a fan out because it can target other executes, routings or output adapters.
Routings
A Routing determines what executable items to target depending on content in current message and it can (of historical reasons) only target executes.
Holders
A Holder is configurable item that contains some data such as a program (typically something developed with DS), certificates, etc. A Holder is can be referenced from several items.
Monitoring
Event Policies
An Event Policy is an item that can be used to filter certain errors from the system and create messages sent to other items, typically for error report handling. The system is usually shipped with a default policy that is used for every item that does not have one explicitly configured.
Metrics
There are items that can be configured to monitor the throughput in the system, such as Queue Metrics (see how much is waiting to be processes) and Process Metrics (see how much is currently processing).
Searches
These are items that can be configured to perform (saved) searches in the database either to just view the result immediately, or to report an error and thus be a part of the error notification mechanism.
Processing
Schedules
Various executable items can, apart from being triggered by a message, be triggered to do their work by a schedule with a certain interval. The interval can be can be configured to just operate during certain time periods and on a defined interval (eg every 30 minutes on Sunday evening). The behaviour of the triggered item varies slightly, for historical reasons, depending of the type of item.
Settings
Various items (adapters and executes) do have detailed settings and those settings could be overridden in runtime (output adapters can even have settings overridden by data produced among processing of messages).
HTTP Listeners & Conversions
IS can also perform light weight conversions without monitoring and without keeping a journal. That is configured by a (at least one) HTTP Listener and then possible multiple HTTP Conversion per listener.
Details
Scheduled vs Event
The default mode for every executables is event based, but all executables can refer to a inobiz_schedule. A schedule can be referred to by multiple executables. If the executable does not refer to a schedule, it is called event based and is triggered in other ways. Due to historical reasons, various kinds of executables work slightly differently.
Scheduled items can either triggered explicitly by doing a run-now-call (via WISM or via an HTTP-call) or letting an inobiz_schedule trigger the queue.
Scheduled Task
The only way to make a inobiz_scheduled_task do its task is by putting an (empty) message in its queue to trigger its unit of work.
Event Input
Certain input adapters can find source messages by waiting (e.g. inobiz_folder_in) and as soon as a message arrives it will create one input (parent) Transaction per message. The message will be duplicated to all its targets put to their queues.
Scheduled Input
Certain adapters can be scheduled and will look for source messages as soon as possible when triggered (e.g. inobiz_email_in). It will be one input (parent) Transaction created per batch of messages. The messages will be duplicated to all its targets and put to their queues.
Event Output
All output adapters can be event based which means that the adapter constantly is monitoring its message queue and will process messages as soon as possible.
Scheduled Output
Almost all adapters can be scheduled based (though it doesn't make sense to have synchronous request reply patterns scheduled to some adapters). When an output adapter gets triggered it works until the queue is empty and then wait for the next triggering.
Event Execute
All executes can be event based and is triggered by a message from the queue and the message is handled to the possible Process and then to the possible Mapping and then to its Targets.
Scheduled Execute
All executes can be scheduled based (typically an SQL mapping) but what happens then is that an empty message is added to the queue and the execute is triggered by that.
Dependencies
Items may refer to other items and those has to exist as long as they are referenced.
Tuning
There are three ways of tuning the system for performance and throughput by configuring various items
Adapters and processes
Each adapter can be configured to run in a separate (operating system) process. To have them in separate processes might improve some performance but it also consumes more resources (sockets, memory, etc) in the system.
Process Groups and processes
Every tasks, routings and executable are running within a Process Group in order to balance and scale its throughput. The system is usually shipped with a default *group* that is used for every item that is not explicitly configured otherwise but the user can create their own inobiz_process_group and let certain executables participate in that group. Each Process Group may have 0..N pipelines. Each pipeline is an separate (operating system) process.
Executes and parallelism
All executes are processed within a Pipeline in a process group that could have multiple pipelines, but the executes does have a Instances parameter so they can be balanced within a Process Group. If Instances is set to 1 (one) no parallel processing (i.e. a singleton) can occur for that particular inobiz_execute.
Note: an inobiz_execute that is targeted from an inobiz_routing is executed in that context and if it is targeted from multiple inobiz_routing or also runs as an standalone *mapping* that *singleton* behaviour is of course set aside.
If the Process Group is configured to just have one Pipeline all of the participating executes behaves like singleons, i.e. no messages could be processed in parallel.
0 Comments