Some implementation details:
1. Off-line schedulling is implemented by JMS queue: jms/CEQDataManipulatorQueue (file ceq-ejb/setup/sun-resources.xml)
2. CEQ events are sent as messages to the queue in com.sun.ceq.session.math.EventService.
3. The message is transferred trough a number of levels:
DataManipulatorConsumerBean (MessageDriven bean)
DataManipulatorLogActionConsumer receives the message and implements a general actions logic (above math model)
4. DataManipulatorConsumerBean synchronizes messages - it is important that they are processes sequincely in the same order they did.
More complex synchrnization that allows parallel and group processing would be a big perf increasment.
5. DataManipulatorConsumerBean also isolates every action processing in a separate EJB transaction.
1. CEQ math calculations are much slower then the other stuff, so strong activity grows the queue unlimitly.
JMS has timeout on messages processing time, default value is 1 minute.
Solution: it is really reasonable to have this timeout, may be we can allow to have it longer, but we need somehow to incrate priority of CEQ math calculations is the queue grows too much.
2. When I increased JMS queue message processing timeout, I got another problem.
Having so strong DB access from CEQ math thread (that is really without delays that web services usually have) we have problem with DB access from web services threads.
After a long time we can see DB access timeouts there.
Solution: better scheduling - If a transaction started it must have higher priority to allow it to complete.
1. Fix the problems above, provide queue synchonization
2. Provice dayly recalculation schedulling
3. Move cache structure creation to on-line
4. Provide administration GUI
5. Provide optimization to process the queue in parallel.
6. Provide ability to execute queue processing on a number of machines