Issue Details (XML | Word | Printable)

Key: COMMUNITY_EQUITY-166
Type: Sub-task Sub-task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: DmitryRyashchentsev
Reporter: max_wegmueller
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
community-equity
COMMUNITY_EQUITY-56

process equity computation offline / non blocking

Created: 08/Jun/09 01:35 PM   Updated: 10/Jun/10 01:57 PM  Due: 30/Sep/09   Resolved: 21/Jan/10 05:22 PM
Component/s: CeQ math
Affects Version/s: Milestone 1.3
Fix Version/s: Milestone 1.3

Time Tracking:
Not Specified

Tags:


 Description  « Hide

We can decrease the request response time by making CEQ math calculations off-line, i.e. after the response is sent in another thread.
We can use Message-driven bean to control the processed events queue and scheduling.
That will not take much time to implement. We just need to provide event container class, bean class, and client part.
The docs are:



max_wegmueller added a comment - 01/Jul/09 04:59 PM

moved to version 1.2,
as the testing and may be the coding is not doable within the scheduled release time of 1.1.


max_wegmueller added a comment - 09/Oct/09 03:40 PM

moving to milestone 1.3


max_wegmueller added a comment - 09/Oct/09 03:57 PM

the current eventService implementation is slow.

when I run the atom feed reader (AtomFeedProcessor.java), I have an average of about 1s processing time for each feed entry, if I disable event updates for info- and tag creation events, it goes down to about 100-200ms.

Some example timings:

AtomFeedProcessor.processSynEntry()

Duration------------------------|#]
Duration P: 11 ms|#] (get Person)
Duration I: 9 ms|#] (get Information by Key)
Duration CI: 143 ms|#] (add Information, set ACR)
Duration eventService.infoCreated: 82 ms|#] (called during add info from above)
Duration L: 1 ms|#] (update information attributes)
Duration D: 0 ms|#] (update information dates)
Duration T: 32 ms|#] (retrieve tags, create missing tags)
Duration TS: 566 ms|#] (add tags to information)
Duration eventService.tagAdded: 72 ms|#] (called during add tag/info from above)
Duration eventService.tagAdded: 203 ms|#] (called during add tag/info from above)
Duration eventService.tagAdded: 281 ms|#] (called during add tag/info from above)
Duration: 762 ms|#] (overall processing time)

Duration------------------------|#]
Duration P: 67 ms|#]
Duration I: 172 ms|#]
Duration CI: 257 ms|#]
Duration eventService.infoCreated: 102 ms|#]
Duration L: 1 ms|#]
Duration D: 0 ms|#]
Duration T: 34 ms|#]
Duration TS: 291 ms|#]
Duration eventService.tagAdded: 73 ms|#]
Duration eventService.tagAdded: 136 ms|#]
Duration eventService.tagAdded: 73 ms|#]
Duration: 822 ms|#]

Duration------------------------|#]
Duration P: 10 ms|#]
Duration I: 31 ms|#]
Duration CI: 469 ms|#]
Duration eventService.infoCreated: 257 ms|#]
Duration L: 2 ms|#]
Duration D: 1 ms|#]
Duration T: 107 ms|#]
Duration TS: 474 ms|#]
Duration eventService.tagAdded: 250 ms|#]
Duration eventService.tagAdded: 87 ms|#]
Duration eventService.tagAdded: 113 ms|#]
Duration: 1094 ms|#]

Can the eventservice performance be improved?


DmitryRyashchentsev added a comment - 12/Oct/09 02:26 PM

Yep. And that is exactly improvement we will get after this work be ready.


DmitryRyashchentsev added a comment - 30/Nov/09 05:49 PM

Checked in the first version.
Now there is no synchronization and monitoring of off-line processes.


DmitryRyashchentsev added a comment - 08/Dec/09 03:22 PM - edited

Some implementation details:
1. Off-line schedulling is implemented by JMS queue: jms/CEQDataManipulatorQueue (file ceq-ejb/setup/sun-resources.xml)

2. CEQ events are sent as messages to the queue in com.sun.ceq.session.math.EventService.

3. The message is transferred trough a number of levels:
DataManipulatorMessageProducer
DataManipulatorMessageProducerBean
DataManipulatorConsumer
DataManipulatorConsumerBean (MessageDriven bean)
DataManipulatorClassConsumer

DataManipulatorLogActionConsumer receives the message and implements a general actions logic (above math model)

4. DataManipulatorConsumerBean synchronizes messages - it is important that they are processes sequincely in the same order they did.
More complex synchrnization that allows parallel and group processing would be a big perf increasment.

5. DataManipulatorConsumerBean also isolates every action processing in a separate EJB transaction.

Problems:

1. CEQ math calculations are much slower then the other stuff, so strong activity grows the queue unlimitly.
JMS has timeout on messages processing time, default value is 1 minute.
Solution: it is really reasonable to have this timeout, may be we can allow to have it longer, but we need somehow to incrate priority of CEQ math calculations is the queue grows too much.

2. When I increased JMS queue message processing timeout, I got another problem.
Having so strong DB access from CEQ math thread (that is really without delays that web services usually have) we have problem with DB access from web services threads.
After a long time we can see DB access timeouts there.
Solution: better scheduling - If a transaction started it must have higher priority to allow it to complete.

TODO:

1. Fix the problems above, provide queue synchonization

2. Provice dayly recalculation schedulling

3. Move cache structure creation to on-line

4. Provide administration GUI

5. Provide optimization to process the queue in parallel.

6. Provide ability to execute queue processing on a number of machines


DmitryRyashchentsev added a comment - 29/Dec/09 08:38 AM - edited

Changed the async scheduling schema.
Now the queue elements are extracted from DB as a difference between LOG and ACTION_DATA records.

THE GENERAL SCHEMA::

1. All actions that affect to CEQ points (CEQ-action) are stored in LOG table.
2. After every CEQ-action JMS message is sent to calculation engine
3. Calculation engine manager is a single tone and it contain a flag that have an atomic state check and update, it indicate if calculation engine is IN-PROGRESS to calculate or not.
4. When calculation engine receives the message it checks the IN-PROGRESS flag:
4.1 If the flag is IN_PROGRESS state - nothing is doing, the message is processed almost immidiatly
4.2 If the engine is not busy:
4.2.1 The flag is set to IN-PROGRESS state
4.2.2 Get a limited size queue from DB and processed
4.2.3 The flag is reset from IN-PROGRESS state
4.2.4 Get the number of left queue elements from DB. If the number is > 0 a new JMS message to the engine is set.

PLUSES:

1. JMS messages are processed in predefined limited time - no timeouts.
2. High reliability - the queue is stored in DB
3. All threads are controlled by EJB container - fully EJB solution
4. Better slabability is provided.

MINUSES:

1. Two more small DB requests per a number of CEQ actions
2. In worst case we have in two time more JMS messages (with almost zero processing time)

PROBLEMS:

1. changeAuthor, deleteInfo, tags, aging correction are not implemented for this schema - that can lead to deadlocks

TODO:

1. Complete with changeAuthor, deleteInfo, tags, aging correction
2. Provide queue sync stuff
3. Provide async calculation admin control panel

4* Provide queue processing optimizations
5* Provide DB scalability (it is easy at least to use 2 DBs: one for objects, another for caches)
6* Provide better priorities control over web/feed and calculation threads


DmitryRyashchentsev added a comment - 29/Dec/09 03:29 PM

Moved changeAuthor, deleteInfo, and tags (TODO-1) to async part - that must fix deadlock/timeout problems (PROBLEMS-1)


DmitryRyashchentsev added a comment - 21/Jan/10 05:22 PM

Implemented, details are in COMMUNITY_EQUITY-405