Issue Details (XML | Word | Printable)

Key: COMMUNITY_EQUITY-424
Type: Sub-task Sub-task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Misu
Reporter: PRE
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
community-equity
COMMUNITY_EQUITY-407

Implemented more granular transaction model for feed

Created: 18/Jan/10 05:15 PM   Updated: 15/Feb/10 10:26 AM   Resolved: 26/Jan/10 10:27 AM
Component/s: None
Affects Version/s: Milestone 1.2
Fix Version/s: Milestone 1.3

Time Tracking:
Not Specified

File Attachments: None
Image Attachments:

1. db.gif
(49 kB)

Tags:


 Description  « Hide
  • feed entry transaction for feed service


PRE added a comment - 19/Jan/10 05:54 PM

If you get JMS start errors make following change
in
/Applications/NetBeans/SUNWappserver/domains/domain1/config/domain.xml

I changed the port to above 80xxx e.g.
<jms-host admin-password="admin" admin-user-name="admin" host="Reiser-2-0.local" name="default_JMS_host" port="8680"/>

and restarted the server


Misu added a comment - 20/Jan/10 10:01 AM - edited

This solution may works but after I take a look on the code I think that this is not the best solution.
The main distavatage is the fact that the actual implementation is memory based, large feeds can bring problems.
The best solution loos like is based on producer consumer pattern. More precisely : The producer component reads the feed entry and send messages for each entry. The consumer does the entire IO operations (in individual transactions). The JMS provider can be tuned to bring maximal performance.


Misu added a comment - 21/Jan/10 03:25 PM

I add a unit test (named ExtremeFeedTest) to the ceq-unit project. This is a stress test for the feed service and its purpose is to show weakness in the FeedService.
If the teest runs long enough (10 minuttes) on my machine the frist exception appears.


PRE added a comment - 21/Jan/10 07:27 PM

Mihai - I have some strange behavior now.

1. I import a very large feed _> works
2. add some other feeds -> works
3. restart netbeans -> the systen continuously process feed entries forever
4. I delete all feed entries and restart Netbeans - does not process anything
5. Add a new feed (e.g.http://identi.ca/api/statuses/public_timeline.atom) -> continuously process feed entries forever

Also I can see that the lastest feed entries are in the database (INFORMATION table) but there are not displayed in the listing but the LISTING counter is correct

Question to Dima - If the import gets interrupted how to you sync the calculation with the last entry imported by the feed after a restart?


PRE added a comment - 21/Jan/10 07:29 PM

get intermittent error messages in the log

bind => [36070]
INSERT INTO CEQACTION (LOG_ID, CONTROLLED_ACTION, INFORMATION_ID, PERSON_ID) VALUES (?, ?, ?, ?)
bind => [36070, 0, 0, 2091]
Local Exception Stack:
Exception [TOPLINK-4002] (Oracle TopLink Essentials - 2.1 (Build b60e-fcs (12/23/2008))): oracle.toplink.essentials.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`ceq/CEQACTION`, CONSTRAINT `CEQACTION_ibfk_1` FOREIGN KEY (`INFORMATION_ID`) REFERENCES `INFORMATION_CEQ` (`INFORMATION_ID`))
Error Code: 1452
Call: INSERT INTO CEQACTION (LOG_ID, CONTROLLED_ACTION, INFORMATION_ID, PERSON_ID) VALUES (?, ?, ?, ?)
bind => [36070, 0, 0, 2091]
Query: InsertObjectQuery(com.sun.ceq.entity.math.CEQAction@10924348)
at oracle.toplink.essentials.exceptions.DatabaseException.sqlException(DatabaseException.java:311)
at oracle.toplink.essentials.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:654)
at oracle.toplink.essentials.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:703)
at oracle.toplink.essentials.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:492)
at oracle.toplink.essentials.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:452)
at oracle.toplink.essentials.internal.sessions.AbstractSession.executeCall(AbstractSession.java:690)
at oracle.toplink.essentials.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
at oracle.toplink.essentials.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:214)
at oracle.toplink.essentials.internal.queryframework.DatasourceCallQueryMechanism.insertObject(DatasourceCallQueryMechanism.java:346)
at oracle.toplink.essentials.internal.queryframework.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:191)
at oracle.toplink.essentials.internal.queryframework.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:205)
at oracle.toplink.essentials.internal.queryframework.DatabaseQueryMechanism.insertObjectForWrite(DatabaseQueryMechanism.java:564)
at oracle.toplink.essentials.queryframework.InsertObjectQuery.executeCommit(InsertObjectQuery.java:89)
at oracle.toplink.essentials.internal.queryframework.DatabaseQueryMechanism.performUserDefinedWrite(DatabaseQueryMechanism.java:760)
at oracle.toplink.essentials.internal.queryframework.DatabaseQueryMechanism.performUserDefinedInsert(DatabaseQueryMechanism.java:724)
at oracle.toplink.essentials.internal.queryframework.DatabaseQueryMechanism.insertObjectForWriteWithChangeSet(DatabaseQueryMechanism.java:602)
at oracle.toplink.essentials.queryframework.WriteObjectQuery.executeCommitWithChangeSet(WriteObjectQuery.java:162)
at oracle.toplink.essentials.internal.queryframework.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:390)
at oracle.toplink.essentials.queryframework.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:109)
at oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:628)
at oracle.toplink.essentials.queryframework.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:555)
at oracle.toplink.essentials.queryframework.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:138)
at oracle.toplink.essentials.queryframework.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:110)
at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2248)
at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:909)
at oracle.toplink.essentials.internal.sessions.CommitManager.commitNewObjectsForClassWithChangeSet(CommitManager.java:269)
at oracle.toplink.essentials.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:190)
at oracle.toplink.essentials.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:2668)
at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1044)
at oracle.toplink.essentials.internal.ejb.cmp3.base.RepeatableWriteUnitOfWork.commitToDatabase(RepeatableWriteUnitOfWork.java:496)
at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithPreBuiltChangeSet(UnitOfWorkImpl.java:1160)
at oracle.toplink.essentials.internal.ejb.cmp3.base.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:368)
at oracle.toplink.essentials.internal.ejb.cmp3.base.EntityManagerImpl.flush(EntityManagerImpl.java:366)
at com.sun.enterprise.util.EntityManagerWrapper.f
---Log File Rotated--
SELECT ID, TQK, TQB, TAG_ID, PQ, TQ, PQK, Author_ID, PQB, USAGE_COUNT FROM PERSON_TAG_CEQ WHERE (ID = ?)
bind => [10997]
SELECT ACTION_LOG_ID FROM ACTION_DATA WHERE (ACTION_LOG_ID = ?)
bind => [36030]
INSERT INTO ACTION_DATA (ACTION_LOG_ID, ACTION_PERSON_CQK, ACTION_PERSON_CQB, ACTION_K_PQ, ACTION_AGING_IQ, ACTION_B_PQ, ACTION_AGING_PQ, ACTION_INFORMATION_B, ACTION_EXP_INFO_K, ACTION_PERSON_PQB, ACTION_EXP_INFO_B, ACTION_B_IQ, ACTION_EXP_PERSON_PQK, ACTION_PERSON_PQK, ACTION_EXP_PERSON_PQB, ACTION_INFORMATION_K, ACTION_EXP_PERSON_CQK, ACTION_K_IQ, ACTION_EXP_PERSON_CQB) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
bind => [36030, 16038, 17817276, 216, 2010-01-24 19:28:29.0, 241920, 2010-01-24 19:28:29.0, 471744, null, 8200260, null, 241920, null, 7308, null, 432, null, 216, null]
UPDATE CEQACTION SET LOG_ID = ? WHERE (((CONTROLLED_ACTION = ?) AND (INFORMATION_ID = ?)) AND (PERSON_ID = ?))
bind => [36030, 0, 6005, 2091]
consume: Ok
consume: log=36031 action=0 info=0 person=48 date=1260309169000 parameter1=3
getOrCreateEntityPerson: person exist:48
SELECT INFORMATION_ID, INFORMATION_IQB, INFORMATION_PQ, AUTHOR_ID, INFORMATION_PQK, INFORMATION_IQK, INFORMATION_PQB, INFORMATION_IQ, INFORMATION_STATUS, INFORMATION_TYPE_ID FROM INFORMATION_CEQ WHERE (INFORMATION_ID = ?)


PRE added a comment - 21/Jan/10 07:36 PM - edited

another obeservation.
Table INFORMATION has 18023 records ..
Table INFORMATION_CEQ has 6049 records

second last record has informatonID 0 .. see attached screenshot


Misu added a comment - 24/Jan/10 10:08 PM

Hi

About Peter comment MySQLIntegrityConstraintViolationException

I just look in the mysql script and I found the :

CREATE TABLE CEQACTION (
CONTROLLED_ACTION INT NOT NULL,
INFORMATION_ID INT NOT NULL,
PERSON_ID INT NOT NULL,
LOG_ID INT NOT NULL,
FOREIGN KEY (INFORMATION_ID) REFERENCES INFORMATION_CEQ (INFORMATION_ID),
FOREIGN KEY (PERSON_ID) REFERENCES PERSON_CEQ (PERSON_ID)
– FOREIGN KEY (LOG_ID) REFERENCES ACTION_DATA (ACTION_LOG_ID)
);

on my script is on the line 548

this script part builds a a table and sets a relation with other tables (INFORMATION_ID and PERSON_ID). This is perfect legal, but, the corresponding java model does not supports this relation. The next code snippet shows the CEQACTION properties.

@Id
@Column(name = "LOG_ID") private int nLogID;
@Column(name = "CONTROLLED_ACTION") private int nControledAction;
@Column(name = "INFORMATION_ID") private int nInfoID;
@Column(name = "PERSON_ID") private int nPersonID;

As si easy to see here - no relation is defined (none of the relation annotation is used). In my opinion this can cause the problem early described by Peter. More precisely, the ORM framework is responsible for the generating the primary key and the foreign key also- so the nInfoID and the nPersonID integer properties must be replaced with the proper relation. The actual implementation gets this responsibility from the ORM framework and replace it with its own code.

The solution can look like this :

E.G.

@OneToOne
@JoinColumn(name="INFORMATION_ID")
Information information;

@OneToOne
@JoinColumn(name="PERSON_ID")
Person person;

For the moment I am not sure if the relation must be uni or bidirectional.

Because this can cause some side effects I will try to fix this in a separate branch. I also think that this problem is a different bug than one described in this JIRA report (424) and I will generate an other one.


DmitryRyashchentsev added a comment - 24/Jan/10 11:51 PM

Fixed problem with INFORMATION_ID=0, the problem was with deattaching of removed info objects.

There are also some EJB3 effects that causes exceptions, I created a new ticket for them COMMUNITY_EQUTY-426.
I workarounded some of the "effects", but not all, I need a help.


DmitryRyashchentsev added a comment - 25/Jan/10 12:02 AM

The problem with MySQLIntegrityConstraintViolationException is deeper here, and I already fixed.


PRE added a comment - 26/Jan/10 10:27 AM

resolved - for future tuning we should open a new ticket


PRE added a comment - 15/Feb/10 10:26 AM

closed all ticket for Milestone Release 1.3