Alexander Koller
|
Posted: February 04, 2010 21:01 by Alexander Koller
|
|
Hi all, we are almost ready to start the public evaluation phase for GIVE-2. We originally planned to start at the beginning of February, but haven't been able to get everything finished in time. We now plan to start just after the ACL deadline (figuring that some of you may appreciate it), in the following way. On February 16, we will distribute the evaluation worlds to all participants. As we said earlier, these worlds will be similar to the worlds you've already seen in size and complexity. They may contain some types of objects that you haven't seen yet. However, you will have a few days to adapt your lexicons and make sure that your NLG systems work properly with these worlds. We're counting on you to not abuse this period by tailoring your systems specifically to the evaluation worlds. On February 22, we will start the public evaluation. At this point, we will set up a public Matchmaker instance which points to all your NLG systems, and we will invite users to come and play the GIVE game. We will continue to collect data until the end of April or maybe early May. The way the GIVE Challenge is set up, we would like you to run your NLG systems on servers at your own institution (or wherever else you like). We will not be able to provide the computing power. Once you have set up your NLG systems, please send us the hostname and the port number that you specified in your NLG system configuration file; we will then enter this address into our Matchmaker. Please be aware that your NLG system should be running 24/7 once the evaluation starts, and that the necessary ports need to be open so clients can connect to the server. The ports an NLG system uses are documented in the Wiki. We would like to ask you to send us your NLG server configuration details by February 16. This makes it explicit for us who is actually participating in GIVE-2 and who isn't, and gives you and us enough time to debug your network connection before the evaluation. We will only send the evaluation worlds to participants who have signed up by this deadline, so make sure we hear from you in time. I'm planning to release GIVE 2.2.2 tomorrow morning. It fixes a lot of the bugs that you guys have been complaining about, such as the X-ray vision bug and the bug where a button is part of a region outside the room, and implements the "back to base" task we outlined in evaluation document. I made some pretty big changes to the underlying network protocol for this, so I probably introduced some fresh bugs. Please let me know about them asap, and we'll get them fixed. If you have any questions or comments, please feel free to post them here. Happy implementing, Alexander. |
GIVE: Timeline
by: niels_s
|
niels_s
|
Posted: February 12, 2010 17:21 by niels_s
|
|
Hi, we are not sure if we are actually going to get the server online by the 16. (lots of unexpected administrative regulations for getting a connection through the firewall...), would it be okay if we just register with the server info we are hoping to get and then get it running as soon as we get through? Niels |
|
dipankar.dipnil2005
|
Posted: February 17, 2010 08:17 by dipankar.dipnil2005
|
|
I am extremely sorry to ask question that seems easy for others. Time is already running over our heads.Late for ACL too. As we are not familiar so much with these networking facilities, hence, the questions are Actually we are buying a DNS name with 100 MB space and a port. Is it possible to run the system? What are the items that we need to place in the server? What are the items that we need to send for evaluation? We have updated the server code of example nlg server and run it as the example nlgserver runs. will it work for the evaluation? If not, what else we have to do? My mail id is <dipankar.dipnil2005@gmail.com> Kindly send me the detail process of uploading the system and networking scenario. |
|
Alexander Koller
|
Posted: February 18, 2010 10:31 by Alexander Koller
|
|
Hi, the documentation for setting up the server can be found here: http://kenai.com/projects/give-challenge/pages/Manual#ConfiguringtheNLG_server You can ignore everything that has to do with "matchmaker" and "databases". All you need to do is run your NLG server as usual. Then you should double-check that the ports on which your NLG server listens are accessible from the outside world (and not e.g. blocked by a firewall) and send me the hostname and port number by email. We will then send some users to your NLG server automatically. All the data collection and evaluation happens on our end. I would expect that you need less than 100 MB for this, and you also don't need a domain name of your own, because nobody except for me ever sees your hostname anyway. However, you should hurry up getting your system to run, because I will start the public evaluation on Monday, and will not accept NLG servers that are not public by that time. Cheers, Alexander. |
|
Alexander Koller
|
Posted: February 05, 2010 13:13 by Alexander Koller
|
|
I'm sorry, I don't understand the question. You need to send me the network connection data for your server by Feb 16, and then keep the server running until the end of April. You may restart the server between Feb 16 and 22 to fix the lexicon and such, but not after. Does this answer the question? Cheers, Alexander. |
|
Alexander Koller
|
Posted: February 18, 2010 10:31 by Alexander Koller
|
|
Hi Salvador, yes, we will ask you to write system descriptions again. The time for writing these will be in May, after we have finished the public evaluation and analyzed the data. Cheers, Alexander. |
by: niels_s
