Project Kenai Documentation and Training: General: Kenai WebDAV with sitecopy

Posted: September 28, 2009 19:45 by Jørgen Austvik

» Permalink

Hi,

I run maven builds in a behind-my-firewall Hudson, and I'm trying to set up a job that copies the maven site to the Kenai website URL with WebDAV.

I have looked around long for something that copy a directory recursively with WebDAV from the command line, and have found http://www.manyfish.co.uk/sitecopy/ (e.g. cadaver didn't do the recursive part).

I have configured sitecopy like this:

site kenai
server kenai.com
remote website/jet/site
local sitecopy/kenai
username ...
password ....
protocol webdav
http secure
port 443

...but a "sitecopy --update kenai" fails.

It sends this:

---------8<----------------8<----------------8<----------------8<-------
OPTIONS website/jet/site/ HTTP/1.1
Host: kenai.com
User-Agent: sitecopy/0.16.6 neon/0.28.2
Keep-Alive:
Connection: TE, Keep-Alive
TE: trailers
---------8<----------------8<----------------8<----------------8<-------

And receive this:

---------8<----------------8<----------------8<----------------8<-------
HTTP/1.1 400 Bad Request
Date: Mon, 28 Sep 2009 19:38:45 GMT
Server: Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8a mod_perl/2.0.4 Perl/v5.8.8
mod_perl/2.0.4 Perl/v5.8.8]
Vary: Accept-Encoding
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
---------8<----------------8<----------------8<----------------8<-------

Does anybody know whether this is a problem with Kenai or with sitecopy?

Does anybody know any good alternatives to sitecopy, which let me recursively copy a directory tree over WebDAV from the commandline?

-J

john_brock	Posted: September 28, 2009 22:05 by john_brock » Permalink
	Try using HTTPS Jorgen. I think that will do it for you. --jb

Jørgen Austvik	Posted: September 29, 2009 06:49 by Jørgen Austvik » Permalink
	I have: http secure port 443 The first time I do it, I am asked to verify your SSL sertificate. I think I am using HTTPS. -J

john_brock	Posted: September 29, 2009 14:17 by john_brock » Permalink
	Ok, thanks Jorgen, I'm looking into it and will ask around the engineering team to see if anyone else is using sitecopy on their Linux system. What OS are you doing this on by the way?

Jørgen Austvik	Posted: September 29, 2009 19:00 by Jørgen Austvik » Permalink
	Up to date Ubuntu linux. My guess is that there is something in that OPTIONS request that Kenai doesn't like, but I have no idea. davfs2 mounting Kenai isn't stable under high load (hangs after "rm -rf" deleting 15-20 files, e.g.), and sitecopy seems to copy only the necessary files, so it should save some bandwidth. -J

john_brock	Posted: September 29, 2009 20:22 by john_brock » Permalink
	I've gone ahead and filed a bug on this. You can track it here: http://kenai.com/jira/browse/KENAI-1288

Jørgen Austvik	Posted: September 30, 2009 15:15 by Jørgen Austvik » Permalink
	Thanks!

bhate	Posted: November 19, 2009 22:23 by bhate » Permalink
	Can you try to change your site configuration? You can try to update the remote configuration to have "/" before website. remote /website/webdavtest thanks Ajit

Martin Fousek

Posted: November 23, 2009 15:02 by Martin Fousek

» Permalink

I was trying that with similar config with the "/" but I hit another troubles with certificates...
cawe@cawe-nb:~$ sitecopy -u kenai
sitecopy: update place `kenai' (na kenai.com v /website/androidmenu/newdir/)
WARNING: Server certificate is not trusted.
Certificate was issued for server `kenai.com'.
Fingerprint: 6b:4f:11:c5:27:e0:7f:b2:38:7f:53:1e:6d:6b:9e:69:b3:6f:e3:e2
Issued to: Class B, cloud business unit, Sun Microsystems Inc, Menlo Park, California, US
Issued by: Class 3 MPKI Secure Server CA, VeriSign Trust Network, Sun Microsystems Inc
Do you wish to accept this certificate? (y/n) y
sitecopy: Error: Server certificate verification failed: issuer is not trusted
cawe@cawe-nb:~$

Jorgen did you have the same troubles - any solution?

Jørgen Austvik	Posted: December 05, 2009 15:29 by Jørgen Austvik » Permalink
	Sorry, I don't remember how I managed to get it to remember the certificate. Maybe I tried manually with curl or ssh first? I have a .sitecopy/kenai.crt file - maybe you can get that through some other means? I have tried searching on the internet for the HOWTO that I must have followed, but I haven1t found anything.

john_brock

Posted: December 05, 2009 16:50 by john_brock

» Permalink

Hi Jorgen,

We've done a lot of looking into different clients and the webdav functionality over the last couple of weeks. What we found in regards to sitecopy, is that they read, and create, the directory structure in reverse order. It doesn't work properly when you try to update something that is multi-directories deep.

We talked to the sitecopy developers and they acknowledge that this is a known issue, but they don't have a fix date for it yet.

Using the config answer to have the / in front of the URL will get things working, but the nested directory stuff can only be solved by doing multiple updates in a row until things all go up. Not perfect, but a workaround until the sitecopy folks get the fix in on their end.

Hope that helps.

--jb

The Project Kenai Team

Jørgen Austvik	Posted: December 07, 2009 12:06 by Jørgen Austvik » Permalink
	Hi, John. Thanks for looking into it so thoroughly and for taking it to the sitecopy developers! (If you have found another command line wool to copy a directory structure from UNIX while looking into webdav clients, I would love to hear about it, if not: I'll just be patient.) -J

mw46d

Posted: January 25, 2010 17:42 by mw46d

» Permalink

Can you try it again, please?

I just played with sitecopy after some updates to the apache config and it seems to be pretty happy now.

marcow@feather3:~$ sitecopy --version
sitecopy 0.16.6: FTP WebDAV rsh/rcp sftp/ssh, debugging, Unix platform.
neon 0.28.6: Library build, IPv6, libxml 2.7.3, zlib 1.2.3.3, GNU TLS 2.8.3.
marcow@feather3:~$ cat ~/.sitecopyrc
site mw-test
server website.kenai.com
remote /website/mw-test
local ~/tmp/mw-test
username mw46d
url https://website.kenai.com/website/mw-test
protocol webdav
http secure

Create a new local copy & download all the files:

marcow@feather3:~$ sitecopy --fetch mw-test
sitecopy: Fetching site `mw-test' (on website.kenai.com in /website/mw-test/)
WARNING: Server certificate is not trusted.
Certificate was issued for server `kenai.com'.
Fingerprint: 6b:4f:11:c5:27:e0:7f:b2:38:7f:53:1e:6d:6b:9e:69:b3:6f:e3:e2
Issued to: Class B, cloud business unit, Sun Microsystems Inc, Menlo Park, California, US
Issued by: Class 3 MPKI Secure Server CA, VeriSign Trust Network, Sun Microsystems Inc
Do you wish to accept this certificate? (y/n) y
File: mw1/HTML File - size 111
File: today.html - size 111
File: i.html - size 118
File: foo.html - size 17
File: foo - size 11
Directory: mw1/
sitecopy: Fetch completed successfully.
marcow@feather3:~$ sitecopy --synch mw-test
sitecopy: Synchronizing site `mw-test' (on website.kenai.com in /website/mw-test/)
Creating mw1/: done.
Downloading mw1/HTML File: [..] done.
Downloading today.html: [..] done.
Downloading i.html: [..] done.
Downloading foo.html: [..] done.
Downloading foo: [..] done.
sitecopy: Synchronize completed successfully.

Changing a file locally & uploading it:

marcow@feather3:~$ vi tmp/mw-test/foo.html
marcow@feather3:~$ sitecopy --update mw-test
sitecopy: Updating site `mw-test' (on website.kenai.com in /website/mw-test/)
Uploading foo.html: [....]Authentication required for Website Area on server `website.kenai.com':
Username: mw46d
Password:
Retrying: [.....] done.
sitecopy: Update completed successfully.

Create some new directories, add a file and upload:

marcow@feather3:~$ mkdir tmp/mw-test/mw2
marcow@feather3:~$ mkdir tmp/mw-test/mw2/mw2a
marcow@feather3:~$ vi tmp/mw-test/mw2/mw2a/foo.html
marcow@feather3:~$ sitecopy --update mw-test
sitecopy: Updating site `mw-test' (on website.kenai.com in /website/mw-test/)
Creating mw2/: Authentication required for Website Area on server `website.kenai.com':
Username: mw46d
Password:
done.
Creating mw2/mw2a/: done.
Uploading mw2/mw2a/foo.html: [....] done.
sitecopy: Update completed successfully.

Please let me know, if there are still problems Wink

Thanks,
-- Marco

Jørgen Austvik	Posted: January 26, 2010 07:53 by Jørgen Austvik » Permalink
	Something was wrong with my remote "site" directory. When I changed the local name from "site" to "site2", upload works. Maybe some bad effect of early experiments? Sitecopy and Kenai works good now - great for uploading a complete directory structure (e.g. the result of your continuous build platform from time to time). Thanks a lot for your help!