Public Redirects: Ideas needed!

**13th_Disciple** · 02-14-2008, 11:55 PM

Ok, the short and sweet. I am considering doing public redirects. I am not right now considering bandwidth or price, as I am more interested in features, then I can base what it may cost me off what is necessary to meet maximum community needs. Here are the basics, and you folks can offer any advice you can think of. Forget the idea of bandwidth right now or any other internet based limiting factors. Assume server is configured for maximum connections, and assume all files uploaded are compressed.

So number1 is:

1: All files uploaded must be compressed (will only allow file extension of UT3 compressed files to be uploaded)
2: Uploads to be done via anonymous FTP and then check for duplicate file names/sizes. If a file has the same name, different size, the we can assume that it's a new version of the file. I am sure a lot of you are like me and HATE having to deal with changing a file name in a config file for mutators and such.
3: The uploads are placed in a queue for approval before being moved to a standard web accessible directory
4: Files will be be distributed to multiple servers via rsync, so that no one server is loaded to the gills with connections, so to speak. Of course that is up to each server admin to use the closest redirect server.

Now, that is just that absolute base of what I am thinking. I have tons more ideas, but the core of it all is a basic public redirect service for those with either crap connections or low storage availability. Any ideas, thoughts, hints would be greatly appreciated.

Yes, I know this can turn into a rather large deal. As I said, forget any aspect except the basic premise of providing a place to upload files and serve as a public redirect. Thanks fellas!

**ANick** · 02-15-2008, 12:23 AM

Well I been using gameservers public redirect for several years and it has been great going back to UT2003 and I think I used it for XMP and before that with Unreal 1 and UT ect I was using a public site located in NY but was shut down.

All ucc compression .uz compress

Right now there are issues with UT3 and the compression process does work in the latest beta patch 1.2 but will only compress large files a few %. Like from 39 mg to 36 mg just is not going to work. So I dont know if that will be fixed or if the cook is somehow effecting the compression but never the less the compression process does not seem to be like it should and I dont know if the public patch when it is released will have a good working compression.

So at this point uploading uncompressed files to clients especially if they are large map files is very risky for servers and would tend to be a turn off to clients who have to wait for a large file then get caught in a map switch ect ect.

If the next public released patch has a good working compression for the files then your idea will be good. Like I stated before in UT2004 and earlier I can compress a 30mg file down to under 10mg so ??

Good luck

**13th_Disciple** · 02-15-2008, 12:37 AM

the old pub redirect was STNY.. the file extension was uz2, as best i remember for ut2003/2004, but i may be wrong because i have not played either in forever.. this will be standalone redirects, no web hosting, no game servers attached. dedicated only to redirects, so they'll be fast. map sizes won't be an issue because storage is cheap right now, in the grad scheme.

i am not going to do anything until a patch is released, except test.

**ANick** · 02-15-2008, 12:51 AM

Ya that was it STNY and I used to have conversation with the guy.

Anyway I just dont know how UT3 compression is going to work or if it will even work as we know it so we will see with the next patch release?

**13th_Disciple** · 02-15-2008, 01:05 AM

Originally Posted by ANick

Ya that was it STNY and I used to have conversation with the guy.

Anyway I just dont know how UT3 compression is going to work of if it will even work as we know it so we will see with the next patch release?

indeed.. the compression is about the only thing i worry about from the game side..

**AZ-{OVGY}** · 02-15-2008, 04:54 AM

Originally Posted by ANick

Right now there are issues with UT3 and the compression process does work in the latest beta patch 1.2 but will only compress large files a few %. Like from 39 mg to 36 mg just is not going to work. So I dont know if that will be fixed or if the cook is somehow effecting the compression but never the less the compression process does not seem to be like it should and I dont know if the public patch when it is released will have a good working compression.

I doubt compression will get better, you must remember in previous versions of UT maps weren't cooked. Cooking is a form of compression, hence the reason why you start with a map that is a couple of hundred MB and end up with a map thats 15-25MB. Maps in UT3 have alot more data in them now. Since they are cooked it is virtually impossible to get any more compression out of them. If you compress a file with winzip or winrar and then compress it again, you'd be very lucky to shave half a MB off it and the same thing applies to UT3 maps now. Faster redirects are the only thing that will help this problem. Also music is now in the map files (if author added music) which makes them bigger, in UT2004, music was a seperate .ogg file and that wasn't pushed with maps.

I've setup my own redirect for my UT3 server,

82.29.79.45:7777 (patch 2 beta 1, compression on, 18 custom maps so far whilst testing patch)

for me a 10MB map comes off in 30 seconds (redirect is in my town), others from UK and around reporting good speeds too.

**ANick** · 02-15-2008, 07:35 AM

Yes indeed a cooked map will not compress much more that it allready is.

I see this situation as being stuck between a rock and a hard spot and I dont think there is a good server to client solution for content push with UT3!

As I spoke earlier in other threads about ucc compress I was under the impression that UT3 was going to be like UT2004 and earlier unreal builds as far as compression redirect but it will NOT be the same so I am now bewildered as to how community stuff will flow?

O well good luck to whoever wants to spend the time and energy on it because when it becomes "work" versus a "fun Game" im out!

**AZ-{OVGY}** · 02-15-2008, 07:57 AM

The unfortunate thing for people is that public redirects will no longer cut it, not for now anyway. With a public redirect the max bandwidth it has is distributed between whoever is using it. With a public redirect there could be thousands using it, this is why sometimes public redirects are fast and at others they are not. Sometimes the redirect will be under heavy load, for example say a redirect has 1000 servers using it, if 900 of them servers (lets say they are all 10 slot servers) are using it at the same time and they all are full with players and each player needs the maps the servers are using that would be 9000 clients downloading from the redirect, causing a slowdown due to the total bandwidth being split between all 9000 clients. UT3 only adds to this problem with its larger content. However if you have a private redirect there will be alot less likelihood of your redirect becoming under heavy load. Better still if possible, set up a redirect from home (preferably on a seperate connection to the one you use for your main pc) and use that for redirect, obviously this is the most expensive option as it requires 2 internet connections but will provide the best results as all bandwidth is used for your servers redirect and nothing else. Private redirect from a good provider should be good enough for most people though until public redirect providers can pump the extra bandwith needed to move UT3 content quicker.

Originally Posted by 13th_Disciple

Ok, the short and sweet. I am considering doing public redirects. I am not right now considering bandwidth or price, as I am more interested in features, then I can base what it may cost me off what is necessary to meet maximum community needs. Here are the basics, and you folks can offer any advice you can think of. Forget the idea of bandwidth right now or any other internet based limiting factors. Assume server is configured for maximum connections, and assume all files uploaded are compressed.

Bandwidth and cost are the first things you need to look at and assess, it's going to take alot of both to come anywhere near to what gameservers and others allready have nevermind better them. Then you need to look at who the redirect will be tailored towards. For example you can have alot of bandwidth but if your based in the US then your no good for the UK and vice versa, nevermind having an optimal server setup so having multiple redirects, like you said is a must. Planning where your redirect servers will go will take alot of thought in order to meet your requirement "to meet maximum community needs". Then you must realise that if you do manage to set this up, the quality of service will degrade just like any other public redirect due to the fact that having a few servers using it at first will be great for them but the more that come (and they will) the more the bandwidth will be split between everyone resulting in everyone getting slower redirect speeds, this then puts you in the position of thinking "do I put more money into this" (answer to that being how much do you have? and how much can you afford to put in?) or "do I just leave it as it is" which makes it no better than any other redirect or maybe even worse. Your plan sounds good but ultimately can you fund it? and even if you can fund it, you could be out on a losing battle from the offset.

**13th_Disciple** · 02-15-2008, 01:34 PM

I don't plan on gearing the redirects for anything other than NA, at this point. The majority of people getting game servers are offered a free redirect from their provider, but some are not. STNY was a single server offering redirects. I have an idea on a distributed redirect system that might could work for several hundred servers simultaneously.

There is a lot I am not stating here, that is why I asked for a short and sweet set of requirements, as if the entire thing was in place.

**AZ-{OVGY}** · 02-15-2008, 02:56 PM

Originally Posted by 13th_Disciple

4: Files will be be distributed to multiple servers via rsync, so that no one server is loaded to the gills with connections, so to speak. Of course that is up to each server admin to use the closest redirect server.

Could you elaborate on this for me please.

**13th_Disciple** · 02-15-2008, 03:46 PM

Originally Posted by AZ-{OVGY}

Could you elaborate on this for me please.

basically, have multiple servers that receive their files to be redirected by a main server so that all servers have the same updated data.. of course it will depend on where the client is in regards to how fast the redirect is, but chances are, a client machine in Los Angeles won't try and join a New York based server simply due to ping..

i know even this setup will have flaws due to potential client/server connectivity issues, but it's better than just a single server handling all of the load.

**ToxicWaste** · 02-15-2008, 04:55 PM

Do you really have any idea as to what this will cost? If you have large buy in from the community, it can cost you hundreds, if not thousands of $$$ per month. But you said you did not want to talk about that....

2: Uploads to be done via anonymous FTP and then check for duplicate file names/sizes. If a file has the same name, different size, the we can assume that it's a new version of the file. I am sure a lot of you are like me and HATE having to deal with changing a file name in a config file for mutators and such.

Not a good assumption. You cannot allow the overwriting of files.

1. A troublemaker can easily corrupt hundreds of maps simply by overwriting them with different one and giving it the same name
2. Admin 1 has loaded MapX. Admin 1 decides to make a couple of tweaks to Mapx for HIS server, and uploads MapX to the redirect. Admins 2-200 who are running MapX (without these changes) are totally screwed.

3: The uploads are placed in a queue for approval before being moved to a standard web accessible directory

Are you personally going to review every file? What happens if this takes off, and you have 100 files being uploaded per hour? 200? No one will wait threee days for you to review each file. And in the mean time, you have another 1,000 files waiting....

4: Files will be be distributed to multiple servers via rsync, so that no one server is loaded to the gills with connections, so to speak. Of course that is up to each server admin to use the closest redirect server.

You cannot rely on people to manually chose the correct server. Now if you are talking about some kind of auto-load balancing at connection time, then that's different. But again, you're talking big expense.

Also, the location of the server really doesn't matter much, unless you are talking about the deepest bowels of the rainforest which can only be reached by dial up. Global location is more important with regards to ping. As long as a location is well peered, it could be in Russia for all that matters. Once the transfer gets going, you'll get good speeds. I believe gameservers is in the UK, is it not? It served my users (NA) very well until I stopped using them.

I think what you are trying to do is admirable, but I really don't think you've thought it through enough. My suggestion is to get a group of trusted game admins together, and pay monthly dues to rent a VPS. This way you know who is using what, in terms of the load, you can manipulate file names and coordinate the update of existing files, and you can share the workload of administration.

TBH, I use a simple web hosting account. They're cheap, and they are usually well peered, enough to provide all of my servers good speeds.

Good luck!

Rich (TW)

**crianp** · 02-15-2008, 05:10 PM

Now that was one positive post...

**13th_Disciple** · 02-15-2008, 05:25 PM

files will not be overwritten, ever. if a duplicate file exists, same file name and size, the newest file gets kicked. i think you misread what i was saying. if a file is uploaded, and EXACT (file name and size) match exists, new file is not allowed into web accessible location as part of redirect files.

no, i will not be doing this myself per approval of each upload. it will be scripted. just like everything else i do. by trade, i am a unix admin working on hp-ux, solaris, aix, linux, tru64, etc..

as for money, i am more aware of cost than anyone else because i already know what to do to get the basic premise to work and pricing on hardware and connectivity.

as for getting them to pic the right server, there will be a list of servers and their respective locations. while i know there will be instances of issues, the redirect network will be fairly well spread out, so that no one server gets all of the load. even if that's the case, there are ways to mitigate that at my servers level to redirect the source from another server.

**13th_Disciple** · 02-15-2008, 05:42 PM

i realize i am being quite vague about several things, and i really do apologize about that.. but i have an good idea and don't wanna just spit it all out at once.. but i honestly appreciate all input i can get..

i do understand it's a large undertaking and that it could potentially be quite expensive.. but i even have a few ideas of how to offset some, if not a majority of the cost almost from the get go. this won't be an overnight setup, so i am not foolish enough to believe that will happen.. and this whole thing could possibly never see the light of day.. so that's why i am looking for what people would want in a service like this..

i also know hosts often provide redirects, but a vasy majority of them limit not only storage, but bandwidth from the redirect server, as well.

**AZ-{OVGY}** · 02-15-2008, 06:32 PM

The only thing that people would want or that you could offer is fast download speeds and what people pay for a redirect for, the ability/privilidge to change/modify files. To do this would mean alot of problems if everyone was using the same redirect folder. The only way around that would be for everyone to have their own redirect folder which means creating them yourself (people would have to apply for an account). You could get an automated account system that would create the accounts for you but you'd most probably have to get it made for you which would be expensive or even make it yourself. With this would then come a large storage space demand as you would have duplicate files in each account's folder. Other than that I cannot think of anything else that people could possibly want from a redirect, I doubt very much that this thread will yield any better ideas. Like ToxicWaste said I think what you are trying to do is admirable but that being said it is also going to be time consuming, expensive and very hard to achieve what you goal is, a free redirect to meet maximum community needs.

**ANick** · 02-19-2008, 07:01 PM

I think the issue is cooking!

I have a completed map build and it works great un cooked for PC and its file size is 4.5 mgs. Is the same as what I would use and do for UT2004/3 ect.

After publish/cook that map turns into a 30 mg file and will not compress.

I am going to try to ucc compress the 4.5 mg and run it on a redirect and see how this works server to client.

I think the issue is PC versions dont really need to be cooked and that process is more or less used to publish the map for PS3 ect.

Ill get back..

**ANick** · 02-21-2008, 04:03 PM

Na I think for the maps to work in total they have to be published/cooked. At least im pretty sure any maps with vehicles ect in them.

So still at a loss here about compression and I think the publish/cook process will over ride the compression redirect as we know it and maps ect will need to be sent to clients uncompressed or downloaded by clients prior to joining.

**recoiledsnake** · 02-21-2008, 04:21 PM

You can use round robin DNS to distribute the load.

**Lexx000** · 03-25-2008, 11:35 AM

Hello,

Just reading this thread, as I have been dealing with connection errors on my dedicated server since v1.2 game out. I am assuming they are due to maps overwritten or the compression. (I'll admit I could be wrong, but these issues never existed on v1.1).

Has anyone tried using a free web hosting provider to see if they can host a redirect on that?.

I am looking for solutions as well, we have a small group of about 6-8 people who sometimes don't always grab the maps from urls that I provide (not direct download links) and thus have a person scrambling for a map as the rest of the people play on.

Since this whole issue with the compressed maps has creeped up and different versions, etc, with gameservers.net's redirect, I have been researching free web hosting sites.

UT3's redirect function only requires a very very basic HTTP download functionality correct? has anyone tried this with a free web host?. I am going to give it a go myself, but was looking for anyone else's experiences.

Cheers All,

Lexx000

**AZ-{OVGY}** · 03-25-2008, 10:14 PM

Free web hosting comes with alot of requirements that would restrict you from doing redirect, such as on most you have to have an actual web site there or your account is closed, alot don't allow them to be used for file sharing and if you use to much of their bandwidth/resources your account can be terminated for that.

**Warped_Out** · 03-27-2008, 06:47 PM

The main problem you will have with a free website host is that you won't have room to put any maps, plus, you will quickly run out of bandwidth as soon as 2 or 3 people download a map.

Most of the public redirects are slow as hell because so many peeps are using them.

The easiest and most inexpensive solution to your redirect problem is to purchase a hosted account (which will include a domain you have to purchase) with www.godaddy.com. Purchase a domain (they are cheap) and then purchase a hosted account (3.99/month). You'll get plenty of bandwidth and super fast redirect downloads.

Godaddy needs to commision me on this..........hehe

**Lexx000** · 03-28-2008, 05:24 PM

Originally Posted by AZ-{OVGY}

Free web hosting comes with alot of requirements that would restrict you from doing redirect, such as on most you have to have an actual web site there or your account is closed, alot don't allow them to be used for file sharing and if you use to much of their bandwidth/resources your account can be terminated for that.

Thank you for the advice.

I tried one free web hosting provider, and quickly learned that even though
they promise a ton of diskspace and bandwidth for free, the file sizes are limited to 10mb in size.

I don't mind paying actually, but wanted to see what was out there.

One thing that always crosses my mind, is why a site like FileFront that must have a TON of bandwidth doesn't just offer a redirect? Since many authors are posting their maps their anyhow, why not just offer a redirect that the authors of maps populate as well?

Just thinking out loud of course, but I would be willing to PAY for that vs. going the route of hosting my own.

Thanks for your insights.

Lexx000

**Lexx000** · 03-28-2008, 05:26 PM

Originally Posted by Warped_Out

The main problem you will have with a free website host is that you won't have room to put any maps, plus, you will quickly run out of bandwidth as soon as 2 or 3 people download a map.

Most of the public redirects are slow as hell because so many peeps are using them.

The easiest and most inexpensive solution to your redirect problem is to purchase a hosted account (which will include a domain you have to purchase) with www.godaddy.com. Purchase a domain (they are cheap) and then purchase a hosted account (3.99/month). You'll get plenty of bandwidth and super fast redirect downloads.

Godaddy needs to commision me on this..........hehe

Thanks for the advice, going to check them out once I finish responding to you. I guess one thing I was worried about was just how fast some of the redirects would be, especially at a low price, but now that I see someone recommended one of the literally thousands of providers out there, I will definitely check it out.

Thank you!.

Lexx000

**Warped_Out** · 03-28-2008, 08:04 PM

With the hosted account, you'll just need some kinda website software or a simple FTP client to create the folder to hold your maps and upload them. It's very simple, inexpensive, and you have complete control.

Good Luck............

Lemme know if I can help

irc.gameradius.net - Channel: #War-Mongers

**JD8** · 03-31-2008, 12:46 PM

My clan (www.arcclan.com) has been hosting our redirects and website on Godaddy for a long time. The redirects are very fast and we have clients pulling UT99/2k4 and UT3 files from it all at the same time.

**oldkawman1** · 03-31-2008, 10:15 PM

Originally Posted by Warped_Out

The main problem you will have with a free website host is that you won't have room to put any maps, plus, you will quickly run out of bandwidth as soon as 2 or 3 people download a map.

Most of the public redirects are slow as hell because so many peeps are using them.

The easiest and most inexpensive solution to your redirect problem is to purchase a hosted account (which will include a domain you have to purchase) with www.godaddy.com. Purchase a domain (they are cheap) and then purchase a hosted account (3.99/month). You'll get plenty of bandwidth and super fast redirect downloads.

Godaddy needs to commision me on this..........hehe

Also, most commercial hosting services have limits, as i recently found out, in upload and download speed. My readyhosting site has a 100 Kb/sec max limit on uploads and downloads. The more connections, the slower the speed on each. Way to slow for redirect. Maybe, several server admins could co-op and split the cost of a redirect site? I for one would be interested.

I wonder if a bit torrent type of thing would work? I know it was great for the DEMO and other large things like CD and DVD content. But, I really have no clue on exactly how that works, just that it spreads out the bandwidth somehow. Each downloader is also an uploader.

**dave neun** · 03-31-2008, 11:37 PM

godaddy is very good IMHO I have been using them for years now hosting my site and redirection. I have tried at least 6 other hosts, the only better deal would be BlueHost.com and they are also a bit more forgiving with permissions. In my personal tests download speeds from BlueHost avg. 450kb/sec. godaddy's avg. is about 2500kb/sec.

testing through the game it seems as though I get about a meg/sec at godaddy. With any hosting you can expect to have some dropped out downloads here and there, it is inevitable unfortunately. godaddy has been fairly good, I have had a few issues here and there but nothing I would consider to be bad or lasting any length of time. I also have several Maps etc. for download on the site, if you want to test the speeds you are welcome, I am still hosted w/ godaddy BTW.

occasionally I have a file peeter out but not often and the players usually try to rejoin if that happens as they know the speeds are normally blazing fast.

heres a link to our maps page, unfortunately all 2k4 as I need to do some updates and have been busy with other things, but there are several so you can get a feel for speeds .

http://www.poweredbygas.net/maps.htm

I recommend the linux hosting as it has a lot more to offer then the windows plans and you don't need to know Linux to use it. You get a pretty easy to understand console that automates almost everything you would want to install. okay enough ... otherwise were gonns end up on some godaddy ad somewhere.. lol

cheers,
dave

Thread: Public Redirects: Ideas needed!

Thread Tools

Display

Public Redirects: Ideas needed!

many thanks for the advice, learned some things.

They sure do!, I will check out GoDaddy.. thank you.

Bookmarks

Bookmarks

Posting Permissions