Poll question answers / Network Stats LanOCv16

1. Which table used the most bandwidth? No one voted for the answer - Table I used the most bandwidth at the LAN. At one point they were passing 400MB/sec of traffic. I think someone had some shares that people were hitting.

2. How much aggregate (inbound and outbond) traffic did we pass at LanOC v16?

a. WAN traffic - 1.6TB Aggregate
b. LAN traffic - 6.02TB Aggregate

This is a new LanOC record. We had 3 TWC 50/5 modems in a load balancing round robin config and most of the time , we were at a solid 35Mbits on all 3 modems with it peaking to 45Mbits at times.

Even with the massive HTTP / Web traffic going on , we still maintained a steady LoL ping of 90 - 100ms for the whole event . We had a steady 60Mbit of HTTP/ Web traffic during the LoL tourney and still had those pings.

There were several open play server up and running , at one point we had:

1. 32 people in a TF2 server
2. 4 people in a UT2K4 server
3. We had CSGO / CoD4 servers up with people in them .
4. DOTA server was up at one point.

Thanks to everyone who came to the event. If you have any questions or want a copy of the PFSense config that we used let me know and I can get you a copy of the config.

Doesn't a round-robin NIC team allocate bandwidth on a per-session basis?

You told me that it was a hard algorithm in PFsense dividing and provisioning a per-host allotment? Were we using Round Robin or Adaptive?

Its round robin but it remembers the state as well so if Packet A goes out WAN interface 1 , it stays there till the state closes. You set the round robin in your gateway group inside of PFSense by choosing a tier for the WAN.

We were using a LAG group for Vmware on the switch side that had 4 1GB NIC's LAG'd at Vmware and switch side.

There was no round robin NIC teaming anywhere except for the management NIC's for Vmware.

The stats were pulled from Observium running on the LAN side using SNMP monitoring. I didnt get a chance to pull the RDD graphs from PFSense.

When looking at the interface of each WAN during heavy periods , each modem was about 30Mbit - 42Mbit of traffic. For how PFSense does load balancing this thread has a nice chart that explains it .

forum.pfsense.org/index.php?topic=16923.0

Sideout wrote: Its round robin but it remembers the state as well so if Packet A goes out WAN interface 1 , it stays there till the state closes. You set the round robin in your gateway group inside of PFSense by choosing a tier for the WAN.

We were using a LAG group for Vmware on the switch side that had 4 1GB NIC's LAG'd at Vmware and switch side.

There was no round robin NIC teaming anywhere except for the management NIC's for Vmware.

The stats were pulled from Observium running on the LAN side using SNMP monitoring. I didnt get a chance to pull the RDD graphs from PFSense.

When looking at the interface of each WAN during heavy periods , each modem was about 30Mbit - 42Mbit of traffic. For how PFSense does load balancing this thread has a nice chart that explains it .

forum.pfsense.org/index.php?topic=16923.0

Sick.

So it was only round robin packet distribution inside the virtual machines in VMWare - cool - but which VM's ran there?
I saw game servers, but were firewall and DHCP running somewhere in there as well? or?

I guess, the biggest question I have, is at 4 AM, when there were all of 3-4 users, and something like 20 machines left on, why I was still only capable of pulling down like 6Mbps.

http://www.speedtest.net/my-result/4249344925

I only had a single network drop.... but I also never saw more than 600KBps.

Could there have been a way to pull more than 100MBps using the backup modems downstream /only/... or even that fourth modem?

I ask only because any games that weren't backed up by someone else or pre-installed were impossible to get a copy downloaded quickly. I'd also be willing to bet that a strictly regimented upstream and a "download what you want" would go over a LOT better - supposing you were able to block torrent ports or torrent-eque traffic.

No it was a LAG group for Vmware for the game servers. it was a load balanced connection to Vmware for management. Your not understanding how load balancing in PFSense or any residential multimodem setup work. I will attempt to explain it to you.

1. True Load Balancing relies on both the ISP and customer side using specific protocols to achieve some sort of hash based load balancing. We are never going to have that as these are residential modems.

2. All 3 modems went to the same gateway at TWC so that is a potential bottleneck that we cannot get around.

3. With 3 modems in a Gateway Group under PFSense , the way it works is that:
a. Computer A sends a packet out the LAN to the WAN Group and the Gateway Group says WAN2 is the next available.
b. PFSense remembers that hash for all traffic from Comptuer A until the connection state is closed and flushed from the table.
c. If Computer A restarts the stream before a Table flush then WAN2 is used again.
When Computer B sends a packet out the LAN to the WAN Group , the Gateway Group will round robin the WAN's for each connection.

Under the Traffic Shaping where the limiter is set at , I define a static limit for all TCP connections going from the LAN to the WAN Group. This bandwidth limit means that PFSense will divide the bandwidth as equally as possible for each requested TCP stream. Thus if the limit is 50Mbit and we have 5 people needing bandwidth , it gives each person 10Mbit. This is not perfect as sometimes people will get a bit more , sometimes they get a bit less but it averages it out.

Given that we have 3 modems and each modem is maxed at 50Mbit , this would split it to where one modem would have less on it than the others. Additionally I was using a QoS rule to classify traffic to queue it and apply QoS.

The way the QoS works is that we have several queus setup:

qACK - for TCP ACK packets
qDNS - for DNS queries
qGames - for gaming traffic
qHTTP - for Web Traffic
qOthersHigh - for traffic needing better response times
qLoL - for League of Legends traffic
qCatchAll / qDefault - CatchAll is default queue on LAN / Default is default queue on WAN.

If the traffic does not fall within one of the designated queues it goes to qCatchAll or qDefault which are the default queues and they have the lowest priority and the lowest percentage share of the bandwidth.

We use HFSC for our scheduling with CODEL queue to prevent one stream from taking over the buffer and bloating it. HFSC used percentages of bandwidth and it uses a Link Share queue to help overloaded queues. I define percentages for each queue and link share.

The whole goal of this is to provide the best GAMING experience for everyone and to ensure that tournament games have enough bandwidth so there is minimal lag when they are going on.

Your speedtest shows 4.6Mbit down and 0.63mbit up and that is inline with how the limiter is set. I think at that time it was set to something like 70 or 80Mbit download and 3Mbit upload. When we went to the big LAN at EMU in Feb , that is about what we had there per gamer and they had a huge fiber pipe from EMU.

You are stating you had issue downloading games , was that via torrent or Steam or ?? I know that several people were downloading Heroes of the Storm and were getting 6 to 8Mbits a second downloading it. Sometimes the downloads for games will fall into that area of hitting the default queues that is just a byproduct of how it is setup.

It is pretty much impossible to block torrents with PFSense unless you run Squid and Snort on it and that brings in a whole bunch of other issues I do not want to deal with in a LAN party setting. Torrents use such a wide port range and use TCP over HTTP it is pretty much an impossible task to filter just torrent's. The best way to handle that is to define known traffic , prioritize that and then filter the unknown / non-defined.

I spent a fair amount of time walking around looking at peoples screens and games and I saw people on steam pulling stuff down at a pretty decent rate. For the most part there was not a lot of latency for people playing online games there were people playing War Thunder , Battlefield , Payday , Star Trek , WoW , and Diablo , all online games requiring bandwidth along with the massive amount of LoL that was played as well.

I am sorry you feel like your weekend was impacted because you couldn't download stuff at a rate that was acceptable to you. If you have a better way to do it then I am all ears.

Please don't mistake my comments for dissatisfaction, because I was anything but dissatisfied. However, In game performance with low latency is a must, obviously, otherwise we wouldn't be at a LAN party....

But total downstream bandwidth is INCREDIBLY low for gamers, and obviously high for the short burst of time someone needs to pull larger content. If you saw people downloading games, then you weren't looking where I was looking. Our steam downstream was about 1/3rd of what our actual bandwidth was. IIRC we got something like 200KB/s?

I think that if there isn't an option capable of allocating mini burstable allocations per user to snag what they need in a short window (followed by adequate throttling), then perhaps more could be done promote a lan-wide game registry?

In windows 8, there was a lame app called "Steam Tiles" that only required a steam URL ending to list off the games you owned and allowed you to make live-tile shortcuts for /your/ games.

Perhaps at the time of event registration, a suggested/enforced population of a steam URL field could afford LanOC a pretty simply query base on which we could compile a list of games and their popularity among LAN attendees prior to attendance?

Look, we all play what we want to play, we all enjoy what we enjoy, but it wasn't until I got there and lucked into seeing someone sitting across from me playing a rather obscure game that I thought to myself "Oh! That could be fun at a LAN! Let's go download that!"

And while local/network backups and restores have brought us a long was since the days of WAD file sharing, it's no where near ideal in terms of content sharing. "Hey random guy I just started staring at, mind stopping what you're doing and backup that up for me, transferring it, and getting it somewhere?"

It works, yes, but if reactive and shaped traffic can't be accomplished without sacrificing external pings (Which is a whole other socratic discussion), then I think preparation should be demanded of attendees who are interested in playing games not being exclusively organized-by and discussed by LanOC staff and tournament announcements.

I am looking at using a scheduler for the limiter which would open it up during non peak tourney hours. I am also looking at using the M1 and M2 settings on HFSC to decouple delay and allow for burst on PFSense (See this forum topic - forum.pfsense.org/index.php?topic=89367.0)

This would allow bursting along with a few other things. As Wes and other can tell you , our PFSense config is always a work in process and I constantly try new things to get it to run where it needs to go.

The problem is that HFSC / traffic shaping / QoS at a large event like this using whatever we get donated to us , presents challanges in spades. It would be awesome to run managed switches and core and be allow to apply NAC and other protocols along with seeing exactly where peak traffic was coming from down to the port level.

While the limiter in its current state is not 100 optimal , it is effective at the overall of allowing for good pings and the ability to download as needed within parameters.

Potentially I have looked at adding another modem and setting it specifically for download traffic but that presents some other issues inregards to games like LoL that use HTTP to load and split source IP's.

Additionally we do set excepts for people at times as long as capacity is there. All you have to do give me an IP / MAC and I can add an allow to get you what you need then remove it once your done.

See, THAT would be sick, put in a 5/10/20 minute request for a 10-30Mbps provisioning bump....

segmenting traffic could be turn or request based, and then free it up for everyone when there isn't a specified need for a huge pull....

The only downside is that PFSense is a stateful firewall so when I do it , you have to close all connections , ie a reboot is the best way so that the firewall sees the new state.