Interop

Why "Net Neutrality" is Wrong

In the late 1990s, I worked with an amazing group of brilliant network engineers building the InteropNet for the Interop trade shows around the world. We were always pushing the envelope, introducing next-generation technology before it was really ready. During a number of those years, we delivered real-time video traffic over the network, often using multicast methods that are still not widely used. We were always a little ahead of our time. Before I explain the details, allow me to mention one concept that is critical to understanding everything about the Internet: all transmissions across the Internet are made by packets. This means that every file or stream across the Internet is chopped into little 1400-byte chunks, each of which traverses the network independently of all the others. There is literally no relationship between the packets on the network. They are only reunited at the receiving end after they are off the network and in the device that will interpret them and deliver the result (like a video playback, email, file transfer, or any other end-to-end application).

But over the network, those packets are 100% independent of each other.

Because they are independent, they are subject to all kinds of issues. Sometimes, packets are dropped because a device is overloaded. Since packets can take different paths, they can arrive out of order or with varying time between them (called "jitter"). For many types of data transfer (like email, files, and even instant messaging), most of these things don't matter at all.

However, some traffic is very sensitive. Especially audio and video that is time-sensitive (used for applications like video calls, audio calls, live broadcast).

Back to Interop and the InteropNet... Delivery of video, even over the high-speed networks we were using, meant having to recognize the different requirements of traffic types and using the network resources in ways that accommodated those requirements. During those years, the IETF (Internet Engineering Task Force, the volunteer organization responsible for the standards that allow the Internet to function) defined the Differential Services (diffserv) standards to provide network performance appropriate to the type of service required.

This is an essential concept! Networks must be able to differentiate all of those independent packets flying around the network.

The New York Times has been reporting on both the FCC comments about so-called "Net Neutrality" conversations and the rumored Google/Verizon agreement on network usage. The typical idiotic political conversation has ensued, of course.

The entire idea of "an open Internet" is foolish at best and dishonest political posturing at worse. In this situation, it's actually both. Besides, "Net Neutrality" is not possible! Not only that, it's not even desirable.

Bandwidth costs money. Equipment costs money. More bandwidth costs more. Differentiated services also cost more. We all want them to be offered by the providers so that we can have live video, reliable voice-over-IP, and additional services that we haven't even imagined, yet.

The conversation, then isn't about "neutrality," but rather about universal access to differentiated services... at an appropriate cost that will be determined by the market if we just allow it to do so. After all, nobody wins by denying access, and in a free market, those who do will lose business.

There is one group who benefits: the idiot politicians who want control.

The entire focus is wrong. Typical of the politicians playing at being engineers. It just doesn't work.

Update: The Wall Street Journal ran a bit more detail on the Google/Verizon agreement today. The comments from the so-called "Free Internet" speakers are very telling: they don't understand how the Internet actually works.

Back to the Warehouse

So, I ran off to Tampa for a couple of days to attend a meeting that had been planned before we knew that we would be doing NY. I grabbed an early-morning flight back, and got back here about lunch time Sunday. There were more of the usual suspects here, of course, since most of the team arrived while I was away. It's great to have chelliott back, and the core team reached critical mass.

Of course, as usual, the warehouse T1 is struggling under the load of so many downloads. Although we remain unsure exactly what's being downloaded (ha!), the strain has led to some interesting solutions. At the Spring Hot Stage, you may remember that Jim used RDC to dig into a PC at his house in order to download using his bandwidth (considerably better than the warehouse). Yesterday, taking a hint, chelliott headed over to Jim's house to download Cisco Clean Access and other related stuff. He started in the driveway, but by the time he finished he had helped himself into the house and met Greg coming home while sitting in the living room.

Ah, the Interop NOC team... :D

Come tomorrow, we'll have the vendor engineers here, and we're actually ready for them!

Quote of the day: "This stuff is still way harder than you think it should be." (Karen regarding 802.1X, et. al. that they are working on for the Cisco NAC demo.

...they're baaaaa-ack...

The warehouse suddenly filled up this afternoon. Well, filled-up is a misnomer, since there really aren't very many folks here. But, compared to yesterday, it's more full. Chris and Kevin showed up today, and we have most of the NAC environment re-created from Vegas. We have a bit of configuration to do on the switches to get the network routing, but we got new versions of Longhorn and Vista installing, all of the devices and systems out, and mostly ready to go.

How did that happen?

Fundamental question: is there a "bleeding edge" any more? Is there technology that can cut enough to make you bleed? If so, what is it?

Shake, rattle, and roll...

Well, we didn't feel it here, but there was a little rumbler a bit northeast of here: USGS Earthquake Hazards Program » Event 40187964.

The warehouse is eerie. Quiet. No one here but Chee and me. Minimal equipment. Our racks seem more full than any others. And, of course, I've stolen a number of servers from the OSS rack to populate ours for additional servers required this time around.

It was also interesting to discover that the Labs core actually does some stuff. So, since Chris and his racks aren't here, we didn't have a fiber/copper termination point for the labs. Not a big deal, but just an example of how focus creates those "Huh?" moments. I hadn't even thought about the fiber connection requirements here in a while.

Tomorrow, the team begins to arrive. At this point, the rack is lit, a few extra servers are there, and there's a new Summit 24 acting as a fancy media converter. We'll be sharing space with NOC 2, and have a lounge area next to the Labs area. Perhaps it will help us draw additional folks.

Anyone coming to NY?

Belmont, here I come...

I find myself wondering what this Hot Stage will be like. Only one InteropLab (NAC). There's been quite a segregation between InteropNet (formerly eNet) and InteropLabs (formerly iLabs) since I was re-introduced to the team following the "great black listing of 1999," So I am very interested in seeing how the smaller teams will allow us to connect better.

I'll arrive midday tomorrow and try to get everything completely ready for the team. We've got three largely autonomous teams within the NAC lab, each with responsibility for a single version of NAC (TCG's TNC, Cisco's NAC, Microsoft's NAP). While independent, the approaches are analogous and intended to help attendees understand similarities and differences--and to touch technologies that might not be completely available elsewhere.

More after I've arrived and settled in...

Another Hot Stage

...here it comes! We'll be back in Belmont as the only Lab for NY. I don't have any idea who all will be there, or how it's all going to work. It's feeling pretty weird at this point, with most of the Labs folks sitting at home grateful that they don't have to jump on a plane and head to Belmont.

I'll be arriving there on the 2nd, the rest of the team on the 3rd. Most of us will be there for a week, putting together the Las Vegas demos, upgrading where appropriate, getting rid of demo rot, etc. We have a couple of changes in the team, since crw's not sure if he'll be able to make NY (he's at part of Hot Stage, anyway), and Brett can't make it. Kevin is going to take on NAP as the lead, and Karen is taking on C NAC.

I'll post more as we get closer and get on-site...

It's Just Not The Same

What a weird experience. Everything here--even the InteropNet (formerly eNet) folks--is quite calm. No emergencies. No one going crazy (except for Jim, but we'll get back to that later). Just stuff happening at a pretty leisurely pace.

We got here yesterday to start setting up the Labs. My flight was delayed, so Jan and Joel did the vast majority of the work. Once again they have me wondering what my value is in the equation, but I'll do what I can... We had power, tables (the tops had to be redone), and gear. Joel came up with the idea of zip-tying the power stips just under the table tops, so we don't have to fish around for power on the floor. That gives us a bunch of extra room for storage under the tables, too (our tables are three counters wide, giving us a center (empty storage) and two sides (non-empties and other needed stuff).

The NAP stuff rotted in the crates, so Carig and Kevin are wrestling with it. Last I heard, they were blowing away the changes that an unnamed vendor made earlier today by restoring a backup from Hot Stage. It doesn't help that Microsoft decided that they couldn't afford to have any of their NAP engineers here. It's only a third of our demonstration...

Meanwhile, at InteropLabs CORE, Stradtman is doing an excellent job following in the footsteps of TEFKAS (the engineer formerly known as Swanson, for those of you who have forgotten), by hanging off a 15' ladder by his toenails while stringing cables into the green room. Of course, I was the one standing on a chair to help. No, it wasn't a wheeled chair, Chris. He did give the safety speech, though. In fact, as I type this, he did step up onto a wheeled chair. I made him get off and use a non-wheeled chair. I have to stop to make sure he doesn't get stupid, again...

Why am I here, again?

This is one of those days that I feel like I'm not helping enough. Everyone was going crazy today, with Craig trying to corral Vista and Longhorn (we finally learned that Vista is definitely the client, and Longhorn Server has not got a release name, yet, so it's still Longhorn). The difference explains how what we were installing the past few days never worked. It helps to have a Microsofty in the crew to get this working.

They got a lot done today, but then managed to blow something up when they started working with the Aruba gear. I'm not exactly sure what they did, but the deer-in-the-headlights look from Craig was priceless.

Brett managed to get a lot of work done today as a result of teaming with the LANDesk guy. They banged through a bunch of the Cisco documentation and managed to get pieces of it working--even when the documentation was wrong.

Joel's been busy pointing out all of his articles in the Network World magazines in the break area, especially his huge article about Network Access Control that just came out today. It's pretty good, despite the author. (I wonder if Joel will ever read this... probably not!)

Anyway, we're pretty beat. It's time to head home. Errr... To the hotel, I mean.

I'm turning Japanese

I forgot to mention this in the earlier post for the day, but one of the fun things about coming to the Hot Stage event is always trying to figure out what's going on with the Internet connectivity. This year, amazingly, we had connectivity when we got here. The 45 net was actually coming into the warehouse. The subnets were assigned and routed.

But, going to http://www.google.com/ gets redirected to http://www.google.co.jp/! I assume because the show was last there, and somehow someway Google has been convinced that our addresses are there.

I am beginning to think about how the exhibitors and attendees will enjoy being redirected to Google's Japan site (not that many people at the show are likely to use Google). I suggested that we might want to change that prior to departure from Hot Stage. But, we'll see...

What is it about Fry's?

I know that I shouldn't ask this question, but I think that there must be something that is inserted into the water or air here in the Bay Area. Folks come here from all across the world and are inexorably drawn to Fry's. They return to the warehouse with all kind of interesting stuff. Some of it might even be useful.

Yesterday, Chris and Brett showed up each carrying a new tank with a laptop hard drive in it. These tanks, made by Vantec and called the AVOX Jukebox, not only provide the typical capability of a USB 2.0 hdd, but they also come with a remote control and audio/video outs. As a result, the things act like an audio/video player, including providing playback of ripped DVDs: "DVD Manager - Playback of DVD File same as Original DVD Title" as the box says...

Of course, folks have also showed up with the latest USB thumb drives that double as floss dispensers, blank DVDs that also work as wireless signal finders, and water that comes in bottles that will serve as a signal extendeer for a WLAN.

I think I heard someone mention a personal wireless power subsystem built into a pocket vest, too.

Well, it's time to sign-off. We're doing a Fry's run...

God bless Ron Jarrell...

Good night, Gracie.

What day is it?

Someone said something about a time change. But, this is Hot Stage and who really knows what time it is? I think they get us prepared for 'Vegas by having us in a room with concrete walls and no clocks. I wander back to the hotel, drop into bed, get up when I wake up, and head back to the warehouse.

Yesterday, we actually went and had breakfast. The hotel's powdered eggs just don't work for me.

All of the iLabs folks are here--well, besides Karen who has deigned to grace us with her presence sometime today--and the tables are covered with stuff. We actually had both DL320 servers and T43 Thinkpads when we got here, the racks were up, tables were ready (except for the vinyl, which is another story all-together), and Chris Stradtman was here sorting cables(?!?!). Chris Hessing was helping him, as was Matthew Gast. It was highly intellectual work, apparently.

Ah, yes, the vinyl. The Interop staff had really worked hard to get the place ready for us. And they had done a really good job. Most of the staff are back from previous shows, and most know the InteropNet pretty well. Val is doing her usual bang-up job, and Lora has hit the ground running, too. But, to get the vinyl for covering the tables, it apparently took 4 or 5 staff people multiple half-hour phone calls to get it delivered. It was one of those little things that had folks fuming; how can it be so hard to get something so stupid actually done?

Pretty funny if you have the perspective to drop back and think about it.

Another shift is a timeshift that has nothing to do with the time change. I'm noticing that I'm here in the warehouse at about 8am every day. What's up with that?! Although I was a little behind that today. Other folks are here that early, too, both eNet and iLabs folks. And most folks are out of here between 8 and 10 at night. It's all very interesting.

Last night, we spent some time sharing some of the "old days" stories with Val. Everything from Ron Jarrell's love of the Paris convention building we had to navigate to various unnecessary wrestling matches between NOC team members and staff. She heard a few new stories that helped her understand the history. And Ron, she's really, really sorry; she didn't know any better.

BTW, we're also writing blogs over on the Interop site (http://www.interop.com/blog/) that are technology-focused. This, from my perspective, is still my weak attempt to maintain the tradition of "Notes from the Pit".

Speaking of which...

God bless Ron Jarrell...

Good night, Gracie...

Will we make it?

So we're at that place, again. The time when we wonder if there is any hope of us pulling everything together in time. The experienced part of me knows that somehow we'll figure out how to pull everything together, make it work, and present a compelling educational experience to those who visit the iLabs. But, the practical and logical part of me stares at where we are and thinks that there's no way we have a prayer of pulling it off...

This year, it seems, we started even later than usual. It was probably my fault somehow, but it's taken a long time to get traction. As we're trying to get Cisco, Microsoft, and the TCG/TNC groups to work with us to pull together at least three silos worth of technology (one for each set of standards), it is often a task of helping folks understand how the iLabs works, that we're not looking for canned demos, and that it's not just more floor space for their booth stuff.

Not to mention that someone is actually relying on me for details like who I've talked to and what they said. Right! Like there's any hope of me actually remembering that. Or writing anything down!

Will we do it this time?

I guess you'll just have to check in here to find out!

God bless Ron Jarrell...

Good night, Gracie!

...and so it begins again

It is time to begin preparing for Interop Las Vegas 2006. As Hot Stage rapidly approaches (eNet starts 2/20 or so, iLabs, 3/30), we're working to engage as many contributors as we can. Many of us involved in the iLabs feel like we're behind, and we're anxiously looking for involvement from contributors old and new. This is one of those times when I wonder if we'll pull anything off...

This year, I have the dubious distinction of leading the Network Access Control iLab initiative. The team comprises an all-star cast including Carig, the dynamic duo (Joel/Jan), team-lead-turned-educator Karen, and up-and-comer Brett. In addition, I am working on pulling in additional folks from the past (yes, I've been remiss--if you're one of those folks, please forgive me for not getting e-mail to you before now!).

But, it's really about contributors. We would like to gather sufficient contributions to compare and contrast TCG's TNC, Cisco's NAC, and Microsoft's NAP. Each demonstration obviously requires a few key contributors and we'd like to have a few additional ones. Right now, it's sketchy.

The other two initiatives are SIP and Open Systems, led by Jim and Hege, respectively. I have heard that SIP has similar challenges to ours, but I'm sure they are in better shape. So, my competitive nature kicks in. We need to get it done.

More as it happens.

God bless Ron Jarrell...

Good night, Gracie...

Katrina Influence

Hurricane Katrina ravaged the Gulf Coast of the US, detroying homes, property, and lives in the process. It was a catastrophe. In its wake, it leaves us with rebuilding following the admittedly confused initial response to the needs of the people caught in the storm's wrath.

In his Advanced IP Pipeline blog, Paul Kapustka suggests that it's time for a permanent "Geek Corp" to take on the challenges of getting communications and related infrastructure back on-line following events such as this. Sound familiar?

I wrote to him and suggested that the Interop team would be a great core for this group. If there is enough of an interest--and perhaps even financial support--for this. Would you still be interested in it? If so, ping him, or drop me a mail.

God bless Ron Jarrell...

Good night, Gracie

It's official...

Unfortunately, the new Interop show in New York will not have an iLabs, as was announced last week by MediaLive. Frankly, this makes perfect sense to me. The iLabs are detailed, labor-intensive looks into emerging technology. As a result, they require significant manpower that translates to expenses. Given that this is a new venue, it is not surprising that the return on that investment is unclear.

I hope that those of you on the East Coast will check out the event, regardless. I may be able to be there (we'll see!), and will do my best to keep readers of this blog current with Interop activities regardless.

Do let me know if there are particular interests that you have surrounding the event... (Really, I just want to know if anyone is reading this!)

God bless Ron Jarrell...

Good night, Gracie...

New York, New York

We're still waiting to hear the details on the New York show, including whether or not we'll have iLabs and, if we do, how many we'll have. Given that the blog showed up on Interop Daily, there is an indication that there's interest in having it be available to a wider audience. Carig also mentioned something about making it available more broadly, but I have no idea what he meant.

If you have some ideas for iLabs for New York (the types, the foci, or even some recommendations for the attention-grabbing demos), drop them to me or into here as comments.

I hope you're all well... Stay in touch!

God bless Ron Jarrell...

Good night, Gracie...

Departure time

It's a wrap...

Another Interop is behind us. The iLabs team got together for a "post-mortem" dinner celebration at Morton's last night.We were tired but talkative as we enjoyed the company and the "dead cow."

This morning, many of us are off to our homes. But, with the news of the New York show in December, it looks like we may be doing this again sooner rather than later. Details are still in-process, but the expectation is for a full eNet and iLabs, including Hot Stage. Dates are likely on the Interop web site, but I think I heard the of the 12th. The Holidays are a great time to be in NYC! If all of this happens, it's very likely that (the illusive!) Terry will join me, and I'll finally get to introduce you all to her.

Karen gets the "blow ssh's mind" award for this show for demonstrating just how much of a curmudgeon I have become. As she leaned forward over her notebook computer, looking up at me with one of those looks, she said, "Well, we have been doing the iLabs for 7 years!"

"No way!" I said... Only to discover that my minor victory was that it was the 7th year, not 7 years, but you get the point. Where in the world has the time gone?

Personally, I am very interested in seeing how we can expand what happens on the network at the show, and find ways of drawing the iLabs and eNet closer together. I think that it probably has something to do with services offered over the network (as we did with VoIP to the booth), but other things that are a bit bleeding edge and exciting. Got any ideas? Let us know by e-mail to the list or to me directly. Or just comment here. I'd like to see us avoid engineered complexity, and instead really over-engineer something that enterprises may deply in the next 12-18 months... What would that be...?

Anyway, thanks for having me, again. Thanks to Karen for the invitation, to the iLabs FSS team for the welcome, and to all my friends for putting up with me. I enjoyed the dinner immensely, and am really excited about NYC and being able to have Terry there for at least part of the time.

I intend to continue to add to this blog (at a bit lower frequency) between now and the NYC Hot Stage. Let me know if you have thoughts, questions, or other things that should be placed in here...

Take care, my friends, and God bless...

God bless Ron Jarrell...

Good night, Gracie...

That Cincing feeling


About 2 hours ago the bulk of the network was packed up and loaded onto the trucks. I only know about this because I got a message from a team member saying they were going to the pool, So I have to assume that they finally have some idle time.

Me, well I'm happily at home, sitting on the couch with my feet up taking a moment or two to dump out the last few interesting things in my brain.

I flew back yesterday using "meetings" as a thinly veiled excuse to get out of tear-down. This will actually be the first show where is missed most of Hot Stage, the First day of the show and the Last day of the show.

However it did give me the required quiet time to write all this down. As SSH pointed out in the report from 5/1 (or is that 1/5 ?) its hard to take notes when you have work to do. RJ had the advantage that he mostly worked vampire hours. Me, well I just got on a plane and got the hell out of Dodge.

So back to the eNet for a moment.

Sarcastic comment of the week actually has to go to SSH for saying today that

"I don't think the eNet really had very much going on."

(Well, I /hope/ it was sarcastic). From where I was sitting there were 60+ people who were not only working very hard on over-engineering a network but still had enough energy to develop and devise creative ways of killing their colleagues when someone fat fingered a config.

Now, I will concede that the huge arse TV in the middle of the NOC designed for video conferencing got usurped for a pair of X-Boxes, but other than that there was a lot of networking going on.

When you look at it on paper, any one component's design makes a lot of sense. But trying to glue together a meshed array of firewalls and routers, a cluster of L3 switches and wireless access points with multiple overlapping VLANs, VoIP phones delivered to every booth being driven by in-line POE switches, a cluster of NMS servers pinging the be-Jesus out of everyone, goodness knows how much test equipment and some sort of crazy clustered DNS infrastructure, and you have, well one big FusterCluck.

So that meant we spent a small amount of time configuring things, and a great deal of time trying to work out what what we broke by doing it. But to our credit, we fixed some things in an hour that some IT departments would take months to work out. Oh and it always gave us someone to blame. I think the voice guys got the worst of it but they took it very well. When, for the Nth time, a routing problem stopped the phones booting, I couldn't resist and I handed them a 'VoIP for Dummies" book that was published by their competition.

We even felt the courage to enable 802.1X in the NOC. After the people building it (I won't name names [Alice, Myself]) stopped trying to test it with their own broken laptops. We got it working for everyone except for Neal, who ironically didn't have a laptop that was even close to dot1X capable. His response was :

"Most of my test gear isn't either."

Cowboy style also kicked in while we were publishing the addressing. The magical mystery transparent proxy got the impression that the feedback from the webserver was a big download, and not just syslog output. To our horror we then discovered that even though we had killed all our browser sessions, the proxy was holding the connection open for us and wouldn't let the publish process die.

Barry asked

"how long will the publish take ?"

"Given the overloaded web server, the 5 firewalls, and the
amount of data, I'd guess at about Two Hours."

We quickly pinged Glenn and asked him to kill the process on the server side. Glenn was, of course, too busy with other problems and just re-booted the whole server.

"That should fix the problem." He said.

This sense of faux courage meant that we all secretly upgraded, or swapped out equipment entirely, in the middle of the night and mostly got away with it. But not always. There were a few morning surprises for some people (and more finger pointing) But you had to admire the defense of "It would have been worse if we didn't do something". I stopped at rebooting the NTP servers (again) when I noticed that they were out by 19 seconds and drifting at about 3 seconds a day. Alice was quick to set me straight :

"So what if all our clocks are flashing '12:00",
at least they are ALL showing the same time"

I think the clock will have finally caught up by tomorrow.

Anyway, the show is over for another year.

I'm sure that show floor is littered with the detritus of bits of Cat5, random metal parts, snack food wrappers and all the left over white papers. Some poor folk will come through tomorrow to clean up after us and try and make sense of what the hell was going on

"It's like a geek exploded in here"

( That may also explain the incident a few years ago when, on the Friday, as they tore down the NOC around us, someone snuck in and stole all our beer from the NOC fridge. Talk about a hint to go home )

Meanwhile the rest of us will face another 24 hours of looking for bars with nerd friendly dress codes. As we walk the strip we will quietly deal with the annual separation anxiety that overwhelms us when we realise that we won't be spending another night drinking too much and telling bad war stories to our friends until it is way past our bedtime.

Ssh, someone is sleeping.