Ed Brill

Post a Comment

1. 1  Mika Heinonen http://www.siipi.com/mika | 9/12/2007 4:20:32 PM

   Sounds to me rather like a missing Windows Firewall exception. In most cases Windows Firewall doesn't even alert that some program is trying to get out, and you have to add manually the .exe file into the exception list. I just had this case today again when I installed a VNC server, it doesn't even show up in the normal "Programs" list of the Windows Firewall so you have to browse for the server .exe file manually.

1. 2  Jim Casale  | 9/12/2007 4:45:37 PM

   Basic check would be to telnet to port 1352 to the server host name. If you can't get through then it is most likely blocked. If you can get through then maybe they are filtering in some weird way.

   We have had some issues with blocking but usually at a hotel where a user was staying. It seems they blocked everything except port 80 and 443. The users couldn't even use VPN. Hope this helps

1. 3  Pete McPhedran  | 9/12/2007 4:49:02 PM

   Hmm,

   That doesn't make a lot of sense, it wouldn't apply to just mail as there is no differentiation between database replication and sending an email. All the traffic is using port 1352, so it would apply to attaching a file, in this case larger than 2MB to a database as well as sending a mail message with a >2MB attachment. Same goes for pass through vs direct, it should happen to all or none if it is port 1352 filtering.

   Perhaps it has only been noticed in mail as that is where most people would add a file larger than 2MB.

   Am willing to assist in troubleshooting if some Comcast user(s) would like.

   --Pete

1. 4  Jim Casale  | 9/12/2007 4:50:52 PM

   Sorry I should have read the link first. My first suggestion wouldn't really help. Not sure if this would work but you might try OpenSSL VPN. Assuming you can install the server and clients, you should be able to VPN in over port 80 or 443 which might get around whatever filtering they are doing.

1. 5  Kevin Kanarski  | 9/12/2007 5:42:25 PM

   The filtering only kicks in when a single transfer exceeds approximately 2Mb. Posting text to a blog probably doesn't exceed their filtering limit. I've been testing by composing a message with a 5Mb attachment and saving it as a draft directly to the server replica. It fails every time. A simple text message works fine.

   I have a thread going on the ND 6 & 7 forum as well and people are reporting this issue across the country. { Link }

   IBM Support has also acknowledged that other customers have been calling in with this issue.

   -Kevin

1. 6  Richard Schwartz http://www.rhs.com/poweroftheschwartz | 9/12/2007 6:01:21 PM

   I may have experienced this -- and not just in Notes. But it's really hard to tell whether it's Comcast or my home network that is responsible for my problems. I guess I'd have to plug directly into the cable modem to find out for sure.

1. 7  Rich Greaves  | 9/12/2007 6:03:16 PM

   We are seeing exactly the same thing. I've had a number of reports of this problem now and have spent hours troubleshooting notes and domino connectivity until we finally figured out that all the affected users were connecting via Comcast! The info I've read on the Comcast help forums indicates that it might have something to do with Comcast's Sandvine implementation. Somehow Notes traffic over port 1352 is being grouped with a class of P2P protocols that Comcast is attempting to control. If that's the case, this is not good at all. I've been advising my clients to talk to Comcast about this, but my guess is they are not going to get very far...

   -Rich

1. 8  Nathan T. Freeman http://nathan.lotus911.com | 9/12/2007 6:04:54 PM

   This stuff is not unusual. The strategy is typically to watch traffic between two IPs on port X, and to send a TCP RESET if there's over some amount of traffic. That's how they downgrade the P2P traffic, at least.

   Whether they're doing it in this case, I couldn't say. You need a packet sniffer to really tell.

1. 9  Darren http://www.dadams.co.uk | 9/12/2007 6:31:54 PM

   Just a guess, I had problems with my first broadband package and it turned out that the Maximum Transmission Unit (MTU) needed to be adjusted from 1500 to (I think) 1372, although this figure would vary. There's a ping test to see the size that causes the packet to fragment.

1. 10  Ed Maloney  | 9/12/2007 6:32:34 PM

   I've had trouble connecting to the office Notes servers via Comcast. It seems to have gotten worse over the past few months. The Notes data is painfully slow, while HTTP traffic works fine.

1. 11  Erik Brooks  | 9/12/2007 7:13:41 PM

   Comcast user here - I've definitely noticed slower performance, but no suspicious network timeouts or inabilities to work with large files. I am in a smaller city (Tallahassee, FL), though, so it could be that they're trying things in certain markets and I haven't felt the full blow yet.

   @9 - Some broadband providers are sensitive to MTU, but I don't believe Comcast is (or else theirs is sufficiently high for most).

   It *should* be easy for them to check 1352 and I would say go ahead and ask somebody to check anyway, Ed. But - we know how things can be in MegaCorp.

1. 12  Carlos Rivera  | 9/12/2007 8:13:24 PM

   We we're experimenting same situation here in Puerto Rico with onelink. Since most of our employees use the same isp our boss called them and asked them to do something or they would loose all our accounts. At the end they accepted that they had implemented some filtering technology, but that was after denying it for almost a month.

1. 13  Ed Brill http://www.edbrill.com | 9/12/2007 8:29:19 PM

   @12, did they turn off the filtering?

1. 14  Lee Davis  | 9/13/2007 12:59:13 AM

   This is bad. With an opportunity to get worse; e.g. M$ gives $ to ISP, not Google and Notes are disadvantaged relative to WindowsLive.

   And Comcast has been getting press for their secret quotas that if you exceed your account is shut down for a year,

   But giving this regard for customers, I dont think your "Unfortunately they didn't do their homework and have applied this filter to port 1352 as well which is a registered Lotus Notes port." is fair.

   IF they choose to apply secret random filters to hurt their customers (which I am NOT endorsing) , then don't they have to do it on 1352 as well as other ports? Otherwise, wouldn't users just change the Gnutella/BT software to use port 1352?

1. 15  Mika Heinonen http://www.siipi.com/mika | 9/13/2007 1:27:38 AM

   Oh, I missed the fact that it works with smaller than 2MB files, so the 1352 port should be not blocked then.

   Maybe you should record some evidence that it cuts the 1352 traffic during large file transfers. You could run NPING all the time while you send the 2MB file, and if it gets timeouts, this would be clear evidence that ComCast is breaking the connection. The issue could be also different when sending mails or doing direct 1352 database transfer. You could also try to attach a 2MB file on a document via LotuScript on a remote server. Also yahoo mail doesn't send larger than 10MB attackments, no error comes, they just never reach the recipient. As a workaround I use always a public upload database in domino, so customers don't have to send any attachments, just a link to the URL of the attachment. I saw something that Notes 8 would have this built-in, but haven't tested it yet.

1. 16  Scott Leis http://www.isw.net.au | 9/13/2007 1:37:08 AM

   Kevin's problem sounds similar to one I had over a year ago. Here are details in case it's of any help:

   On my home PC I have installed (among various other software) Lotus Notes set to replicate with my employer's servers when I'm working, and Ventrilo for voice chat when I'm playing World of Warcraft.

   Around April 2006, my ISP (iiNet in Australia) made an unannounced configuration change which caused the network speed of both Notes and Ventrilo to be vastly reduced, but had no visible effect on any other software. Web browsing, downloads, and internet games were all fine. An example: replicating 5 small emails (no attachments or images) took at least half an hour on an ADSL connection, where it is normally less than a minute.

   They changed the configuration again about a month later (again unannounced), and both Notes and Ventrilo returned to normal operation. The problem has never recurred.

   (Evidence that it wasn't just my PC/connection: multiple customers of the same ISP reported trouble with Ventrilo in the same time period. See { Link })

1. 17  Flemming Riis  | 9/13/2007 3:54:42 AM

   @14 - This is bad. With an opportunity to get worse; e.g. M$ gives $ to ISP, not Google and Notes are disadvantaged relative to WindowsLive.

   did you loose your tinfoil hat.

   If comcast is throttling everthing it would hurt microsoft apps just as much.

1. 18  Nathan T. Freeman http://nathan.lotus911.com | 9/13/2007 4:57:26 AM

   Here's an explanation of the suspected technique that ISPs use to throttle bandwidth consumption: { Link }

   Here's an article about what Comcast customers think is really going on: { Link }

   Please note, you CANNOT determine whether this is happening to you without TCP packet analysis. Firing up NPING is not going to cut it. You'll need to have lengthy conversations that move sizable amounts of data on ports higher than 1024.

   If I remember correctly, Comcast has not yet admitted to even HAVING a Sandvine implementation, so you'll be hard-pressed to get them to admit that they're doing anything to any particular port.

   The best move is to make it expensive for them. If you encounter a problem, call their 800 number and make sure you stay on the phone for as long as possible with a support rep. And make sure you tell them that your buddy next door has a DSL line that's much much faster and you're thinking of switching.

1. 19  Flemming Riis  | 9/13/2007 5:01:01 AM

   @18 is it possible to move notes and domino to a port below 1024 to test it.

1. 20  Jim Casale  | 9/13/2007 5:17:07 AM

   Call me a pessimist but I think you have to start thinking of ways around it. It doesn't sound like Comcast is going to listen. I just got Fios and I made sure I got the business class so as not to worry about any of the BS that the broadband companies like to pull

1. 21  Lee Davis  | 9/13/2007 6:06:49 AM

   @17 VOIP is at the forefront of the issue because it is sensitive to performance and in an area that Telcos/Cables are trying to make money themselves.

   But after Whitacres comments last year, I certainly read concerns about vendors payments being used to affect bandwidth:

   from CNET

   "But some people fear that AT&T and other network operators, such as Verizon Communications, may abuse their control of the network. While they may not block traffic outright, they could limit the available bandwidth to degrade the service of competitors or companies choosing not to pay extra fees to enhance their service."

   from freepress

   Whitacre said, “They don’t have any fiber out there. They don’t have any wires. They don’t have anything. They use my lines for free — and that’s bull. For a Google or a Yahoo! or a Vonage or anybody to expect to use these pipes for free is nuts!”

   For the first time, Whitacre was talking publicly about charging the Internet companies a fee to reach an SBC customer. And if Google or Yahoo didn’t pay up, he implied, they would presumably find their Web sites more difficult to reach. Technology geeks immediately saw Whitacre’s words as a threat. “It certainly has the feel of extortion,” said the TechDirt blog. “Be afraid. Be very afraid,” said telecom consultant Kevin Werbach, a Clinton administration telecommunications official.

   But hopefully you are correct and it unthinkable that Telcos would abuse their monopoly nor that Microsoft et al would use mere money to compete against upstarts.

1. 22  Charles Robinson http://cubert-codepoet.blogspot.com | 9/13/2007 8:07:44 AM

   @18 - I've done packet captures. It's a TCP RST at around the 2MB mark. I set up a test server using a port below 1024 (as @19 suggests) and it worked just fine. The same computer would get a dropped connection on the 1352 server and it would work just fine with one on port 24.

   @21 - That's why net neutrality has become such a hot issue. :) Unfortunately the FTC and FCC are so full of lobbyists and cronies that nothing good will come out of either of them.

1. 23  Irv Schor  | 9/13/2007 8:47:26 AM

   For the suspicious and admittantly paranoid, or perhaps those still wondering about 'could it be true', etc., perhaps the $1 Billion Microsoft had just invested in Comcast about this time frame helps determine what kind of protocols get filtered by their systems.

   { Link }

1. 24  Daniel Lieber http://www.iiui.com | 9/13/2007 8:54:59 AM

   Expanding the Options - How to Win against a Cable Company

   If you have cable modem service and suspect this type of behavior by your provider, there are some remedies available. As Nathan Freeman points out (@8), you should isolate the problem to the Comcast network, which is not trivial. Networking is inherently complex and thus isolating a root cause is both challenging and important.

   Next, check your Terms and Conditions to be sure you are not violating them (depending on your service and market, certain behaviors are prohibited such as running servers). Next, contact your City/Town/Village Hall to determine who is on your local Cable Advisory Committee. These committees are legislatively specifically permitted in nearly all states to be the primary negotiators for cable service licenses on behalf of municipalities. In most licenses, service expectations are listed. Deception, whether intentional by the representative or not, would put the cable company in violation of both your personal contract and the municipal license which permits them to do business in your area. The Cable Advisory Committee can probably help you find out what is happening if you submit your complaint in writing requesting a written response. Be sure to send it to Comcast as well. If it is ignored, you do have the right to sue them for failure to meet the terms of the contract and possibly deceptive business practices for advertising the service to home office users with the intent of blocking the service after undisclosed limits are passed (thus resulting in punitive damages).

   In my experience, just proposing a lawsuit against a cable company has not produced results, but filing a claim does (including in small claims court, which is surprisingly easy and does not require a lawyer). Keep your municipality informed as these issues really do affect negotiations over licenses in a positive way for customers. Comcast is a large company and is not infallible. They make mistakes like others.

   Using the velvet glove/iron fist approach works well. Be courteous, clear, and explicit with your problems and needs. Document everything and you'll get resolution; it won't happen in a day, but really can work.

   Background: My experience with this issue is personal. I am an appointed member of my local town Cable Advisory Committee and have helped resolve similar issues for customers, including myself. In my case, I did end up suing a local cable company in small claims court for failure to provide service, and won a judgement (and collected) including the service fees, court costs, and other secondary costs and interest totaling over $2,000. They are also being investigated for violations of their license which could result in sanctions including the loss of their performance bond $100,000.

   I hope this information helps!

1. 25  Alan Dalziel  | 9/13/2007 8:55:32 AM

   Send all your info to consumerist.com and they will most likely help you get to the people who make the decisions at Comcast. A great site to improve your skills at getting customer service to actually serve customers . . . .

1. 26  Nathan T. Freeman http://nathan.lotus911.com | 9/13/2007 9:01:11 AM

   @23 - Would you like some padding for that tin foil hat?

   MS's investment was a decade ago.

1. 27  Rob McDonagh http://www.CaptainOblivious.com | 9/13/2007 9:32:42 AM

   @Ed Based on @22, one would think that IBM might have a legal interest in the matter? Maybe? Comcast is preventing your customers from doing business. Can you release the hounds (aka IBM's lawyers)? Comcast can ignore individual customers easily, but ignoring IBM and its legions of lawyers is an entirely different risk analysis...

1. 28  Charles Robinson http://cubert-codepoet.blogspot.com | 9/13/2007 9:56:17 AM

   @26 - I think Irv was referring to the investment coming on the heels of the Telecommunications Deregulation Act of 1996. Even with that bit of knowledge it's still a little crackpot to trot out a decade-old investment to explain business practices today.

   @27 - I got the same response from IBM/Lotus as Kevin has. They determined since their product was working it was an issue I would have to take up with Comcast. Incidentally, I added a response to Kevin's forum posting detailing what I did to resolve the issue.

   I want to make it clear that Comcast is not singling out Domino traffic, it's just collateral damage in Comcast's traffic shaping and filtering. I had success getting individual POP's changed, but I never attempted to take on the Comcast bureaucracy head on.

1. 29  Irv Schor  | 9/13/2007 10:11:04 AM

   @26 Thanks for the correction... I just don't want to turn 40 next week. Too late, Alcheimer's has already set in ;-)

1. 30  Kevin Mort  | 9/13/2007 10:27:12 AM

   @All - is anyone using a Comcast@Work account and having these issues or is this restricted to residential accounts?Just curious. There has also been a good bit of discussion on these issues in the Comcast forums at Broadbandreports.

   @21 - Without trying to get into some big NetNeutrality discussion here, the problem with Whitacre's comments is that those orgs ARE actually paying for bandwidth from someone. They aren't getting a free ride like he contends.

   But I think he's gone now if I recall correctly...

1. 31  Barry McGovern  | 9/13/2007 11:35:18 AM

   This problem also appears to be causing some instability on the server. I've had some issues in the last few weeks when trying to replicate 1,000's of documents and the replication stalling, then the server crashing.

   Before this discussion, I was assuming the problem was a corrupted database. This morning, I forwarded this comcast discussion to our development team, and found out the same thing happened to another developer on a completely separate database on separate server (7.02). She is in California, I am in Maryland. The only similarity is Comcast and replicating 1,000's of documents.

   If it's only 1 large document, it seems to be fine (no server crashes). I've only seen the server problem when I needed to replicate 1,000's of documents.

1. 32  David Price  | 9/13/2007 11:51:08 AM

   @30 I have the same issue with a Comcast@Work account. I have also seen replication blocked. I had to relocate to another location to complete replication.

1. 33  Ed Brill http://www.edbrill.com | 9/13/2007 1:55:11 PM

   @28 I'd be willing to explore executive contact and/or other options, if I can get a lot more data. I also e-mailed Kevin to find out what IBMers his execs have contacted. He and I e-mailed a couple of weeks ago, but I admittedly let the ball drop.

1. 34  Flemming Riis  | 9/13/2007 2:25:10 PM

   @22 Good Find.

   prehaps 8.0.1 will have resume in the protocol :)

1. 35  Charles Robinson http://cubert-codepoet.blogspot.com | 9/13/2007 3:19:11 PM

   @28 - I'm no longer working with that company and I don't have access to the documentation. I managed to get the locations my users were connecting through to update their filtering software, and we used a VPN to help mitigate it further. Reading through the various forums it does sound like Comcast has been changing some stuff around in the past month or so.

   @34 - It's not a matter of having resume in the protocol. Comcast is modifying the packets so client software thinks the server has canceled the connection. In hacking circles this technique is called a "man in the middle" exploit, where you insert an app between the client and server and modify packets to make both ends think something else is happening. There isn't anything Notes or Domino can do about that.

1. 36  Lee Davis  | 9/13/2007 3:30:55 PM

   @35: I do not understand. If you were to FTP a 1GB file from a resumeable server, then eventually it would get the whole file, albeit less efficiently since it would have to do it 2MB at a time - 500 restarts.

   Similarly, I can envision Lotus software not discarding the 2MB, but resuming from where it left off to eventually get the whole thing.

   The irony is that the technique described wil not prevent p2p software since it will get chunks from other peers. So it seems to me it will hurt "legitimite" users and use more bandwidth for the p2p traffic due to increased overhead.

1. 37  Carlos Rivera  | 9/13/2007 5:40:38 PM

   Yes they turned off filtering.

1. 38  Nathan T. Freeman http://nathan.lotus911.com | 9/13/2007 9:03:47 PM

   @35 - Actually, technically, this is not just a man-in-the-middle attack. Man-in-the-middle involves intercepting EVERY PART of a conversation, usually for the purpose of impersonation. This is a spoofing attack that's not targeted at eavesdropping, but at simple denial of service.

   I'm taking your word for it that you've actually observed the TCP RST packets, but if you have, you're totally right -- this is a Sandvine conversation reset that will attempt to abort ANY conversation that's larger than 2MB.

   @36 - An aborted TCP Sequence in the middle of > 2MB will think it needs to start the transaction over from scratch. Only if the protocol supports a resume from a point in time does it work.

   P2P programs still work in part because they generally break into 1MB chunks. But once you get 2 chunks from a source, your conversation is reset. To the P2P program, that's as if you disconnected, and since most of them have to throttle requests, the bandwidth is allocated to someone else. Then you have to find a new donor, slowing your process.

   FTP isn't plagued by this because they only apply it to ports > 1024.

   Try passive mode FTP. It'll start failing over 2MB.

1. 39  Kevin Kanarski http://kkanarski.blogspot.com/ | 9/13/2007 9:34:34 PM

   I have taken Ethereal packet traces and you can see the stream of TCP RST packets being sent to the client. We have also done a trace on the server side and the server isn't sending these packets which only leaves the ISP.

   After about 1Mb of data is sent the RST packets start. Notes will tolerate a couple of them and re-connect but after about the 3rd stream of RST packets Notes gives up.

1. 40  Bill McCuistion  | 9/13/2007 11:25:45 PM

   Maybe this is a little off subject, but wanted to chime in. Comcast seems deceptive in its network practices on numerous services, several of which affect a Notes client and/or Domino Server, beyond port 1352.

   For example: Comcast & Reverse DNS Lookups.

   I like to use DynDns.org to provide DNS services on a dynamic basis. Makes switching ISP's easy, and is useful in disaster recovery situations.

   Several months ago, shortly after Comcast took over Time-Warner's cable operations, some parts (services and functions) of my network stopped working.

   After working with DynDNS.org support, (very good folks, BTW), they reported that Comcast was blocking "Reverse DNS lookups", as required by SSL-type connections (e.g. HTTPS, SMTPS, etc). This was in Comcast's attempt to prevent consumer network segments from having "server" services.

   Unfortunately, there are several "well-written" services that require bi-directional SSL authentication. (e.g. Soap).

   So, to the extent that a Notes client acts like a "server", then Comcast may interpret this as something it is not. (e.g. Pull, then Push replication or Mail Routing).

   Remember, there's a lot of AOL in Comcast, so their view of the Internet is somewhat skewed.

1. 41  Ed Brill http://www.edbrill.com | 9/14/2007 9:58:14 AM

   @All, what great feedback and work. Behind the scenes, much has gone on as well.

   If you have experienced this issue and have data to confirm the scenario, please contact me @ work (ed_brill at us.ibm.com). A workaround is being examined.

1. 42  Chris Whisonant http://cwhisonant.blogspot.com | 9/15/2007 10:27:43 AM

   I am on Comcast residential cable modem in GA. I just saved a draft as well as sent myself an email that has a 2,456 kbyte attachment with no problems.

1. 43  Keith Brooks http://lotustech.blogspot.com | 9/15/2007 8:05:56 PM

   I also have comcast, which was Adelphia. Since their take over a few months back we have had many issues, from DNS resoluton failures to not wanting to issue an IP to my router because they do not support home networks.

   Since resolved that issue.

   However we have seen our P2P drop heavily on the UPLOAD side only. Download is faster than ever.

   I use a Notes client(Standard R8 now) for pop3/imap and notes mail/replication. Some emails with larger attachments have definitely been getting stuck and I put it down to the beta R8 client.

   If there is a test we can all run, let us know.

   BTW, as much as Adelphia sucked, Comcast makes them look amazing.

   Didn't want to go DSL but might have to now.

1. 44  Steven  | 9/19/2007 10:59:44 AM

   At least we know the US as a whole sees that Comcast is one of the worst ISPs...

   { Link }

1. 45  MarvinK  | 9/20/2007 10:50:59 AM

   I don't think it is unreasonable for places to block above port 1024. If companies want to provide reliable access to users outside of their internal network, it seems like they NEED to provide an alternate option. Seems like there are 2 easy solutions:

   - iNotes

   - VPN (that works over 80 or 443)

   It is impractical to expect every location to allow 1352 out.