The problem with unsophosticated customer support

Do not enter internet
Photo by: Joshua Hoehne @mrthetrain

False positives causing a legitimate application to be blocked is a common problem with security software, and if not handled properly and quickly, it is one that could hurt, or even destroy a security product’s credibility, or in the worst case, the credibility of the entire sector.

It is therefore very important that whenever a security vendor’s product is incorrectly flagging a legitimate product that the vendor resolve the issue within hours, or at most a couple of days of being notified about the problem. Such problems should really be handled with a priority just barely short of problems threatening the customer’s system (like security vulnerabilities).

If a user cannot use their chosen, legitimate products because a security product blocks it, they are far more likely to disable, or uninstall, the security product, than to change their chosen product.

If the problem is caused by some actual problem with the flagged product, the security vendor should immediately contact the application vendor with detailed information about what the problem is, and how to solve it.

Easier said than done

As an example of how to not go about handling such cases, consider this recent case.

About a month ago, in early September, the Vivaldi users at a small German company discovered that they were no longer able to use Vivaldi, since their Sophos firewall was blocking it.

They contacted Sophos customer support and were effectively told that «The block was a management decision», «Vivaldi does not support content filtering», «Vivaldi does not support a required API», «Submit a feature request, we can’t do anything before we receive that» (the latter had been filed over a month before this case started).

No information was provided about which API support was «missing», or why «management» had decided to block Vivaldi.

Since Vivaldi is based on Chromium, just like Google Chrome, if the blocking was really due to missing support for an API, then Sophos should be blocking Google Chrome as well. We have the same feature support as other Chromium-based browsers. The only real difference is that (e.g. on Windows) our executable is named «vivaldi.exe», not «chrome.exe» and our UI is implemented differently.

After receiving the replies from Sophos, one of the users in the company reported the problem in a post to our German language forum, and it was then forwarded to those of us in the security group.

I decided to look into the Sophos support site, and did find their chat support, but after two hours of back and forth, being passed from one person to another, their response was effectively «We need a support ticket number, file it from the upload site».

There were several problems with that upload site, mainly that there was no option to upload a file as «Affected vendor». You had to be either a «registered user» or «evaluating before purchase». It was also difficult to choose the right product or product category, and the upload size limit was 30 MB (Vivaldi’s installer is ~55MB), although an FTP option existed.

Since I could not upload Vivaldi’s installers, I uploaded an empty text file, and told them in the message where to get the installers. Their Labs people explained that they were not allowed to download installers from the Web.

After an FTP upload, and a few days wait, they reported that the «problem has been fixed».

The users said «No, it hasn’t been fixed».

55+ emails back and forth later (to Sophos and the user), direct involvement with the customer, and 5 weeks after this all started, the problem still hasn’t been resolved. Effectively, they have acted like a brick wall.

In my opinion Sophos has not handled the case well. They never told us, or the customer, what is causing the problem, and they have so far spent at least 5 weeks not fixing the problem, so they definitely did not drop «everything else» to solve it.

I recommend that all security software vendors check their processes to make sure they can handle false positives quickly and efficiently.

Problems I have seen during the process with Sophos

  • The support people kept assuming I was the customer using their product, and repeatedly asked for information I could not possibly provide. My suggestion is that they create a separate support ticket category for application vendors.
  • They were unwilling to contact the reporter via the forum thread, saying they were not allowed to do support except through their issue system. My suggestion is that they communicate with reporters through the reporters’ chosen channels, and then invite them to use the vendor’s own channels. This will improve the impression of their customer service.
  • As mentioned, the upload system is not suited to normal-sized applications, or affected vendors. The minimum size should be increased significantly, and I think they should offer SSH upload via SCP instead of FTP.

An unsophosticated test

While working on this article, I started thinking about the question of exactly how Sophos blocks Vivaldi. My conclusion based on what I know about other firewalls, was that the most likely method is to just check the process name which, as mentioned above, in our case is «vivaldi.exe» on Windows, not «chrome.exe». It could be that they are doing something more sophosticated, but I doubted it.

So yesterday I created a special version of Vivaldi 2.8 where I undid the changes that rename our Windows executable to «vivaldi.exe». Even if this experimental build would not be able to get through the firewall, we would learn something about just how sophosticated Sophos’ implementation is.

This morning we sent this special build to the reporter and asked him to run a quick test for us. He has just reported back that the special build was able to access the internet through the firewall.

For other affected Sophos users, the special build (which works as a Snapshot channel, so you might want to disable updates for this particular installation) is available for download here. It should be installed as a standalone version using the advanced installation dialog, NOT over the main Vivaldi installation.

Similar cases from the past

This is not the first time we have had similar problems, either in Vivaldi or back when many of us worked in Opera, and they are usually resolved quickly, without much publicity. For the most part an exchange of a couple of emails were enough to get the problem solved.

There were two cases that didn’t get resolved quickly, and which required a bit more work. One was the old 2003 Opera Bork edition targeting Microsoft and MSN, and the 2016 Vivaldi case when some AV software decided they did not like «Vivaldi Technologies AS» as a text string in our installer, «Vivaldi Technlogies AS» (without the first «o»-letter) worked fine. In both cases our public response caused the issues to be resolved very quickly.

In a more recent example, Eric Lawrence from Microsoft’s Chromium Edge team was trying to chase down why recent versions of a Chromium support executable was triggering warnings from a significant number of Anti-Virus scanners. Although he never actually found the problem (it disappeared in newer builds), as he closed in on what triggered the problem, it started to remind me about our 2016 case, which is why I sent him a link to our 2016 snapshot announcement, and it subsequently made a short appearance on Twitter.

14 thoughts on “The problem with unsophosticated customer support”

  1. Hey, thank you for your time and effort.
    I hope something «happens» soon so I can use it again. Especially after so much effort and time.

  2. So to get things clear you believe Sophos should contact every Application Vendor in the WORLD EVERY TIME a new application is created and used by a machine using Sophos Endpoint or one of the Firewalls?

    Doesn’t that sound stupid? Is that how you believe Applications / Detections work?

    Can i ask what you’ve ACTUALLY done at Vivaldi to work WITH Sophos to resolve this issue?

    From this post you’ve just complained that Sophos a world recognized business doesn’t like you software due to security reasons and you’re not happy — I would assume the Security company would be the leading authority in if this is safe or not.

    Did Sophos offer to complete testing and troubleshooting on an endpoint or did they just refuse to help?

    As you’re not a Sophos customer — do you expect to be able to raise a Support ticket without an account and get a paid service level of support? Didn’t you ask YOUR customer who has an issue to raise a case so they could have Sophos troubleshoot the issue?

    1. Perhaps you have not read the article, as you seem to be arguing something completely different?

      It seems that the Sophos customer and Vivaldi between them repeatedly reached out to Sophos, but as is the case with many large corporations, «the right hand doesn’t know what the left is doing» — therefore no individual within Sophos had the complete knowledge, competence or authority to be able to fix the issue. Additionally, the systems and tools they have in place are too restricted and crippled to adequately report such issues. Finally, when they believed they had fixed the issue, it remained.

      Additionally, nowhere does it suggest that an AV vendor should contact every software manufacturer in the world. What is acually proposed, is a very pragmatic and sensible solution: That AV vendors simply make it easier to report false-positives!. In fact, many other vendors make it this easy already!
      Sophos was incorrectly detecting Vivaldi through a broken or over-generic detection-rule, which could also be bypassed quite easily. For Sophos, this should be extremely embarrassing. Quite clearly, this is a false-positive, therefore it should be corrected regardless of whether Sophos are «the experts» and more qualified to determine if a file is a security-threat, and regardless of whether Vivaldi are a customer or not. The fact that they are experts does not equate to them being infallible, and the request to remove a false-positive that would inconvenience numerous users of their software is in no way attempting to get a «paid service level of support». On the contrary, I would say that this is the minimum basic level of support that a responsible AV vendor should provide for free (and most I’ve encountered already do), as it protects the AV vendor’s reputation and stops them losing their existing-customers!

      I don’t use Sophos, but had been considering them. What I can say is that for the moment I will probably stick with my current security vendor, who responds well to false-positive reports and produces stable, reliable software. I will still consider them in the future, as I know the protection to be largely «pretty good» — but for the moment I will stick with a vendor who provides adequate support rather than excuses.

    2. I said they should contact vendors when there is an issue that have to be resolved on the vendor side. If a legitimate application is used by enough users they are going to be swamped quickly with reports, so it only makes sense to help the vendor’s product work if something have to be done on that side. (And even if they are not proactive, the vendors will quickly come banging on their door, like we did). If they don’t help, then it is likely that vendors will tell users to change to a different security product.

      A security product like Sophos’ should NEVER block a legitimate software product. Their goal is to block malware and their connections, or access to bad sites. If they think making sure legitimate application work is too much work they are in the wrong business (any they are also likely not going to stay in that business for long if they work that way).

      One way they can reduce overhead deciding legitimate vs questionable software, is to check code signatures; e.g. Vivaldi for Windows is signed with an Extended Validation Code Signing certificate, which is automatically accepted by Windows Smartscreen. (It is not foolproof, there have been cases of stolen signing keys, but it is a reasonably good first filter)

      In this case, Sophos has made no claim that we are insecure. They are blocking us because they claim, *incorrectly*, that we don’t support an API (apparently for blocking URL loading) . That goes beyond legitimate vs questionable software, into technical requirements that they require third-party vendors to comply with. Then it is on Sophos to work with those vendors to help them work with Sophos’ products, assuming they don’t already (which we do, they just don’t recognize it).

      In this case, regarding what we have done:

      1) The customer have repeatedly contacted Sophos, but got no help. See the quotes in the article. Effectively they were telling the customer that they blocked Vivaldi because is supposedly did not support an API (but it *do* support that API, actually)

      2) The customer then contacted us, 5 weeks ago, we then contacted Sophos through multiple channels

      3) We have repeatedly asked why they block us, what is needed to be unblocked, without getting a response, and we have provided installers that they could use to test.

      4) We have specifically told them that Vivaldi supports the same features as Chromium and Google Chrome. They have never responded to this assertion.

      Results so far: 5 weeks averaging 8+ emails back and forth each week, no results. Vivaldi is still blocked (unless we «cheat», as mentioned in the article).

  3. Hi,

    Just to get things clear I have downloaded and tested this browser myself now installing on a test VM machine, I installed sophos Central endpoint and put it behind a xg firewall.

    So here are the results, hope it helps :
    The Sophos Central doesn’t block this download or installation by default.
    xg firewall doesn’t block the browser download or installation with standard settings enabled

    When installed with default settings the browser (Vivialdi) — is able to bypass the Firewall rules and access sites which should be blocked.
    Testing with Chrome and IE at the same time — website block successful.

    I would like to ask why the browser works differently from Chrome and IE if you said this uses the same features?

    1. As I have said above: We have never been told why we are blocked, and which API/feature support is supposedly missing, or how Sophos determined that, or how it wants to access that API/feature.

      As for why different, one possibility is that Sophos does not recognize Vivaldi as a Chromium based browser.

      Did you test with the standard Vivaldi version, or the special standalone build I linked to?

  4. I used the standard link i found from the official download.

    So that Special version download link you have above — from whats been said this specific version works correctly and doesn’t bypass the firewall WAF. I haven’t tested this yet but i will test and report back results. In addition will this become a standard download link and be updated/patched or only a once off?

    As for why Sophos doesn’t recognize Vivaldi as valid browser, i would guess because as mentioned the standard download (for some reason) bypasses the Sophos firewall making it an «unsecure browser» where the other browsers don’t bypass the Firewall rules.

    As per my testing earlier i wouldn’t think this browser does work like Chrome or IE as with the basic download / install it doesn’t work in the same manor.

    As the company who created the software can you tell me what’s different?

  5. Hi yngve,

    why are you focusing on this „we are being blocked“ or the API thing?

    Do you have any official statement for any of these assumptions or is this all just based on „person a just told me that a friend of person b heard that it is blocked.“ chitchat?

    As someone who wrote such a long article I‘d expect that you have tested an verified what you wrote about.

    So my question for you are

    1. Have you actually tested and verified on your own about what you are writing about?

    2. If you have not tested it — how could try to raise a ticket with the vendor? (From what i‘ve read it looks like several other things did not work well on the vendors site but that is something different)

    So my assumption for question 1. is — you have not tested it and it would be great if you have the courage to state this here.
    And if you have tested it it then should be easy for you to let us know how to replicate this blocking issue.

    I have tested it in several scenarios but there is only one where I can replicate. This is when I tell the software to explicitely block Vivaldi. Why might I want to do this? Yeah for the same reason I tell configure Sophos to block Safari (why by the way is a fully supported browser) — I want my employees to just use Firefox (for example) cause I have set up a corporate policy that no other browser must be used in my network 🙂

    Just wonder if the Vivaldi Team always reacts like this if someone writes something in their forums 😉

  6. It is frequently not possible to actually test a reported security software, nor do we have the extra resources to do so except in special cases.

    We based our questions to Sophos on what was reported to us by the user, which was that Vivaldi was being blocked, the reason was reported to be due to at least the supposedly missing support for an unidentified API.

    The basic problem was that, despite approaching Sophos via multiple channels, prior to our article above, we never received enough information from Sophos to understand what was happening and why.

    In the past day, though, the case have been escalated much higher within Sophos, and we have received a lot of information about what really caused the block that clears away most of the confusion that has been a problem in this case since before we heard about the problem (short summary: The customer admin added the block, at present we do not know why; this information was never communicated to us before this afternoon).

    We are still missing information about the supposedly missing API support and how Sophos wants to access it. That is next on our agenda of our discussion with Sophos.

  7. Fully understand your point and I know that it is hard up to sometimes impossible to free up ressources to test stuff that actually is more or less none of your business but you wrote this blog entry and it starts with some hard accusations about the entire security software sector based on things you‘ve just heard.

    In my opinion this is a very bad practice.

    As when reading this blog without reading the comments I get a wrong impression about whats actually wrong.

    Nevertheless your support experience seems to have been pretty bad especially if it happened the way you have described it.

    At least it is good to hear that you finally seem to get the information from Sophos you are looking for.

    Would be great to get an update on what the actual problems are, the results of your discussions and how this is planned to be resolved (if there are any plans).

    Regards

    Chrishi

  8. I would like to briefly describe the situation from my side:

    A few months ago I was able to use Vivaldi with Sophos. It has been blocked since the beginning of September. I can’t say what the trigger was or why it’s being blocked now.
    I then asked our admin at the company. He contacted Sophos and tried to clarify the situation (allegedly because Vivaldi does not support any API). Then I asked in the Vivaldi forum if something was already known and created a ticket for Vivaldi.

    Then there were some messages (and tickets, phone calls) between us and Sophos and me and Vivaldi.
    At some point Sophos told us that it was a management decision and that we should write a feature request (as mentioned above).

    The only thing our admin can do (locally) and has done for a short time is to create an exception for Vivaldi. He doesn’t want this exception to be permanent. He didn’t block Vivaldi locally (why should he?).

    To test: I don’t know anyone else who can reproduce the problem and I don’t know how much effort it takes to reproduce it. But for me it is very simple: I click on the Vivaldi icon to see what happens: Either I get an error message or Vivaldi starts. So it’s no effort for me and I like to do it for Vivaldi and the team.

    Regards

    1. As I currently understand it, that is a list of browsers known to work with the *feature* discussed on that page, not a general list of browsers that are allowed through the firewall.

      There is, generally, a potential for «we support X» to be understood differently by the author and the reader, with the author in many cases intending to say «We answer questions about X, but not Y», while the reader may understand it as «Only X will work, not Y», instead of the intended «if you use Y, you are on your own».

      In this particular case, my information is that this «supported» section means «X works with Feature A, Y may or may not do so (probably not), but it will not be blocked by the firewall» (it might for example be that feature A needs to know installation details about each supported product X in order to work).

  9. This must be terribly frustrating for you.
    As a Sophos Anti-Virus user I can honestly say I have never had a problem with the AV software and Vivaldi.
    — it downloads the Vivaldi Browser without any exceptions
    — It allows for installation/updates of the browser again without any exceptions

    As a Home User I do not use the Sophos Firewall and that is a whole other matter. But I do not see how they cannot have had this issue resolved by now…
    (As a single user) I have had a few issues with the AV and was always escalated to the proper person to resolve my issues (even though they do not provide the same level of support to Home Users as to Corporate Users). Sophos personnel have always been both courteous and professional. That was one of the reasons I switched to their product.

    I wish you Good Luck that the issue will be resolved A.S.A.P.

Comments are closed.