Spam Blocking & Operation Gemstone

Those of you who manage email servers at a similar level to myself will have noticed a huge increase in malware-infected spam during 2013.

In fact it has got to such a level that it was becoming unmanageable without a recognised Spam filtering application. And with no plans to venture into the realms of SpamFighter, GFI MailEssentials, Baracudda or others I decided it was time to go all on an all out spam war.

Part 1 started earlier this year with a number of emails being received relating to Stock Market purchases and “upcoming targets”, these were obviously spam, and so started “Operation Gemstone”, so-called due to the first set being related to a Gemstone Mining Company. This was becoming a nuisance for all staff and so we started blocking emails by familiar key words, i.e once we had 3 or 4 of a similar nature we were able to deduce a keyword that we could block and that wouldn’t block (too much) valid email. We started with a transport rule “Gemstone” and we now have 5 of these! The Gemstone rule set blocks key words found in either the subject or body, it excludes emails sent to the boss (who manages his own spam) or from an internal address, and rather than deleting, it redirects the message to a holding account as a quarantine where we can forward on false-positives if necessary.

Part 2 came about after analysing hundreds and thousands of spam emails collected over a number of months and actually looking at the message headers to find more similarities between emails of a seemingly different subject matter. Naturally, the first thought was the source IP, and in some cases we found multiple occurrences of the same IPs, however on the whole they were different every time. (Where we found similar entries we blocked them at firewall level) So the one area we found similarities was in the “Return-Path” message header, with a huge number coming from addresses pretending to be American Express related (aexp.com etc.) so then came our second rule set “Return Path Block”, this again was a transport rule with a redirect to a holding account, the difference this time was to set the rule to read the message headers and look for a “Return-Path” containing various phrases. This rule was so successful that we could turn off Gemstone’s 1-3 meaning less load on the Exchange Transport servers.

But then, another realisation hit, as the months have gone on this year, the spam was becoming more and more convincing, apart from one thing… Zip attachments! On instructions from above I blocked all incoming Zip attachments (by redirect, again). Since 9.05 Monday 18th November 2013 (7 Days) 1208 emails have contained zip files and have been redirected to the quarantine account. Of these only 4 have been genuine files meant for our staff! so our “Zippy” rule sits at the top of our transport rule set and does its job admirably.

Sure these methods may seem a little archaic, but I see a couple of advantages:

  • By redirecting rather than deleting at source it gives a chance to filter through emails to ensure nothing is missed
  • By using built in Exchange rules instead of a 3rd party tool adds less overall load to the Exchange servers (from our experience)
  • We can add new keywords, return-path sources or attachment types instantly
  • The transport rules allow us to TAG the emails, by pre-pending with for example: <BLOCKED – Gemstone Rule> or <.zip file attachment> allowing us to filter emails within Outlook

If anyone has any comments about all this I would love to see them, please feel free to contact me.

 

 

Updated Batch Installer Script

So it’s been around a year since I first created the automatic installer script for when we deploy new machines. And it has been a huge success, cutting down deployment time significantly, and meaning we can easily do more than one at a time.

I realised that as it’s been a year, and every product under the sun has a vulnerability in it, that I needed to update the script for the newer versions of the standard products we all know and love. Most notably Adobe Reader, Flash, Java Runtime and Microsoft Security Essentials. On top of these newer versions I also decided as our core business is based on JD Edwards, and it now supports Google Chrome that we should also have that included too.

Another addition that has been made is a simple file copy to create a new folder and put in standard internal documentation that helps people connect from home/customer sites and Accessing our proprietary software.

The latest updates are as follows:

@ECHO off

echo Installing Adobe Reader…

rem “[location]ProgramsStandard SoftwarenewpcAdobeRdr1000.exe” /sAll /msi /norestart ALLUSERS=1 EULA_ACCEPT=YES
rem instructions for compiling network installer can be found at: http://community.spiceworks.com/how_to/show/27718-install-update-adobe-reader-to-to-latest-adobe-reader-xi-11-0-2-using-cmd
msiexec.exe /i “[location]ProgramsStandard SoftwarenewpcAcroRead.msi” ALLUSERS=1 /q /norestart TRANSFORMS=”[location]ProgramsStandard SoftwarenewpcAcroRead.MST”
msiexec.exe /update “[location]ProgramsStandard SoftwarenewpcAdbeRdrUpd11003.msp” /qb /norestart
echo …done

echo Install ActiveX Flash…
rem “[location]ProgramsStandard Softwarenewpcflash_iexplore.exe” -install
msiexec /i “[location]ProgramsStandard Softwarenewpcinstall_flash_player_11_active_x.msi” /qn
echo …done

echo Install Browser Flash…
rem “[location]ProgramsStandard Softwarenewpcflash_firefox.exe” -install
msiexec /i “[location]ProgramsStandard Softwarenewpcinstall_flash_player_11_plugin.msi” /qn
echo …done

echo Installing Java Runtime…
rem “[location]ProgramsStandard Softwarenewpcjava.exe” /s
“[location]ProgramsStandard Softwarenewpcjava_new.exe” /s
echo …done

echo Installing Microsoft Security Essentials…
rem “[location]ProgramsStandard Softwarenewpcmse.exe” /s /runwgacheck /o
“[location]ProgramsStandard Softwarenewpcmse_new.exe” /s /runwgacheck /o
echo …done

rem –install chrome–
echo Installing Google Chrome…
msiexec /q /i “[location]programsstandard softwarenewpcchrome.msi”
echo …done

rem –Copy Useful Docs–
echo Copying BMS Useful Documents…
mkdir “%USERPROFILE%DesktopBMS Guides”
xcopy “[location]batch scriptsDocs*” “%USERPROFILE%DesktopBMS Guides” /Y /q
echo …done

rem –DONE–
echo Installer Complete!
pause

NOTES:

I have left in the older code to compare the differences (if any) in the install commands.

The Adobe installs are now done using a great little utility which you can download from the Adobe website, this helps create a transform file which is basically a set of rules that define the install parameters. The adobe reader install needs a base install of Version 11 and then the update package on top, at time of writing update 3 is the latest, in theory when update 4 is available you should be able to download the file to the standard location and then change the code to reflect this version. The instructions I followed can be found here: http://community.spiceworks.com/how_to/show/27718-install-update-adobe-reader-to-to-latest-adobe-reader-xi-11-0-2-using-cmd

The Google Chrome MSI is available from here: http://www.google.co.uk/intl/en-GB/chrome/business/browser/admin/

For security purposes I have modified the file paths for this post.

 

 

 

Busy Times

So, as 2012 draws towards its close, I sit here, typing on a funky iPad2 with a bluetooth keyboard,
wondering why it feels busier than ever.

This latter part of the year has seen me running (virtually on my own) the infrastructure of 2 very different companies. Firstly, my career job, the home from home that is my Infrastructure Manager position at BMS. A very successful year in infrastructure terms, despite the network outages a few months ago, and the conficker spread in March it has been reasonably quiet. That said the workforce has practically doubled and the server rooms have seen numerous additions. The company’s ambitions continue to match my own and 2013 looks set to be even busier.
Secondly, Redsumo.com, the friendly, local web design, it services and hosting company, founded in 2010 and run out of an office in Alcester with servers hosted on the interweb. Well, what a change we’ve seen there. With an office move into Studley and an even more ambitious server migration to locally hosted servers, a mere few feet from my own desk! Redsumo has continued to take a lot of my time, with building, configuring and maintaining the web and email systems as well as helping with the migrations and even a fair bit of web coding, something that has always been a passion of mine. Redsumo has become more of a Web Development & Hosting company over the last few months, with the IT services gradually phased out. The site has seen a revamp, with new products sitting alongside the old and trusted services that have been provided since day 1. A new WordPress based blog has also been developed to provide a news platform for existing customers to share their stories and prospective customers to see into what we do and how we do it. The Blog project has been something of a success for me, practically recoding an entire theme with many customisations to make it like no other out there. Whilst on the outside it looks plain and simple, I
assure you underneath the hood there’s a lot of clever things going on!

2013 looks set to be a busy, and hopefully compelling year for both companies, and therefore for myself as well. I’m really looking forward to the challenges ahead.

How to become a web host almost overnight…

I’m thinking the title of this post may one day be a best-selling e-book written by myself. The word “overnight” is a slight exaggeration, however since the second week of August, leading up to the present point in time, I have been working tirelessly (yes even through the night and over a weekend!), to migrate a web design/ hosting company’s complete hosting solution to local premises, from the over-hyped cloud.

Now, this all sounds very exciting, and it is (for me), however the complexity of moving from a hosted server environment (thanks to lcn.com) to a more localised, manageable system is enormous. Imagine turning up to a new job, and on day one the boss says, here’s your predecessor’s system, there’s no documentation on how to do your job just log in and figure out what you’re going to be doing without much help from those around you as they’re all too busy; well that’s a slightly over-hyped version of the situation here. Inheriting a system without documentation is one thing, gradually by pooling all the resources you can find you can map it out into your own terms and understanding – and can gain a good basic overview, then with more time (if available) you learn all the in depth aspects of the system and can sometimes if lucky, rewrite certain elements to your own language and document that – this is in essence reverse engineering, and a good skill to have as a system admin.

The second problem, once system knowledge is gained is, how to go about migrating from a hosted environment to one you host yourself. This sounds easy – in short it’s not! With over 60 customer’s websites and emails hosted, moving this is inevitably going to cause some disruption and downtime, this can be managed with good customer service, however what happens if certain things don’t work properly on the new system, or after a reboot not all services come back, or you don’t know the password for the user that runs a specific task? Yes, I have experienced all these and more. In fact, this whole challenge has stretched my skills and knowledge to the limit, and I’m sure once the new systems are fully live and more customers come on-board, I will look back and be grateful for having this experience.

Migration in this case hasn’t been awful, because, even though the servers are dedicated server environments, the actual hosting servers are virtualised within those, and with a little Linux/ virtualisation magic (and a weekend window of 2 hours), images were cloned and copied to the new servers. Cloning did mean taking all systems down temporarily, but I was impressed with the speed of cloning 100Gb images (under 45mins) – and with 100Mbps Fibre at my end the download of the clone wasn’t too bad either.

So, now I have the clones what do I do with them – well assuming my local servers are configured correctly to host the virtual images I can simply turn them on, change the IP addresses, do my usual Firewall magic (god bless WatchGuard!), and away we go… Could it really be that simple, well… Yes, surprisingly. In my head I thought “turn it on, change the ip addresses, reboot, test” – And wuite simply this was the case. So now I have a virtualised hosting environment, running on much more powerful equipment of which I have full control. Sorted.

Well, almost… What about the websites themselves, I may have the data, and even the emails we host, but what about the pointers to those sites, where do nameservers point, who has control…Bugger! – The first panic/ minor snag was realising we dont manage the domains for all the customers, and after investigation, we only manage around half (usually where they have .co.uk and .com and we set up one but they did the other!) – OK not to worry, should we just do it in one hit and modify the nameservers, well we could, but I don’t like the risk, but I do like control, so we decided to manually switch one domain at a time, allownig to test each one individually and ensure customer satisfaction is still at the highest levels. – This is a slow process, especially where back-end DBs and emails are still being modified all the time, as I have a new system not being updated whilst the old is (and vice-versa) so a little manual intervention is required to copy down data from the “old” to the “new”, but again this gives me control so it’s all good.

Now, it’s just a case of going through the list of domains (all 132) one by one and modifying nameservers, and informing customers to do the same where they have the control. I like the personal touch that’s possible with this many customers, but dread what we would have to plan if we did this again with 10x the numbers (which is always possible)…

 

Thanks for reading my longest post ever, for more information on this ongoing project and the tools and skills required please feel free to contact me at: jaward916@gmail.com

Automated Install Script

Due to a huge increase in employee numbers I decided it was time to speed up the deployment process. I had worked with automatic install scripts before whilst at IBM’s Innovation Centre, in fact I rewrote most of them to match newer OS versions etc.

So with a little experience behind me I decided to write a script from scratch that installed the following:

  • .Net 4 (see older post)
  • Adobe Reader
  • Adobe Flash
  • Java Run Time
  • Office 2010 SP1
  • Windows Activation
  • Internal Help Desk application
  • Cisco VPN client (for customer access)

OK, so those who are experts in this will see a very simple, perhaps clunky script, but for what I need it’s perfect. Complete with comments and prompts the script is only 42 lines. and requires only 1 click to run and a further 3 for the Windows activation and Office installer.

A note on deploying Office 2010 – I created my own .mum file and inserted into the Updates folder in the installer location to create this I ran setup.exe /admin and made it as silent as possible to install, meaning you only have to click “Install Now”, it even activates Office for me. For help on this I actually used Microsoft documentation! (http://technet.microsoft.com/en-us/library/dd630736.aspx)

Here’s my entire script (with product keys and file locations modified for security purposes):

@ECHO OFF
rem ———-Installer Coded by Jonathan Ward 13/06/2012————–
rem ——————–Major Update 18/06/2012————————
rem **Updates 19/06/2012** :- Office SP1 & Java included
echo Welcome to Auto Installer
echo Activating Windows 7… EXPECT 2 POPUPS (click OK)
“%windir%\system32\slmgr.vbs” /ipk [key goes here]
“%windir%\system32\slmgr.vbs” /ato
echo …done
echo Installing .Net 4 Runtime…
“\\[file location]\netlogon\dotNetFx40_Full_x86_x64.exe” /quiet /norestart
echo …done
echo Installing Adobe Reader…
“[file location]\newpc\AdobeRdr1000.exe” /sAll /msi /norestart ALLUSERS=1 EULA_ACCEPT=YES
echo …done
echo Installing Adobe Flash…
“[file location]\newpc\flash_iexplore.exe” -install
“[file location]\newpc\flash_firefox.exe” -install
echo …done
echo Installing Java Runtime…
“[file location]\newpc\java.exe” /s
echo …done
echo Installing Cisco VPN…
mkdir %USERPROFILE%\Desktop\cisco
xcopy “[file location]\newpc\cisco” %USERPROFILE%\Desktop\cisco /Y /q
start /wait msiexec.exe /q /i “%USERPROFILE%\Desktop\cisco\vpnclient_setup.msi” /norestart
xcopy “[file location]\All PCF” “%ProgramFiles%\Cisco Systems\VPN Client\Profiles” /Y /q
rd /s /q %USERPROFILE%\Desktop\cisco
echo …done
echo Installing Microsoft Office 2010… (Click Install Now)
“[file location]\Office 2010\setup.exe” /config “\[file location]\Office 2010\ProPlus.WW\config.xml”
echo …done
echo Installing Microsoft Security Essentials…
“[file location]\newpc\mse.exe” /s /runwgacheck /o
echo …done
echo Installing Service Desk…
md “%ProgramFiles%\Beoley Mill Software Ltd\BMS ServiceDesk”
xcopy “[file location]\Updates\Dev\*.*” “%ProgramFiles%\[file location]” /Y /E /q
start cscript “[file location]\csnew.vbs”
echo …done
echo Installer complete!
pause

I hope that some of you find this useful in your own script developments.