Q1 FY20 Part 2

Welcome to Part 2 of the end of year summary on the career side of things. 2019 ended on a high at work, and the previous post (here) started looking at the new job role and my initial project of migrating systems between domains. This project was effectively a merge of all my previous skills and a way to develop my new skillset required to take the role forwards into 2020 and beyond.

It was mentioned in part one that the infrastructure project effectively became a DevOps implementation project, and I’ll try to delve into the bits I can discuss in this post. Firstly there’s the inheritance; over 300 environments, many with specific mods for specific customers, and the remaining with what we call “Extended Solutions” – productized mods effectively. Then there was all the code itself, fortunately, the existing dev team has a fantastic grasp of the deep dark secrets of Git and I have enough of a basic understanding to pick up where others left off, but then, something new to myself came out, and that was the builds of the code. Debug or Release, MSBuild versions, semantic versioning, Git Flow… you get the idea. All very complicated to me at the time, but now it’s in my veins!

Previously the team used an old, unsupported, broken version of Jenkins to build their code, with definitions for each Git Branch, some with bat files, some hardcoded in the Jenkins config screen, basically a mess, and different rules for different people. Well, I like to standardize, so we scrapped the old, and brought in the new. Cue the amazing concept that is Continuous Integration. Having some basic experience with this in Azure DevOps (remember this post?) the concept was not new, however, implementing with Jenkins was a new experience. I managed to inherit a new blank Jenkins server, but first thing I did was reverse proxy via IIS and get it secured with an internal SSL certificate, as well as connect up to the Corporate Active Directory and restrict to our team only, then we got a service account from Corp IT and locked the server down, only myself and Domain Admins can get into the back end, and now we have a secured build server. Why so secure when it’s all internal? Well, that’s because the CTO office allowed us to have the corporate digital cert for code signing so long as it only existed in one, locked-down place. We can call it via the Jenkins application side but not extract, manipulate or otherwise interact with it. With this new updated (and updateable) Jenkins server, plus a couple of useful plugins (Blue Ocean is a must) we have a fantastic platform to manage and analyze our build process. The main feature we are utilizing is the Multibranch Pipelines via Jenkinsfile. This Jenkinsfile is written in groovy and is basically a set of instructions that define a build, for example, we can say build this solution, sign using that certificate and publish the artifacts so we can download afterward. The huge advantage for us is because we build a solution for multiple ERP versions, we can have up to 8 exes output at the end, and we now have one screen to grab them from, regardless of what Git branch we built for. On the subject of Git branches, due to the multibranch pipeline functionality, once our Jenkinsfile is pulled into Master, it will then filter down to all subsequent branches, and with this feature enabled, Jenkins will detect any new branches that were pushed back to the origin with that Jenkinsfile includes. I’ve previously written about VS Code and all the wondrous things it does, but we also discovered a Jenkinsfile checker in the form of https://marketplace.visualstudio.com/items?itemName=janjoerke.jenkins-pipeline-linter-connector. This tool allows us to check syntax, against our own Jenkins server and therefore ensures an accurate rule definition every time we adjust a build. It’s time-savers like these that have boosted the team’s productivity significantly, I recently tweeted about this improvement, as I took hold of an existing codebase and fully integrated it into our new philosophy within an hour or so!

A large part of the battle has been documenting the configuration and the overall process as well as educating colleagues (primarily developers) about how we are using these concepts and tools. I’ve found the majority of developers know the concepts and will have their own experiences with Git and CI/CD but until you document what the process should be in the exact circumstances, they don’t necessarily see the advantages or understand just how powerful these changes are, and more importantly how it improves consistency and productivity across the team. The improvements are already showing for us, and I fully expect that to continue!

Some reference material for a few of the things discussed in the post above:

Few things to cover…

Firstly, thank you for stopping by, taking time to view my blog, read my posts and hopefully take something away from them.

If you do happen to like the posts on here, then please do say so; retweet, Facebook share, LinkedIn, whatever, it would be great to get more of my content out there, and more of you on here!

Also If you have any post requests or tech questions, please send those over too, Twitter is possibly the best for that @jaward916

Secondly, apologies for the lack of posts during February. Over the Christmas break I had some good ideas which I made lots of notes for then came up with the 4 posts during Januray, however the ideas have dried up (already) and family related things have meant less free weekends. The weekday’s are taken up with the job, typically Sundays are when I get “me” time to do some techy stuff for my benfit rather than for customers!


Whilst I don’t have a nice full topic to write up for this week (I promise there are some ideas bouncing around in my head) what I do have is a snippet of 2018 so far in my world of tech/code/software etc.

During January I spent many hours getting to grips with a new major release of the software I work with, as I’m a little bit of a nerd, a lot of those hours were spent in my own time, drilling down into things, working scenarios out, deplyoment strategies etc. What I ended up with by last week was a full test scenario, remeniscient of a real world deployment. Effectivley emulating what a customer would have. The really cool thing is this allows me to very quickly test out scenarios, when a customer reports something “not working” I can run it through my servers and give them an answer same day along the lines of (usually) “try this, I think you’ve done X in the wrong place”. This is in no way a bypass to my wonderful colleagues in Support, but more of a way to assist the customer with getting their deployment up and running. I don’t generally delve into the applications, I’m not that kind of consultant. What I do is design deployments and implement them, I get the back end of a system up and running. The latest version included quite a few new technical enhancements, so getting experienced with them is an essential part of me being able to do my job!

February hasseen a few more interesting engagements for me, site visits all over the place (on top of delivering 2 training courses during January), with some more lined up, possibly even abroad.

What I am being asked to do now is anlyse, review, and in some cases redesign or reimplement deployments. Not because what they have was done wrong to start with, far from it, but more to help them become future-proof, employ best practices and become more agile as the world around us is changing, and the software adjusts to match. There’s no wheel reinvention, just a set of new tyres here and a bit of air there.

I write on here a lot about SQL Server as it is the underlying DB server platform for all systems I support. Another area of SQL that has always interested me is SSRS (SQL Server Reporting Services), basically a very smart, sometimes fiddly report generation toolset. What I have been able to do over the last few weeks is take some reports, rip them apart, analyse a few minor but irritating issues and develop solutoins to those problems. The strange thing is that I’ve not been trained in SSRS, or had the change do do anything with it prior ot this. I just saw an issue, delved straight into the SSRS builder and worked it out, for myself. I forgot I had those abillites and it’s been refreshing to remember how good I used to be solving new problems.

I’m thinking some SSRS tips in a post may be some decent content in the future, think I’ll build the scrapbook up on those!


That’s it for an update, I’ve also updated the About page on this site to reflect the last 2 years!


Oops I missed a week…

Apologies for no post last week, whilst I had built up some content over the Christmas break I hadn’t had the time to write any content this month, until now.

Sometimes life gets in the way, and Family always comes first for me, so now things are improving the content should flow again.

As a thanks for sticking around I’ll have 2 posts on here tomorrow for you!

Virtual Home Server

As one of those people who loves to be running the latest tech in both my home and professional lives it’s critical that I build the correct infrastructure in order to achieve that.

At home I recently obtained a 2014 spec Dell server, which had a fair bit of memory and storage, certainly for what was to become the hub of my home operations!

In the last 2 months I have been building the server up, utilising all the latest platforms I can get my hands on.. VMWare ESX 6.5, Windows 10/2016, Ubuntu 16.04 etc.

I now have 8 VMs, across 2 datastores having upgraded all firmware possible and playing around with various settings to balance performance and noise (it’s in the spare room)

Here’s the outcome of that work:

This first image shows my ESX 6.5 HTML5 based landing page (one of easiest to use web admin tools I’ve seen), you’ll note the 128GB RAM, Dual 2.9GHz CPUs and 8.5TB storage – perfect for running media servers as well as testing platforms for my crazy ideas!

Drilling down into the VMs I have built you’ll see a mixture of OSes and things I’m testing:

I was clever enough (somehow) to make my FTP server web facing, it’s where I store all the freebie utility style programs that I use across many systems, It allows me to use it instead of having to carry a USB around all the time!

Plex is the big one, over 3TB assigned to it for all the media we have at home, we can play it across all our devices, such as the SmartTV, Amazon Fire Stick, XboxOne etc.

What I’ve not yet got to grips with is the VMNetwork side of things, eventually I’d like to VLAN off some of the VMs to do some sandbox style testing with various OSes, maybe get back into Linux and re-learn hardening techniques etc, just need the time!

Tech Update

It has been a while since I posted an update of all the technologies I’ve been working with, projects I’ve worked on and general IT related things that have affected me in recent times, so here’s a brief update of my world of tech!


Latest Technologies used

  • VMWare vSphere, vMotion, ESX 5.5 etc. – underlying technology of the infrastructure in my current role, love the live server migrations between hosts, and the performance monitoring is very good.
  • Windows Server 2012 R2 – finally have the opportunity to work with Microsoft’s latest stable enterprise OS, and I have gradually started to like/ understand the new interface. Some of the updates to certain roles are also very impressive including WSUS (Updates), WFCS (clustering), ADDS (Activce Directory), some very nice improvements over Windows 2008 R2 which is the last main release I am familiar with.
  • WhatsUp Gold – systems monitoring software from Ipswitch, what a fantastic product this is. I can monitor all servers, virtual machines, network components, even WLANs (which is one of the most used features). I have heavily customised the environment to have dashboards for all sorts, most notably the SQL performance (most important system) I can now see real-time memory usage/ system statuses, even in-depth transactional performance!
  • SQL (2008 R2) – Something I’ve had to get to grips with very quickly as it runs the 2 most important systems for my current role. I am fairly confident with performance monitoring, understanding heavy queries, bottlenecks, even a grasp of lazy writes and paging! SQL is usually administered by a professional, qualified DBA, but we cope just fine!
  • SQL (2014) – only within a test scenario but I am very proud of my virtualised SQL 2014 Always-On testing cluster, it has given me an opportunity to better understand SQL installs, Windows Failover Clusters, SQL High Availability and basic database maintenance.
  • Cisco Wireless Networking – heavily used within large organisations across the globe, I now have a fantastic understanding and hands on experience with a network capable of almost 100% wireless coverage within an huge site! I also have great experience with security on these networks and have implemented whitelists (MAC filtering) etc. I’ve also been able to ensure the wireless network is 100% monitored across all nodes and devices, with historical data available to help resolve any issues.
  • Windows 10 – I’m a pure techie, so when the latest version of something is out, I have to try it! I was sceptical of the whole Windows 10 thing, even after trialling the pre-release versions around 12 months ago. So when the opportunity came to try at work I though yes that’s safer than at home! – so 2 months in; every device I have is now Windows 10 where possible. Being an IT admin naturally I’ve butchered it as much as possible to better understand its workings, and to be fair so far I am impressed!

Project Work

Whilst the current role hasn’t really thrown any specific projects my way yet, I am actively involved with a domain migration project initiated by the buying out of our division by another company. That is something that will be happening very soon, and will probably consume a lot of my working hours up to Christmas.

I’ve also been spearheading work to implement a standard way of displaying data across site, this has been done using Raspberry Pi with Screenly OSE which allows scheduling of web content, images and videos for display on HD screens. This again is a recent thing that so far is working very well (no reboots/crashed in 3 weeks)

At home I have also been playing with a Raspberry Pi, and have plans to implement a media server to stream content such as from the home CCTV system and family photos etc. Hopefully this will be done before the year is out once I have a screen in the kitchen!




Spam Blocking & Operation Gemstone

Those of you who manage email servers at a similar level to myself will have noticed a huge increase in malware-infected spam during 2013.

In fact it has got to such a level that it was becoming unmanageable without a recognised Spam filtering application. And with no plans to venture into the realms of SpamFighter, GFI MailEssentials, Baracudda or others I decided it was time to go all on an all out spam war.

Part 1 started earlier this year with a number of emails being received relating to Stock Market purchases and “upcoming targets”, these were obviously spam, and so started “Operation Gemstone”, so-called due to the first set being related to a Gemstone Mining Company. This was becoming a nuisance for all staff and so we started blocking emails by familiar key words, i.e once we had 3 or 4 of a similar nature we were able to deduce a keyword that we could block and that wouldn’t block (too much) valid email. We started with a transport rule “Gemstone” and we now have 5 of these! The Gemstone rule set blocks key words found in either the subject or body, it excludes emails sent to the boss (who manages his own spam) or from an internal address, and rather than deleting, it redirects the message to a holding account as a quarantine where we can forward on false-positives if necessary.

Part 2 came about after analysing hundreds and thousands of spam emails collected over a number of months and actually looking at the message headers to find more similarities between emails of a seemingly different subject matter. Naturally, the first thought was the source IP, and in some cases we found multiple occurrences of the same IPs, however on the whole they were different every time. (Where we found similar entries we blocked them at firewall level) So the one area we found similarities was in the “Return-Path” message header, with a huge number coming from addresses pretending to be American Express related (aexp.com etc.) so then came our second rule set “Return Path Block”, this again was a transport rule with a redirect to a holding account, the difference this time was to set the rule to read the message headers and look for a “Return-Path” containing various phrases. This rule was so successful that we could turn off Gemstone’s 1-3 meaning less load on the Exchange Transport servers.

But then, another realisation hit, as the months have gone on this year, the spam was becoming more and more convincing, apart from one thing… Zip attachments! On instructions from above I blocked all incoming Zip attachments (by redirect, again). Since 9.05 Monday 18th November 2013 (7 Days) 1208 emails have contained zip files and have been redirected to the quarantine account. Of these only 4 have been genuine files meant for our staff! so our “Zippy” rule sits at the top of our transport rule set and does its job admirably.

Sure these methods may seem a little archaic, but I see a couple of advantages:

  • By redirecting rather than deleting at source it gives a chance to filter through emails to ensure nothing is missed
  • By using built in Exchange rules instead of a 3rd party tool adds less overall load to the Exchange servers (from our experience)
  • We can add new keywords, return-path sources or attachment types instantly
  • The transport rules allow us to TAG the emails, by pre-pending with for example: <BLOCKED – Gemstone Rule> or <.zip file attachment> allowing us to filter emails within Outlook

If anyone has any comments about all this I would love to see them, please feel free to contact me.